 Kong here talking about beyond adversarial learning, security risks in AI implementations. So he's gonna take it away. Well, thanks for the introduction. Well, welcome to my session. I'm Kong, and here I'm gonna talk about probably something different from the other talk in this AI village because I'm gonna talk about really the security part, I'm a security person. So this is the work jointly with a couple of people from Chihua 360 and Valine from University of Virginia. I have a full author there, and there's a four bulldog picture there. There's no correlation between the author and the picture, okay? And we had a paper published in the ITP Security Privacy Workshop on Deep Learning and Security in May. So this is pretty much the same talk. I hope you guys were not there. So this is something new to you. Okay, so first a brief introduction about myself. I'm professor from the University of Georgia. In the past, my work is mostly on the software security side. I organize lots of character of the flag teams. And some of them are well known like Sack Dog or Dysack. I'm also the founding mentors for the Blue Lotus team. One of our team, like Team Dysack, was the last, the finalist of the Cyber Grand Challenge like two years ago here. We were one of the 17. So recently, I'm actually right now taking leave from University. I'm doing consulting work in industry, okay? So now let's go back to the main topic. I'm not gonna talk about the machine learning algorithm. I'm actually gonna talk about the, a little bit about the deep learning implementation. We don't have much time, so I'm gonna try to be brave. So I'm gonna start with like a simple question example. So you all seen this, if you work on deep learning and you open the textbook, this is the first example you see. My question for you is like, how many lines of code for you to implement this? This is an MNIST that does the digital recognition. You take a picture, it has a digits in it, they recognize it. How many lines of code to implement this? It's not long, like is it code for implement this pretty much like you need the networking part and you need the software totally. It's like, I would say less than 300 line, okay? Now how can you do a fancy thing like all the magic in such small piece of code? It's because our current implementation on most of the deep learning application, they're not building from scratch. You only use a framework. There are lots of common framework like TensorFlow, Cafe, Torch, you might use some wrapping language but pretty much, at least for the people I talk to, they don't build this from scratch, okay? And when we discuss the implementation algorithm or even you look at the like adversary machine learning, you really talk about on the top layer, like you're looking at your model, your parameters, sometimes you might talk about your training data, but we don't talk about much underneath, okay? So the underneath part actually matters. Let me show you an example. So you probably, if you are in this area, you have a look at lots of cat pictures of kitty pictures, right? So this is the one example, like we got from Cafe, we download, we run it, you can run this easily. I assume you have some model from other place and then you give a picture, produce some result. Now what I did, additional thing is I did like, this particular one can build as a Linux application. So I do an IODD, okay? For people that in this field, like if you do security, no, okay, what this does, it shows all the library dependency of that application, right? So in a particular one that recognize cat, as the code is now long, but then it depends on 137 libraries. Okay, that's a loss of libraries, right? And we did some not really scientific calculation and here's some number I got. This is actually, I did this in 2017, it may be old, but gave you a rough idea. So I go to count the lines of code in Cafe, TensorFlow and Torch itself. Now I said this is not scientific because some of them are implemented in C, some of them in Python, so lines of code doesn't really, you cannot compare them. But roughly you see like, there's a few hundred K lines of code in each of this framework, okay? Now in addition, they all depends on bunch of libraries that not even include that a few hundred K lines of code. So there's lots of package. And then I gave a few example like, for example, Cafe, I put there's like Lib Z, OpenCV, LibProtobuf, right? You know, there's lots of package are common. You need to have some language to parse the model or the parameter, okay? But you don't want to implement this from scratch. Well, when you have this complex dependency, people in security are really excited. We're happy, right? So that's what we, you know, we did some work. This is only in the summer, I was doing some consulting work in the CHIHU 360, like in one summer, this is the number of CV you find related to all the other early package. And then we actually find more. I didn't put there because this is for all these slides for previous paper published early this year and it was written last year, okay? And then I put a picture there. And the picture says artificial intelligence is no substitute for natural stupidity, okay? Now this was different meaning before, but I want to use it to say that in AI, we are talking about so many great things about AI. But if you have this stupid programmer that doing something for this brilliant algorithm, then you have trouble, okay? Now give you, to some more concrete, some people say, oh, this is not in the, you know, where are these bugs? Give you one example. This is, I didn't expect we have such small screen, okay? But I'm gonna try to read it out like, so this is a screenshot. I go to the, like the GitHub of CAFE and this is like their image data layer. So I click the link, like the CPU implementation. I can now even read here. So this is the image data layer. Basically I'm trying to do screen capture to say where the code is. This is in CAFE image data layer. And if you click it and this is the code there and then there's the include that actually points like the use open CV, okay? Now, so which means that even open CV is a dependent package. Every time you use CAFE to write an image parsing, you know, deep learning application, you say, read in the image to this layer, you're calling this code. And this code actually calling open CV to parse the image, okay? And then, you know, I'm just, again, I don't expect you to read the code so give you a rough idea of this. I pick one of the CV, you know, we find that it's loose here. What it does is it read a picture, it tries to, you know, parse it. And the part I highlighted that's actually controlled by the input. That's in the image. So what happened is this piece of code tried to read the image into a layer and then the first they read some parameter from the image that tells you really the size of image. And then this is a particular part of control the color palette, like depending on how many color you have, right? And then they later use this number then they have a memory copy and then, you know, they actually have the wrong number there's the heap overflow there. And then I show you, this is, we report the bug in 2017, an open CV developer actually fixed it. They, you know, even the patch, this is a patch for that bug. And you see that they actually shows that it removed two lines, three lines, actually only one line because the two blue line and the bottom of the two right line are the same. So really the top, the CV assert part that's what they add in. So that adding a line says color use less than 256 because early on I told you, like the input can be larger than 256 though there's a heap overflow, right? The reason I bring this up is that again, this is the programmers now that are good. They put a patch, this patch is not complete, unfortunately, because they didn't consider the number less than 256, but what if I give a negative number which become a large number? So bugs do there, I try to convince them to say, hey, there's still a bug. The guy is like, I don't care, right? You know, this is really not my problem, okay? So I have to go further to show you what's the problem. So again, go back to this kitty case. They take a picture, they gave to this, you know, cafe model, a cafe program that classify image. And to make sure I'm not cheating, so I download the model on the Berkeley lab website and then they claim this is a train from the image data, the competition, right? And they're using the Google model, so that I use all of theirs. And if you gave that kitty picture, then it will tell you, okay, what kind of cat it is, give you different category, what the probability. Okay, so we take that program, I cross this four picture because I need to convince this developer, say there's something you need to fix, okay? So these are the four picture I show on the front of the slide. And the top left one, the Bulldog, is I grabbed from the internet, okay? And the reason I grabbed from the internet of grab a Bulldog picture is because, okay, Bulldog's the mascot for University of Georgia football team, right? So I grabbed that one and then I messed up the other three. Now, visually, you look at it and feel like, okay, there's something different for people to sit close, okay? I'm not doing adversary machine learning, I'm not touching the pixel, okay? What happened is I actually messed up the metadata, I actually lie about how many color they use, that's why it looks slightly different, okay? But the picture content is the same, okay? So I only messed up the metadata. This is the result of the classification. Again, I don't expect you to be able to read it, so I'm gonna use the image to show what happened. This is the same program using Berkeley's model, Google's data, the result. Of course, the initial one I throw in, it's a Bulldog because that's the picture I got from the internet and this, I have to say the machine learning algorithm does pretty good, this is the real picture, okay? They're really good. And I talked to AI expert, one thing I'm really always surprised is that they always assume that data actually come from natural place and you take a picture from with your phone. Well, no, the second one, certainly you imagine, like I put a picture, I messed up the data, psychfog, okay, no surprise, right? I have a key overflow, of course I psychfog this. Okay, almost psychfog here. Okay, so that's easy to understand, but I decided to push it forward a little bit. We put a shell code in it, okay? So this time, I lied it, lie, I always imagine what this Bulldog wanted to be classified, so I imagine there's a Bulldog want to fly, you know, so I decided to lie it to, you know, classify as a flying pig. Okay, so that's the result. In fact, because this is not, again, this is not, I don't see much learning, I'm not trying to get a panda become a gibbon, I can mess up the pixel. The output is generated by me, my code, they are actually writing my code, I can actually let it say anything, okay? I can let a panda become panda, you know, a gongfu panda, anything, okay? So to go further, if you didn't see it, like I have a shell there, so I let them like the cafe program take my picture, then they spin out the shell, and basically the picture I show in there, I own you. We even did this actually in a cloud environment, I didn't have this in the picture, but you can imagine, like you running a classification service using an application, you just say, you're lazy, you just copy the, using the default application from cafe or whatever example, you run this as a cloud service, this same code can own your cloud, basically. You have this interface, I can have a shell come back. We have a demo on that, I have a video, I present that in the POC conference. So to summarize, okay, I mean, there are lots of other bugs, I only show you cafe one, don't consider other frameworks are better, okay? We also have TensorFlow bugs, okay? So, but overall, what I want to tell the audience is that deep learning frame application depend on lots of third party packages, okay? You need to be careful about all this and complexity leads to vulnerabilities. If you have this large complex program, you're gonna have trouble somewhere, and even your algorithm is perfect, you have trouble. And the risk of application here, I only show you, I can do deny service, I can segfault, I can get system compromise, I can certainly do misclassification, I sort of also show that, so the evasion attack. So that's pretty much all I want to do. There are other examples I can show you offline, feel free contact me.