 Cool, so the first top we have is by Sven Katel, data scientist at Endgame, and he's here to talk about Adversarial patches. So basically the plan for this talk was I was going to make t-shirts with Adversarial patches and the volunteer t-shirts all have their stuff. The problem is I tried for about two months to do it and eventually it's not working so well. And I'm going to talk about how to make these patches, how they work, why the thing and then like reasons or things. So I'm a data scientist at Endgame, we've already heard about me. So that's just what we're going to be doing today. So the background of Adversarial examples, you've got two basic classes of Adversarial patches. One where you modify the pixels coming in from images, you modify the pixels of an image so that a classifier screws up. So this is a classic thing where you've got a cross-cutting inside method attack, or you've got to be saliently, or various different types of attacks. And we've most, I hope most of us have seen Ian's panda. We've misplaced it as a given with some noise. These sorts of attacks are good for understanding how their own deep learning works, but they don't actually attack. They can't do anything in the real world because you'd have to grab the input out, the output of the camera, and modify the pixels on the fly. And it's not a realistic attack because it would require more hardware computation than you probably can get into a system. The other type of attack against machine learning is the physical modification. So what you can do is you put a set of stickers on an object and have it misclassified. One of the first papers that did this was one with, they were attacking the classifier that do stop signs because, you know, self-driving cars, stop signs, and they came up with a way of producing pixel modifications that they can put as stickers on the stop sign and causes the classifier to get misacut, you know, work badly. There's some evidence that their attack does not work very well in real life, and you have to have a week of work classifier and a few other things to make it look like it works. But these sorts of attacks are much harder to do. So adversarial stickers are the second type of attack. You actually can put a thing on a person and have it. Okay. So the way you build the attack is there are three parts of the attack. So you have the actual sticker. So listen, we will like, I'll put this little object that contains the image, the floating, the little tensor that holds the sticker. It also holds a mask, so you can have a sticker be different shapes. Then we have a placer that puts the sticker on a bunch of images so that you can train against those images with the classifier. And then we have a trainer that actually does the training process. So it's very easy to build them. Just, we've got computers going to build these three components. And if you go look at the original paper, there's a link to a GitHub account, which has basically these things laid out. So you have a sticker. Well, the way I've built the sticker is you can take an image, a PNG image for a mask, and it can be whatever you want, as long as it fits within your training target thing, or you can take a NumPy array, and then randomly initializes the patch around 50% gray with a bit of noise. And that's the start of the sticker. It initializes that. When you pull the sticker, it gives you back the tensor for the sticker. I did this in PyTorch so you can start applying it to images. And it does a few things where it clips it, so the sticker is an actual image, so it stays in the pixel range, and then it multiplies by the mask so that it looks like that mask. An example, if we got the patch with Def on logo, and then the mask got produced, and then we have a game logo, and the mask got produced. So this is the simplest component, but like the thing that we're actually training. This is the corp, the rest is decorations around this. Probably the most important component is the placer. So what this does is you can take, you give three ranges, so a range of rotations that you're allowing. So any rotation in this range, so plus or minus 15 degrees, 30 degrees, a range of translations you're allowing, and a range of scaling that you're allowing. You can scale it up by 30%, or down by 30%, or whatever, and you randomly select it. When you call it, what it does is it makes a sample of a bunch of affine transformations within the ranges you've provided. It copies the sticker into as many images as you have, and then transforms the stickers to be the right shape. It also does this with mask, and then applies it to a sticker. For how it works, so we've got the stickers on the side, and the masks on the side, and here is an affine transformation. So randomly select a s, tx, and a ty, and I'll select, in this case, 25 of those, and then apply that to each of those stickers, and then you get the transformational amount. So you see on the left, you have the untransformed stickers, and on the right, you have a bunch of transform stickers, and so then you have, you store that, and then put a little bit of math, and you apply the stickers to these images, and you get, I'm sorry, for the image quality here, you get a bunch of stuck images, like, stuck images, and that, at that point, you can feed to a classifier. And then we get to the last component, the trainer. This thing, you can take in several models to train against, so these will be your white box models that you're training against, your attack against. It also takes an optimizer, and a few other things you need to do gradient descent, and this thing places the models and the stickers on the GPU and does all the stuff, the juggling that you don't want to think about, and actually what it does is just perform the training. We have a basic training, the basic training loop here, this is for CIFAR, so we have our sticker on the left, and so the sticker goes through the place with some images, and so it gets placed on the, placed on these things. So we have a bunch of 25 images here that have been, had a sticker attached to them. Then this goes into a classifier, whatever classifier we have, that thing, and then we take the loss with respect to our target class. So in this case, I was going for the ninth class, the tenth class of CIFAR, so we take loss with respect to what class we're trying to get, and then we do a back prop gradient descent against the original sticker, because this entire thing from the original sticker to the loss is smooth, there's a smooth transformation, so we can do gradient descent with respect to the sticker itself. So this is the basic training loop, and if you wanted to do a single neural network, if you wanted to train it against a single classifier, you just do this, and you're good. But if you want to train it against several classifiers, so what happens if you want to train it against two? Well, this, you can train it against two by going, okay, in my loop, I'm just going to do it twice basically, the same, the same basic loop twice, I'm going to have, you're going to do the first classifier, and then the second classifier, and then I'm going to loop until I get a sticker that works. So this is for an ensemble training against my own, and then there's the other way of doing it, which is what you do is you, instead of feeding into one classifier, you feed it into two classifiers, you get a loss added up, you take the gradient of the way back, and you get a result. So those two things work, and then you can train up a classifier, and here we get results. So if we have, I did this with, like, the only thing I could get this working on was Cfarm, and we'll happily talk to people about, like, null results here. So what I did was I got those four models inside, so VGG style network with nine layers, VGG 13 layers, a ResNet for 22, and a DeepNet, which doesn't perform that well with 10 layers, and then I targeted against, okay, I targeted against a ResNet 10, and the graph on the left is for that scale, for the zero scale, we had no performance, so this is a, this would be the image scale down to a point. Obviously it's not going to do anything useful, and then when it covers 20 percent of the image, it's kind of useful, and then at about 40 percent of the image, it kind of, it saturates the convolutional layers and causes misplacification. For this sticker here, you need to cover 40 percent of the image, and then it'll cause a misclassifier, misclassification. And then you put, this was trained in the sequential model, and then if you train it in parallel, it doesn't perform as well. There's reasons for that. So I went, built this out, tested it against Cfarm, Cfarm was the unit test, and then I went, okay, I'm going to get this working on ImageNet, and so I went through the original paper, and with Cfarm, you can, like with this stuff, it was trained with a certain range of rotations, scalings, and conflations, and if you take the rotation to be within, with plus or minus 90 degrees, it doesn't, it doesn't train well, it doesn't work. So in the original paper, they mention, oh, you can do complete rotations, it can be stuck in any orientation, it's fine, it'll work, but what they, when you actually look at the code, they're only translated by plus or minus 15 degrees in their range, which, that's the first thing that I was like, this is a little bit not so good. But so with Cfarm, this thing is with their parameters, and if you go for bigger parameters and more manipulation of the applied transformations, is that curve drops off rapidly, with more view photography. So for these, this is how the, like the gifts of the training through an ImageNet. So what I did was I grabbed four ImageNet, four ImageNet models of the PyTorch repository. I stuck, I pointed my trainer to train against those four, and it's the same set that if I trained Cfarm again, so a ResNet, ResNet 50, two VGGs, and a DeepNet. So I trained that again, against that, and I targeted a ResNet 53 or one of the VGGs. And the performance I got out of this was about 10%-5%. So it doesn't actually function as well as it exists. The paper that they have says it should work at the same as the Cfarm curves. And I got a flat curve. I went back to the paper. They're only using 5,000 examples. They're using models that may or may not, I don't know what the models look like, could not get the thing to work, after like two months or so. But so basically, like, welcome to AI Village. We have a novel, as well. So references, like, I only use, like, the original paper. And then we've got Ian's Panda and move to the side. So if you guys have any questions, yeah. I would be able to then do a white box attack. And these were designed to be black box attacks. And it's more effective with white box attacks. If I wanted this to be a realistic attack against machine learning models, which I should do, and this should be robust enough to be black box. Also, one of the things that makes the image more robust to train against several models is supposed to be an ox. It classifies as a contra shell. That's partially because of teacher printing. So it's not the teacher printing isn't sensitive enough to get the fine details required for this to overwhelm the classifier. So I started this project with, like, yeah, these adversarial patches, the looks, the paper looks awesome. I'm going to do it. And then end of the project with like, yeah, I don't remember. And it might be that I made a mistake somewhere. So the code that I wrote is up on GitHub. And I will be writing a blog post for the AI village and describing exactly why I think this is an old result. And with all the code, and it's designed to be easily readable so you can see what's going on. But yeah, yeah. 40%. Well, from making a few thousand of these, I have inclinations to say you're going to get situations where, is it the deeper the network, the more effective the attack. I think it's because it can diffuse out more before it gets to the final fully connected layer in the combinational layers. Yeah, I trained it on my desktop that has two 1080TIs. And like a 6700K thing. I also trained, I actually trained those C4 things on my razor blade with a 1060. So the, this architecture, the model parallel architecture is about seven times faster than the sequential thing because you can put it on one of each GPU and you don't have to juggle things. So you can train in about half an hour for image against a portion of image. You only want to train it against maybe 10,000 images. So you don't need to go all the way. Also, when you actually go to decide, okay, I'm going to do this, pick a very, very aggressive step size. So their step size in the paper is five. And I found that if you make that too low, it just stops. It doesn't work. So these gips were made with the step size of 0.3 to make them pretty and make them actually look like they're doing something. But they're, yeah, yeah. Do you want to try that out? The code is open source on the repo. So the way I built this was with the so the, the Appian grid transformation and PyTorch, which was designed for spatial sequential networks. So in those, they have a training parameter to unperturb the network. So you know how you put an Appian transformation on an image to do data augmentation. So these networks were designed to learn how to undo data augmentation as well as do a classification. So they're the architecture that for thing is a, there's a pair of operations in PyTorch that do that step and they take in parameters like that. Like the scaling thing that Appian transformation is just a grid transformation and you can see it's a smooth transformation. You can feed data into that thing to fix your transformations to be useful. So there's actually a paper for doing, manipulating the Appian transformations. Yeah, I would just, if I was a specific network that I was targeting, I'd just do this. I wouldn't try to do our ensemble training and stuff. Just focus on the net one. No, I wouldn't have to do anything. You might be able to be a bit more aggressive in your training too. Yes, that's the thing. The, when you do this, when you take this, if you allow the rotation to be sampled from within a plus or minus 90 degrees, the whole system bake down. So the rotation that you is actually effective is plus or minus 30 degrees or 15 degrees. So they already pretty much say these stickers have to be sitting straight up and night. Yeah, but yeah, but they, they learn how to recognize dogs that are upside down. Maybe. So I tried, sorry, I didn't with the VGG 22. I think the scale is lower than one. So the scaling is the receptive field at the bottom is three by three, but then you go up one layer. The receptive field that I think think when you go all the way back down to the image is nine by nine. And then one layer up, it's like 27 by 27. Sorry, it's not, not by nine. It's five by five and then six by six. It's sorry, seven by seven. So the receptive field for this epics of the top of the VGG 22 is a three type, yeah, 22 times two plus one. So it's massive. So this pixel sees a lot. Yeah. I wanted to make, yeah, we also with resin engines. And that this we've tried all sorts of also, you can try it against every architecture. Inception is also one of the ones that you can train against. So I think we're going to mean Mark Maeger is the next guy. One quick question. And then I don't know, I'm going to write the blog post and see, and then after that, I'll feel if I feel like continuing to work on this is fine. But I think we need to move on to our next speaker. So Mark