 Please welcome Steve Dower to the North Bay Python stage. All right, good morning, everyone. So for this session, I'm going to be telling you about two things. The second thing is PEP551, which is a proposal that I've written and I'm putting forward for inclusion in, hopefully, the next version of Cpython. And that is how to make Python more security transparent. The first thing I'm going to tell you about is why would you want that? Why do you need that? What is security transparency? And why does it matter enough for me to write a really long document about it and get up on stage and talk about it? So to prepare for this talk, I did some research. I went into your business, I went into your company, and I took a photo of your network security. This part at the top here are the bad guys. This part at the bottom are the successful bad guys. And I can tell you all recognize this because this is exactly what your network security actually looks like. The details of this. This is a little bit of fun, a bit of interesting stuff. This is what a lot of people think of for security vulnerabilities. This is an example packet that you could send to an old version of Python to cause a buffer overrun in one of the socket functions and execute some arbitrary code on someone else's machine without their permission. If I take out the padding that was just there to space things out, what we have is a couple of hexadecimal values that are going to be written into the stack and overwrite some values. The first two are just to keep things working so it doesn't crash. The last one is actually a pointer to the system function in a particular version of libc. And the part just to the right of it is the command that's going to be run, in this case bash, in reality, probably something more fun than bash. What you may notice from this is that those are very specific hard coded numbers and they're addresses. You need to know exactly where in memory someone's function is. That means specific version of libc, specific build of the operating system. Chances are you can't just fire this packet at some random server and get access to their machine. But when you know the machine you're going to, or if you have a lot of machines to try it against, you can eventually extract this information. And there may be other vulnerabilities that give you the information you need to construct this packet to actually execute something interesting. This is what most people think of for security vulnerabilities. It's a buffer overrun. It's remote code execution. It's not the scariest vulnerability that's out there because there's an easier way to get into someone's system. And before I get there, the fix for this one is to patch everything. So many of the successful attacks we hear about are because people did not patch all their things. And you may only be a few months out of date on some components, but if you aren't patching, if you aren't applying security patches, if you aren't tracking CVEs, then you don't know that you need to add these patches, but that's the only way to defend against attacks that take advantage of bugs and code. But as I was saying, there is a more, much scarier way to get access to people's machines. And it's simply to ask them for their password. One of these pop-ups is fake. Can you tell which one it is? Who thinks the one on the left is fake? Who thinks the one on the right is fake? The answer is the one on the right, but there's only one way to tell, and it's because the battery's at 100%. But also, for those who don't know, if you do see this pop-up and you hit the home button, then it won't go away if it's a real one, it does go away if it's a fake one. But this is actually a genuine attack that is being used out there, and it shows up in a whole lot of other places. In some of my research for this, I found a Python script called Bella that uses very simple Python code to pause iTunes, pop up a dialogue saying you need to enter your password to get iTunes back. You put your password in, and then it just unpauses iTunes. And people fall for this. You may also be familiar with this kind of section of your email inbox. You'll notice I haven't read any of them. They're all unread still. Most of them have attachments. Who thinks those are good attachments? Have I really got an invoice payment copy that I need to read? These are more attacks, and these are going to vary from exploiting vulnerabilities in PDF readers all the way through to simply just giving you a script that when you double click it, it runs, and someone now is inside your machine. But of course, no one here would ever click on one of those, right? So, who has a colleague who would probably click on one of these? How secure do you think your network is given that this is what people are doing to attack? So, how do we protect against this? Okay, well, let's try something else. There are plenty of solutions to security issues that unfortunately run right up against the ability to get worked on. And so, those are not the options. So, we only really have one thing left to do here. But perhaps you're thinking this is not so bad. That at most, it's going to get into someone's, you know, sort of marketing intern is going to open this email, and now they get to pretend to be the marketing intern, but they don't have root, and they can't upload software production or whatever, so it's probably okay. Let me walk through why this is not okay. Here is kind of the idealized idea of breaking in from somewhere out in the world to get into someone's piggy bank and take emails, social security numbers, whatever information you have about your employees or customers that they want to get. This is never the case. This is the worst case. This is someone connects from out in the world, somewhere on the internet, through your crown jewels and takes them. You protect against this. This is a firewall. That's easy. So what attackers do is they find a computer they can get into somewhere in your network that has access. And all they've got to do is get into one other machine and then if that user has access, then they can just go in and get it. But if they can't do that, I forgot to kill notifications. I'm sorry. But if they can't do that, all they have to do is get into another machine that then at some point has access to that second machine that can eventually get to the crown jewels. And what if that machine they get into is not one of the ones that you maintain, is not one of the ones that you know is on your network because you have employees and employees have phones and they connect those phones to Wi-Fi. They have laptops, they connect them to Wi-Fi, they take work home and suddenly you've got all these ways that an attacker may be getting into your network that you don't know about and from there, getting to the crown jewels. You've also got network switches. Do you think your network switch is secure? Have you changed the default password from admin, admin? You have network connected printers. Network connected projectors, if you're lucky. Network connected coffee machines, thermostats, light switch, everything you could think of is on your network in some way. If you go and ask a system administrator, what is your network layout? What devices do you have on it? They'll give you some of the workstations. If you ask an attacker, what is on this person's network, they will give you this diagram because this is the truth. Your attacker is more patient than you. If they can't get to their goal within five minutes, they don't give up. They just spend longer getting there and they know your network better than you. They will find everything that is in any way connected to your network and use it. If you were tracking some of the recent malware outbreaks, the ransomware that hit the entire world, most of those were designed to only propagate to machines on the same local network as an already infected one. They weren't designed to cross the internet. Think about that. These were released in like a few companies in Eastern Europe, typically, and within hours spread to the entire world despite not trying to. What devices are connected to your network? So, given network security looks like this, what do we do? The answer is to switch mindset because everything so far has been, how do we stop people getting in? There is a concept that's gaining popularity now and is recognized as the only sane way to exist and that is to assume breach, to assume that despite all of our really, really good protection, the attackers are in. The attackers have gotten past them. They're on the other side of the sieve. They're inside our network. What do we do? First, what are they going to do? They're trying to infiltrate, but we assume breach, so we assume they're in. We try to protect, but if you only protect and have no security cameras inside your building, it's as good as locking the door. When someone gets through, you don't know that they're in. They're going to try and persist. I'll dive into this in a bit, but while they're persisting, we're trying to detect. They're going to get in. They're going to run their code. They're going to hide. They're going to bounce from machine to machine, and they don't want us to find them, so we're going to detect them. And then their final goal is to exfiltrate, to collect whatever data they're interested in and get it out, either so they can sell it or alternatively they're just going to cause damage. Either way, it's our responsibility to respond, typically by removing them from the systems. Persistence requires a handful of things to be really effective. This is what attackers look for in a target machine when they're trying to persist on it. They want atypical execution. You don't want to have to install something on a machine that you're hacking into. You don't want to have to use GCC on the machine that you're hacking into for your hacking tools. You want some way to get code on there running that is atypical. It's not normal. You don't want any validation of that. You don't want to be caught in any whitelist or blacklist of bad code malware scanners for those of us who use them. You need the ability to write files and to access the network because that's how you're going to persist, that's how you're going to keep settings on the machine, that's how you'll get onto someone else's machine. And you want to be able to create servers, creating a running server of any sort that someone from the outside can connect into to get information out is a very valuable thing for exfiltration. And of course you don't want any logging because everyone reviews their logs and will see you very quickly if everything you do is being logged. You laugh about that and if you laughed about that you don't care enough about security for any of this to matter because the people who care about security review their logs and their logs are much bigger than your logs but they review them anyway because it's the only way to detect when attackers have gotten through your wall. So the tools that are very popular for this. Bash, obviously, you can do a lot, you can do all of this stuff with that but then you have to write Bash scripts. PowerShell also very popular especially on Windows machines, it's already there, does a lot of this functionality so much so that it became so popular with attackers that the PowerShell developers went in and added a whole lot of the functionality I'm about to talk about for Python. And now it's not so popular. There have in fact been many, many stories of attackers completely taking over networks and simply upgrading PowerShell has enabled the defenders to get them out. Pearl falls under the scripting language category, can do all of these things. Again, you have to write Perl scripts and of course, the reason that we're here is because Python is also a popular tool for this part of attacks. Again, we could fix this. It's very easy to fix this. But it doesn't work, it doesn't let us continue on business because even though we can be the most paranoid security person in the room have the most power and you still can't override the CEO who wants to sell a product and make a profit. So this is not a viable solution but the viable solution is to go from banning everything to knowing what's going on. This is not ordered as in reveal the code and sign off on it and then forget about it. This is seeing everything that's happening on your network. This is knowing the things that are going on at all times so you can detect and respond. This is security transparency. Your operating system can already do this. If you enable it, your operating system can log every single process that's run, every single command that's run, every single file that's written, every single network connection that's made and you can have a massive, massive stream of log coming from your operating system that will tell you when someone is doing something funky. But unfortunately, if someone starts running Python, you don't see those events. And so this is a legitimate statement that many paranoid security people have made. We cannot have Python running on this system because it's possible for someone to start running their code in it and now we have no insight into what they're doing. Don't take photos of this slide yet, it's gonna build up. So, PEP551 is my proposal to add security transparency. The main goal is to give a response to people who say this so that we can change their statement. We don't want any employers, any networks banning Python outright because it's insecure. We want them to say it's okay because it's audited and because we can show that what it is being used for is what it's intended to be used for and even add some, albeit weak guarantees that that's what it's doing, that it's actually okay to be on there. Now you can take a picture of this slide. So, here's an example of some malicious Python code. The fun thing about Python is you can put as much code as you like on a single line and then you can execute it from the command line. You don't have to put a file on the file system to run as much code as you want. This big chunk here is base 64, which we can tell because we're gonna decode it. Simply looking at this log, this is all your operating system's gonna show, by the way. If someone jumps onto your machine and runs this using one of the exploits earlier, you'll see this in the log. You can't automatically detect this. It's encoded simply, albeit, but you can easily imagine that you can use different sorts of encodings with similar import decode calls. And then they're gonna execute a valid. You'll see it used like this. If you wanna copy that down and run it, go ahead. To save you the time, that's what happens. If you do wanna copy it down and run it, I'll save you the time, go ahead and take a photo of that. I promise it doesn't really actually download tools to your machine. So, by making Python security transparent, we get to see information that we wouldn't otherwise see. In that case, you would get that command line to show up in your operating system logs. And then the fact that your machine is totally compromised may also show up eventually or it'll hit the news first. With transparency in Python, you get more events in your log. You get to see exactly what code is running. You get to see what files, what network it accesses, and especially what red flags there are in that code that can help you trigger alerts based on that logging without you actually having to read everything. So, with my sample implementation of the PEP, I basically dumped all the events that come out of running that bit of code. This is heavily abridged because Python imports about 120 modules at startup just to run that code. But the rest is here and there are two red flags straight away. We're compiling and executing arbitrary strings. Most production Python code does not do this. Most Python code that has passed code review does not do this. And the production code that does do this, I'm sure is only temporary and has only actually been running on the production machines for a year or so. But you can see here, we have the full string that was executed. That top one there is that base 64 chunk, but now it's decoded. And this is being passed out from the interpreter at the time when it compiles it. You can't encode this anymore because the compiler has to compile it now. So it can't still have any sort of encryption. You can't even rot 13 this because Python can't compile it. So you get the completely decrypted code that we didn't see in the operating system log. And then the second bit is the code that it downloads and executes. So this is a simple bit of code to obtain a payload from somewhere else through a redirection link in this case and then execute it locally. You can also see this chunk of very useful information here which are the network addresses that were accessed to get that code. You'd likely see these anyway from the operating system. But in this context, we know they came from Python. We know they came from the same Python process that executed that code. And we know that it was immediately followed by compiling and executing more code. So much more context on what happened. But it also gives the ability to intercept this stuff in Python. But to be completely clear, if this change gets into CPython, it does not automatically make Python more secure. In fact, if it gets in, it has literally zero impact on Python right away. This is about a way for system administrators, a way for end users to add the hooks that you need to integrate with your logging system. If you don't log these events in any way, they don't go anywhere. They don't do anything. Detection is your job. You will see these events, but if you're not looking at them, if you're not reacting to them, you don't get any benefit. And overall system integrity is also your job. Someone can simply bring their own copy of Python and not use any of this at all by dropping their own one on there. You haven't achieved anything. And of course you can pay people to be paranoid for you, which if you're not already paranoid, I would recommend. But this is not a fix for security in Python. This is enabling the security people to be told you don't have to ban Python. Here is how you can make it work. So what kind of events do we trigger on? I've broken down into three categories here. There's things that all code does. You will get thousands and thousands of these events because everyone imports, everyone compiles. Not everyone should use sockets, but eventually you do. And everyone's going to use URL lib at some point. There are things most code never does. You'll be interested in these events when they show up, but they will show up from time to time. Exec, eval, ID, array, C type, struct, monkey patching can also be logged because we can raise events when you set an attribute on a type object, when you set an attribute on a module. And this is really important for detecting if someone's trying to bypass your detection. And finally, the things that nobody should ever, ever do, except when everything's dying and you need to temporarily put it in a production for a couple of years. Add in clear audit hooks. If you're listening to events and someone else comes along and says, stop listening to events, you want to know about that because they should not be doing that to you. Directly creating code objects and function objects. Great for party tricks, but not good for any other use. And a really big red flag that someone's trying to do something they shouldn't. Directly accessing frame objects. Or if you've ever been told that pickle is not secure, it's because of this function, unpickler.find class. So we'll just let you know that someone, hey, someone is unpickling something. Here is what they were trying to unpickle. You might want to have a look because they shouldn't be doing this. So this is the two parts of the API for adding a hook. There's a few more pieces, but this is the most interesting part. You can do it from C. This is really valuable because if you're prepared to rebuild the Python entry point, which is literally that's it, the POSIX version has a bit more encoding stuff around it. But this is all the entry point is, when you add the hook before that, you will get auditing events for the entire initialization process. Everything that gets loaded at startup, everything that gets processed at startup, this is a lot of logs. On the other hand, if someone has already compromised your Python before you start, then you really want to know about it before you start. You can also do it from Python code. This is great for testing. This is great for demos. I'm about to do a demo with this. It's also very handy if you just want to actually get the information for some other reason. If you want your unit test to report who did monkey patching while I was unit testing, you could add a hook and collect all the monkey patch events and dump those out somewhere. So there are uses beyond simple security transparency for this, there are testing validation uses for this, but this is the motivating reason for bringing the proposal. And of course, these hooks don't have to succeed. And if they do throw an error, then it's going to abort the process. In most cases, that's probably going to bring down the entire application and that's likely what you want. But most of the time you're not going to fail it because this is really, really aggressive and actually going to be highly disruptive. So detection is the main thing, but there is the ability to have some simple prevention in place using these hooks. So let's look at an example for that from our earlier bit of code. I've got a hook here. We're looking for the URL event, which is going to send through the URL that someone's requesting. We don't use Bitly internally, but that attack before was using Bitly. Okay, we decide it's good enough to say no Bitly for our app. This might actually be useful if you're concerned that someone might figure out a way to coerce your app into downloading an arbitrary URL and executing it. That's an application level vulnerability. Python can't protect you against that, but many attacks do involve things like send a carefully constructed header and then we'll take that URL out of it and redirect to it and download it. So this may be useful for that case. First, we're going to print a message. You probably don't just print this. You should be sending it to a log collector, syslog, wherever that happens to be for your case. But we're also going to raise an error and simply just stop. Finally, we add the hook. We do this in Python code. So this won't apply during initialization. This won't apply during site packages, during import sites, site customize. A lot of places where this won't be run. But after we've added the hook, it can't be taken out. So once the hook is in, you can't actually remove it. So from that point on, we'll have any indication that someone's trying to access a bit.ly link. So when we run this sample again, instead of getting the nice ASCII art, we get a warning message. This is the print. This is telling us, hey, someone's trying to download from this URL. We also get an error. It's actually just blocked and those tools that were downloaded are completely and entirely blocked. To be clear, this is a proposal. This is not yet in CPython. We're still discussing it. We're still arguing about the motivating cause, the value of it. Does it need to be in core? Should we be giving it to the people who need it separately as a patch so they can build it? Right now, it is available as a patch. That link at the bottom will take you to my GitHub branches against 3.7.0, alpha 2, and 3.6.3, where you can simply grab the code, any operating system, build it, play with it, use it. You can definitely get it now. If you have questions, if you are interested in using it, if you believe your company organization would be interested in using it, feel free to get in touch with me. I can be reached at that email address, that Twitter handle, or since we're about to go to lunch, feel free to find me and let's go have lunch and chat. Thank you. Thanks. Thanks, Steve. I love these talks, finding out what the future of Python is going to hold. It's always great to see things coming straight from the core team. As a token of our...