 Yn ymlaen, mae'r unrhyw gwn i'r cymdeithas. Rydw i'r cymdeithas cyllid CGI. Yn ymdyn ni'n gweithio ar y llwyffol y gallwn gwirionedd a gweithio'r gyrwch. Yn ymdyn ni'n gweithio ar y module python picle, sydd yn ymdyn ni'n cymdeithas cyllid yn ymdyn ni'n gweithio'r picle. Mae'r hynny'n gweithio'r picle? Mae'r hynny'n gweithio'r picle? Okay, that's pretty much nice if you're sent across the board for both. I want to start with a bit of backstory. On our project at CGI, we make heavy use of an orchestration tool called Ansible. And Ansible is very nice if you're doing DevOps because you can write roles and playbooks and they will configure your servers into some deterministic, independent state and they will do it across multiple environments. So you might have a dev environment on laptops, you might have another test environment in the cloud on Amazon, for instance, and a third one that you're deploying to and we can reuse the same playbooks. So it's very nice and we use this heavily. The problem is because we use this heavily, our Ansible roles and playbooks got heavy. We have them running lots of tasks, we have them communicating with lots of servers, doing lots of steps, and a lot of stuff that's in the loops and it has got, it got slow. Every round trip to a server was costing us milliseconds and when you have hundreds or thousands of round trips, those milliseconds add up. Thankfully, just shy of a year ago, a new project appeared on the scene called Mitogen and it has within it a provider for Ansible that can speed up your Ansible connections greatly and there are testimonials from people who have used it. My playbook has gone from one and a half hours to three minutes. It's that dramatic. So great, Ansible is fast again. There are a couple of caveats that we had to, that we're having to overcome. So if you use Mitogen just on its own, no Ansible, this is the sort of thing you'll see. You're in Python set up a chain of hosts that you're connecting through until eventually you hit the box that you want to run something on and Mitogen will behind the scenes tunnel all your connections. Doesn't matter if you're using Docker or SSH or some other odd thing that only you use, it will do the tunneling for you and unlike Ansible it will do all the tunneling without having to write things to disk at each stage. In this example we're ultimately calling OS.system with a random Python script. Equally that OS.system call could have been any other Python function call. It could have been a standard lib module, it could have been a third party module. Mitogen would have set things up so that when Python on the other end tries to import that module, Mitogen will be there ready and have provided it. So you don't need to install Mitogen or your third party modules on the boxes that you want to run this code on. Essentially Mitogen is a zero install remote procedure call system. This is Mitogen without Ansible but when integrated with Ansible you carry on using your Ansible inventories and playbooks without needing to write Mitogen specific code. The problem that I alluded to earlier is that at the moment Mitogen uses the pickle module to return the results. So when master sends the request to run OS.system the result of OS.system is pickled on the remote system and then pickle is sent back up the chain and the master system unpickles it. And as you may know this can be problematic. So if you want to use Mitogen with Ansible it can be as simple as installing Mitogen and setting these two lines. But with the caveat that you need to be aware that when you are doing this you are essentially trusting all the systems that you are controlling with code execution on your bastion host, on your master. The benefits of running Ansible with Mitogen can be quite dramatic. This is a slightly artificial benchmark but we see vanilla Ansible in the red line taking a total of about a minute and about four and a half megs of traffic to run a task 100 times on a remote host. The same things doing it with Mitogen 90 kilobytes eight seconds. This is our goal. This is why we want to use Mitogen for this. But we would ideally do it without having the trust issues of potentially giving all our remote machines code execution on the box that we are running Ansible and Mitogen from. So for anybody not already familiar with Pickle it's a built-in standard it's a library built into the standard library import Pickle and then has a very simple pair of Dump S and Load S functions. You provide Dump S with an almost arbitrary Python data structure or type or user class if that user class has the support and you get out a byte string. You can send that byte string over a socket you can write it to a file you can write it to a queue or to a cache or to a celery and when you load on the other when you retrieve it and load it again you get back almost exactly what you put in. Pickle has the advantages that it works everywhere Python does it preserves types that the likes of the JSON and YAML don't for instance tuples go in, tuples go out date times go in, date times come out so it makes it very convenient very tempting to use Pickle for remote procedure called systems such as this. It also handles recursion as a bonus we're not specifically using that but it's something nice to have and the serialisations are quite compact typically a Pickle of some nested dictionary will be less than half or maybe a quarter of the size of the equivalent JSON. However, I want to reiterate the standard advice don't use Pickle it is by default dangerous if you unpickle data from an untrusted source you risk blowing your foot off at the hip. This is the standard advice it's included in the Python documentation it has been reiterated by various blog posts various white papers that you will find around the internet. An example of how easy, how trivial it is to get remote code execution through Pickle through unpickling is in fact I've lifted this straight from the standard library what's happening here is somebody centres a Pickle which contains a call to os.system and in this case it's echoing hello world but it could equally have executed RM-RF and if you're running as root you've lost everything otherwise it could have posted your SSH keys or your top secret formula for Coca-Cola or finger-licking good chicken to your adversaries and goodbye your business. Just to give a quick example of this my colleague Rick has centres sent me a Pickle to try let's load Rick's Pickle I'm guessing it's some sort of joke dancing ASCII something I don't know no wasn't expecting that see a Pickle can do anything Java sorry anything Python can if there is a module with a callable thing then Pickle can invoke it and that includes the web browser module so if we have a look at the contents of Rick's Pickle then it's very similar in form to the echo example we saw earlier but in this case we're using the web browser module so what can we do about this? the standard mitigation is to subclass the unpickler class we create a whitelist of Python built-ins Python functions that we are willing to allow Pickle to invoke and we only let things on those whitelists be invoked when we unpickle this works to an extent a lot of the Python standard library has not been audited has not been checked for ways that if you put certain things on your whitelist they won't be able to drill out through those use dunderdicks, dunder, get and possibly reach built-ins once they reach built-ins they can do just about anything the other trouble with this technique is that it is opt-in by default if you have other people dealing with Pickles in your code base who aren't aware of you must use the restricted unpickler subclass nevercallpickle.loads directly you still open yourself to getting Rick rolled or worse and assuming we fix this there are other attacks that most articles, most presentations about Pickle don't go into because frankly getting remote code execution is job done as far as adversarial people are concerned if they can get code execution then nothing else matters but if we if we can come up with something that doesn't allow code execution by default doesn't require extra means to whitelist things then what other attacks remain? the ones that we found so far are fall under denial of service and the possibility of data exposure and weird machines so first example I mentioned earlier that or I showed earlier that Pickles can be pure ascii that's because this Pickle in question is what's called a protocol zero Pickle so we've got new line delimited strings and no binary data but equally we in Python 3 default to higher protocols so if I import Pickle and dump a a string it will default on Python 3 to protocol 3 which is a binary protocol the string you see there is length prefixed that makes it much faster to unpickle with this protocol and the protocol that you use is determined when you create the Pickle not when you unpickle so if I want to be backward compatible with Python 2.2 because who doesn't I can create a very similar Pickle but this time it is using the new line delimited and this protocol zero is much much slower to unpickle and the attacker chooses the protocol so benchmarking this we see that the worst case is unpickling integers if your attacker wants to send you a one megabyte Pickle that's just a very very very large int it'll take you the better part of 10 seconds to unpickle that at protocol zero at protocol one it's quite a bit better it'll take you under a millisecond to unpickle this so we would like to be able to just not allow protocol zero because protocol one or protocol two is what supports the binary encoding and that will get us all the way back to certainly Python 2.7 Python 2.4 I believe also supports it once we've got protocol zero out the way we then have the problem that our attackers can send us self-referencing pickles this is a very small payload similar to a zip bomb if anyone has come across those that just repeats the same element again and again and again and again but by referencing it and by references of references of references we can make a very small payload that expands to an extremely large object in memory or on disk now pickle handle this surprisingly well so I have a billion laughs payload that I've already prepared this is a pickle two payload and it's a couple of hundred bytes long let's unpickle it the problem comes when you try and send this on if you're if you want to log this to disk or you've got another system that accepts JSON and you try and encode this to JSON the problem will be that as soon as you try and serialize the result in anything that doesn't support self-reference then well I've just locked up b python the problem there is that b python is trying to syntax highlight the output and the output is I can't remember if it's several tens of megabytes or hundreds of megabytes in size we can't just consider ourselves when we are using these so a further mitigation you may want to do if you are having to deal with untrusted pickles is to find some way to remove self-reference although there are possibly use cases where you would want to keep it excuse me one second while I kill this so what else can we the other problem is of which we are aware is that there are a much larger number of pickle payloads that the pickle module will accept unpickling than the pickle module will generate so our attackers don't have the restriction that they need to restrict themselves to what the pickle module will output they can they can handcraft pickles and in fact that's how the OS dot system example and the web browser example were generated earlier we need to be able to test our thing whatever it ends up being against handcrafted and also randomly generated pickles in order to try and prevent effects that were never thought of when the pickle module was coded to that extent we've been trying to fuzz test the pickle module with limited success so far but we'll be re re-looking at that in the future final thing is that I want to reiterate is that the use case we have here is a pickle for RPC results so the pickles are living very short lives on the wire they are not being put into long term storage and they are not living longer than the processes that created or will be reading them have lived in addition to any security considerations about pickle there is a very important thing put well here that if you are using pickle and writing the results to disk or to a database or anything where the pickle you created could be read days or years later you will run into problems you should not use pickle for long term storage regardless of the security so the question becomes should you use pickle? the answer is still no but perhaps in the near future you might like to use the module that I am working on called pickle light pickle light is essentially taking the pickle module and cutting bits out we are giving it a lobotomy so at the moment this is still very much a prototype but for instance if we try and generate a protocol zero pickle which might be a vector for denial of service attacks denied oh wait I did, thank you live demo gods so any attempt to create or to load a protocol zero pickle such as wrong window any attempt to load a protocol zero pickle fails so in this case we haven't been protected against the we haven't been protected against Rick's joke here because we whitelisted anything instead we have been protected because he happened to use protocol zero but likewise if we try and set object which normally would require this load a global object and call it effect then pickle three will fail because we've excised the bit that the Rick role relies on instead we need at the moment to use a very little used feature of pickle light, sorry of pickle where we register extensions in this case we're going to register built-ins set and we're going to pick an arbitrary integer that one might be in a reserved range I don't remember but no one is using these at the moment and we can choose better ones later what that is now done is allow us to dump dump and loads objects that we have explicitly whitelisted and we have had to explicitly whitelist them this means that there is no there is no opt out of arbitrary code execution there is only opt in to selected objects definitely too much live coding in this anyone spot any typos? thank you there we go so we have managed to round trip an object that is not natively supported by pickle without opening the door to arbitrary code execution thank you very much for your time everybody slides will be available at the URL you see there very shortly if you'd like to know more about Mitogen please visit the URL there there is an additional project that is working on sanitising pickles called Picara and there will be a presentation about that at Pi Ohio this weekend which I will be eagerly seeking out if you'd like to know more about pickle attacks and what you can do when objects are whitelisted there was a very good presentation by the company sense post in 2011 and if you are looking for an alternative for pickle for long term storage then there is an excellent blog post by a lady named Evie and I recommend checking out that thank you very much