 Hi The Harriet has some technical issues. His machine keeps crashing. So I'm harrowing myself. My name is Falkon and I Work a lot at CCC and one of the things I work at is the members desk and Maintaining the data register So we of course want to communicate with our members With the encryption when we communicate and so I want to integrate PGP into That system and today I'm going to talk about how I do that with Python Or not Okay, so I gave you the context If you are interested How the member management actually is working and what problems there are Why don't you go to media CCD and Look for Falkon I have a couple of talks about how this entire system works Let's start at the beginning We have a member management application written in Python and as naive as I am I just Was want to use something like PGP to send encrypted emails So I'll start with pit what Python packages are there that support GPG and You get some results and you look at them and Some are better than others and some are outdated and and maybe there's something that everybody's using Then So at that point my mood got a bit worse So I'll take you along on a short tour on what things there are Let's start Let's start with Python new PG An author That was a Python binding that Was very popular But it's a really simple wrapper Around the GPG executable internally It simply runs GPG with a sub process be opened add the command line parameters and run it and I added the shell equals two because this is one of the big problems about this this package has a lot of Security issues with a shell exact injection. I I can't really tell you the exact Patterns, but you you you add a couple of interesting things to the data you want decrypted and Bingo, it's executed pipe GPG So what does the interface look likes? Is the calling the command line interface is super in stable because It changes all the time. So for example GPG Let's you use the command list public with with colons gives you a Machine readable version of the data and the colons here are the delimiters between the various fields Hopefully that you can use In your own program, it's the problem is it's your own format. It's a custom format. You have to write your own parser To read that data. That's only some are great Fun fact if you add a cold on to your username or for a comment they did cover that so it's properly escaped so So and if you use the escape character backslash x3a that is escaped Itself again, and so when you read that you have to take that into account But what happens when the format changes? for example in in GPG version 2010 the format changed That the time stamps were changed so the wrapper Completely fails because it cannot read the time stamps and if it doesn't check for this change explicitly So weird things happen so this module So this was the first module that the members administration worked There were interesting things in there. So for example if a key expired Only an empty return was given but no acceptance wrong and that meant that simply An empty email was sent and if you have been a member In many years ago, you might have seen exactly that happen and you Might have gotten an empty email and we get good responses Oh Why did you send me that empty email? So I got a template to reply to those. Hey, the problem is probably that your key has expired So Very bad Very bad detail at this. So this is not a completely empty email It just looks empty. It has some weird Weisberg's characters in it So you have to strip the white space you could have dealt with that and You could have tried to anticipate that problem by checking the keys or illidity before trying to use it But that's not a good Good program practice. I want that The entire system fails if something goes wrong in the encryption So there's a new package to new PG Which is kind of an update? No, it's not an update. It's it's a re-implementation of the implementation and The suppressors invocations were so horrible with Horribles with security issues and so on and this this is approved. This is Is Is not compatible to the old version to the old package The API the Python API is actually different But it still has the problem that it is a wrapper that calls the GPG command line tool with an unstable interface So what else do we have? Let's look into Outside of pi pi The Koya is a Is it's its own custom open PGP implementation? It's it's not a wrapper. It's a complete replacement for PGP or GPG and Sequoia is written in Rust, which is pretty cool Because rust helps you Avoid many of the common pitfalls like buffer for overflows and so on It helps you completely avoid that and So that's a pretty cool thing It's a pretty performant There is an attack on PGP keys called Signer signature flooding last year there were two keys of two of the GPG developers were spammed basically by Uploading 150,000 keys for each of the two people and That basically meant that you couldn't use with GPG anymore and New PG simply Didn't didn't work anymore. It slowed down so significantly That you couldn't use it anymore because it was trying to use all these 150,000 keys So Sequoia Managed to do the import of these hundred fifty thousand keys in only five seconds. So that sounds cool So let's guess what kind of keys we see in our membership database So would be very cool to use What the Python sequoia Python sequoia bindings status early prototype I looked at it Looks very nice, but there is a lot of stuff missing that we need in our membership management So let's continue looking though Let's look at the people financing sequoia The pretty easy privacy foundation and they have their own product called pet So they use GPG me and Provides their own Python bindings but It is open source But the installation is really hard To say it like that I had You have to Multiple dependencies you have to pre-compile multiple deep dependencies there's no ready-made packages for Debian or similar and Finding the instructions on how to actually do this is really hard to find on on on the website The link to the instructions gives you four four So I finally found the repositories and there's a read me and The read me doesn't tell you which brands the most recent branch and you have to use that other branch They cycle of dependencies With and you have to use the the right command invocation to get out of that and luckily I was I Know one of the developers personally so I could get around all that and I finally had the Python bindings ready to go And it looks very nice. I like the API There's identities you can create that have keys and then you can create a message to that identity and Whenever you make a mistake You immediately get an Ise assertion and the entire Python crashes and That I don't want I just want a regular exception And If the if if the Python crashes I cannot deal with that properly I Need to be able to tell like the user that the recipient. Oh your key has expired and And and to find out what has happened. I have to restart everything with a debugger and If I have to send out 20,000 emails that isn't really practical So maybe somebody can look into that and in and improve that software and So when you start sending that Emails with it. This is the my message. We get it's an email message that looks empty And not sure why that happens But so I decided pep looks very nice, but there are too many issues for me to deal with so GPG me pep is based on pgb and And and those are the official Python bindings of glue PG. Let's try that. So how do you do that? you create a project you create a requirements to exceed and That's where it starts So which GPG me do you use do you use the one from pi pi or pretty much every Linux distribution that Includes GPG also includes a package for the Python bindings Why are they doing that? That's because these Python bindings Have to match the version of a canoe PG. Otherwise it won't work So the problem is So you don't want the Python GPG the Python package from Pi Pi, but you do want to use the operating system installed package so you have to somehow manage to pick up the right package from your operating system into your virtual environment You can do that But this leads to Very Bad problem because I'm using Pippin and so I have to give you a short introduction to Pippin and Pippin is a small tool to create virtual environments In a separate directory Automatically finds it again when you use Pippin commands and you can directly run commands with Pippin in front And you can check with Pippin if one of the dependencies that you have in your list Have any security vulnerabilities and you can tell Pippin to automatically upgrade them So that's very good The only problem with that Pit and Has a bug where quite a number of those versions Cannot take over Is like use this the command line parameters for system side packages and that's where Pippin Pippin explodes So with Pippin you can't use packages Outside of virtual environments So I had to remove Pippin from the equation the bucket bucket fixed and eventually There will be a version of Pippin that can deal with the problem But it takes all a very long time So encrypting things is one thing But I also want to download keys So in the member management We only use a key ID that you can add to a user's record and The membership application actually has a field for the key ID And we use the classic key servers to search for the key downloaded and use that And that was wonderful working well until Hagrid happened Hagrid is a key server based on Sequoia Developed in Rust Deals very well with the signature flooding problem and Can survive the signature spamming and it does quite a few things very nicely for example if you look at the classic key servers they Built in a way that you can only add data so if Somebody gives me their key and I upload their key and It's being added to the server and nobody can remove it So the the owner of the key The only thing they can do is revoke the key But they cannot remove it from the key server and the email address Is publicly available on the key server forever? So Hagrid introduced a way where keys are only added If you confirm by email that you actually want your key added to the server additionally you have to Say that the classic key servers are written in a very obscure outcome dialect and software behind it is a Very clever algorithm to synchronize the key servers between the various instances a PhD thesis program But really hard to maintain because very few people can read the cane Can read the code and very few people actually can provide improvements and Hagrid of course is Well a lot better suited to this situation because of Using Rust and so on and so all the tools are Eventually switching over to The new Server so you can see here all the all the New machines and the new packages all switched over For deep as a default server to the new one so The problem is that we have a lot of people who are using the old key servers and New people are using the new key servers and don't even know that there is a different world with the old key servers So So you have to explain to them Why don't you upload your key to the old one and and they they don't want that because they don't get the confirmation and so there's a lot of trouble trying to get that working and So the easy solution here is we just try to find the key on both the new and the old instances So So that's not easy to do in PGP because There's a single conflict file and you can you can add a number of key servers and GPG only use one of them even if you have configured multiple servers. So if you want to query multiple key servers You have to edit the The conflict file for the agent and GPG and So you have to edit that and then So whenever you want to look up a new key You have to repeat editing the file and creating the key and that's not something you want to do for two thousand males and Especially not in parallel that won't work So not very nice. We're not the only ones having these problems. So So for example this HKP for Pi HKP is the name of the protocol that is used to query keys from the servers and the small module Python module Can talk directly to the key servers and download the key server from the keys from the key servers HEP base protocol and and it's very simple as these servers and and as download the list of this this list of keys and Then you can import them directly into GPG So everything is fine but I Fell flat on my nose there because my GPG bindings doesn't have a key import function I like stable software. I have an old version of few want to and 1804 is Only two years old, but is completely out of date from the viewpoint of GPG and The import version the import function is all it has only been available for less than two years So another thing where we go back to the beginning do I want to upgrade my GPG? Is it really easier? So in july is probably the word in the time when you should switch LTS releases So then you can think about oh why don't I do a source in install of GPG very very simple. It's only about 30 steps and So I wanted to keep it simple and and not too many complex steps and keep it simple GPG me if you do it right You can do it. You can encrypt your emails But there's a lot of stumbling stones pitfalls that you have to get around You can do that you can download the keys You can decrypt emails So and and handle the private key So let's continue looking So we don't just want encrypted Contents we want the email to actually go out as a PGP mime message in the old standard PGP Would PGP GPG would simply attach the encrypted message as in the tension, but The new mime format is much better because email clients can work with that a lot more nicely So there's a lot of modules that you can use for on pi pi PGP mime is one of them you find Quickly So if I understood correctly Looking around at the documentation and the source code the as one protocol is what the GPG me bindings use to talk To GPG the problem is So They managed to remove many of the sub-process calls, but for some it is still necessary and The last commit was in 2012 and It's it's like one student who did that and nothing has happened since So email GPG a Very thin layer and that simply builds up mime messages And doesn't really do any of the GPG bindings So I can't remember the name who did this Rise up I think I Think the people behind rise up put this together It would it was created once 2018 there is no documentation and If you look at the source code In the standard configuration, it doesn't do anything with encryption and the only thing Anything everything you need to do is replace the know-of codec with the proper codec I think it could work well, but I haven't tried that yet It's on my to-do list for one of the next times I look at this then there is GPG mail encrypt It's it's the it's the kitchen sink it does everything It uses sub-process calls It integrates Integrates various scans and spam protection and has its own mail server built in And the idea is that this is a gateway Before your real mail server to do everything so way way too much for my purposes So GPG mine Uses Python glue PG the very old module that has all these problems and I don't actually want to use that so crypto mine the same basically GPG lip lip it becomes interesting here it can use Python glue PG or GPG me and GPG me is Is the Name that the official Python bindings for GPG had and it was renamed at the very last minute and They haven't followed the renaming and the last commit is 2017 and It doesn't look like it would work together with the current GPG At that point I decided that that's not something I will look into in detail So finally I Think I'll just build it myself. Mime is not that hard There's the email mime package from the standard library and Creating a mime email Let's look at how PGV mine is actually set put together You have a multi part mixed message the container So the very first line Defines that and the very last line ends it and in between there is a multi part alternative So with text plain and text HTML So you can send out different variants of the email And can put in an explanation for people who can only read the plain text and Then there is an attachment and we can add multiple attachments and This will always be shown because it's not in the alternative lock If somebody gets an email with Apple Mail With Apple Mail and We have to We have to look at the plain text and Since the attachment is part of the HTML part. It will not be shown So all of this is encrypted with open PGP And then Add that To another mime mail container That looks like this So you can see from Begin PGP message to end PGP mission which is put into a multi part encrypted container so far so good That's what you need to do You have to Use GPG you have to pick the version of GPT somehow get Python bindings up and running and then package it all up in mine and at the end I Recommend that if you Do all of that you should set up you have a test setup where you Basically create encrypted emails Into an email file which you can open for example with Thunderbird So to summarize Take the Python bindings from glue PG version 2012 or newer The official bindings Use the Python bindings Delivered by your operating system so it matches the GPG binary Use hkp4pi for the key server communication and Then somehow build some mime around it This can be really hard It it's really helpful to reread How PGP mime is actually to be implemented so reading the RFC is really helpful A really great ecosystem. It's a lot of fun to work with it And now I'm ready for your questions