 Okay, cool. So real quick, if anybody has a laptop out right now, you guys should copy that fingerprint down and sign it because we need signatures on the release key for secure drop. So, hooray bonus points if you can sign that by the end of the talk. So, yeah, I'm one of the maintainers of secure drop and this talk is going to be discussing how we improve secure drops security architecture. And the focus is going to be on explaining it in a way that will allow you to apply some of these other things other projects and not just, you know, laser focusing on the particular issues we face that maybe nobody else faces. So the very high level actually hands up if you know what secure drop is. Okay, a few people. Okay, so for those who don't the high level is secure drop is an open source whistleblower submission system that media organizations can use to securely accept documents from and communicate with anonymous sources. And this is something that we would stick in the newsroom of news organizations to allow them to keep the sources anonymous because do things like just not say the name of our source. We could say we don't know and just, you know, deny it until we get left alone. But, you know, we have the problem now of global mass surveillance where they'd like to say nothing is beyond our reach but and to some extent that is true they can watch the networks they can watch phone communications and whatnot. So, in order to restore the capability of news organizations journalists to be able to keep their sources anonymous we need technological solutions until we can, you know, get all those government organizations back in check. Right now it's deployed with a bunch of major news organizations. This is a handful. There's a whole bunch more, but that's not what this talk is about. So our threat model is explicitly groups like NSA, GHCQ and the rest of the five eyes. We assume that the kind of people who would want to be submitting things to secure drop are the kind of people who would be actively targeted by intelligence agencies or local governments and whatnot. So we have a very paranoid threat model we work with. The two main things we're trying to protect specifically number one is source anonymity. That's the number one thing that we would like to retain secret at the end of the day. And the second one which is very closely related is document confidentiality. So if somebody gets the documents, the source leaks they could use those to de-anonymize them even if the source is communicating over Tor. So these two things are really what we focus on during development. And the question again, who do I protect it from? Well, in reality it's actually everyone, but really nation stage corporations and local law enforcement. And we know that these kind of groups are actively targeting journalists. You can see press freedom violations around the world. There's actually a press freedom tracker that does tracks and these things. So we do know that this is happening all over all the time. And if we know those are the people who are the threat actors when we're developing this, we say, okay, well, what can these people do? Actually kind of can do a lot of stuff, but really intercepting network traffic is a huge one. If somebody can watch network traffic, they can see, you know, 800 megabytes of files move. And then there's 800 file megabyte files sitting on the server. We can kind of correlate that and say that's a lot of Tor traffic. We have a good idea who that probably was. They can maybe hack into the servers, maybe not, but it's a good chance that in a very long run, somebody may hack into one of the servers. And we also see things where they will send agents to physically seize hardware. And in this context, what we do not protect against is where somebody breaks into the office, sticks in a bad USB and malware. We're talking about an agent as part of like a subpoena picking up the server and just taking the server and running off with it. And this could include doing cold boot attacks where they, you know, drop the memory into something cold and then reboot it later and read the memory out. So we do care about the server's memory being pretty clean. And the last one that we've worked on a lot is people submitting malware to users of SecureDrop through SecureDrop itself. Because, of course, they're going to open the document on one of the workstations and kind of guaranteed phishing emails, you know, maybe not going to open it. It looks suspicious, but somebody says, hey, I have this, you know, great PDF about, you know, some illegal operation. You're going to open it. Okay, so another thing with this complicated threat model is it is infinitely complex. Like we could just go down any one rabbit hole until our eyes bleed. We could talk about any one little thing forever. And we're going to try to focus today on just a few of the bigger changes we made lately and some of the bigger screw ups. So the current state of things is we have one application server that runs all of our code that is mostly what SecureDrop is. And that is a GR security patch kernel to help minimize the types of attacks you can do using memory corruption. We run a Python application. We store our data in a very small SQLite database and we run all this behind Apache. And on top of the application server, we have a firewall that blocks all inbound traffic. So this server is only accessible through its outbound connections through Tor hidden services, including for administration and for sources and journalists using it. This again is monitored by a monitor server using OS sec to try to look for intrusions. And lastly, this is connected to sources and journalists both connected all this through Tor and run for the same way. And then a journalist will retrieve documents from the journalist interface and take them off to an air gap to laptop to decrypt them. So that's basically how it works. Our development is actually very simple. We write a feature. We write tests. We write unit tests. We do functional tests with Selenium. We write multi-stage tests to test deployment and provisioning with molecule. We write a bunch of documentation to help sources and admins and journalists and developers. We do mandatory code reviews for everything that gets merged by at least two members of teams. So the person submitting it and the person signing off on it. We automate testing. We do linting. This is all pretty normal stuff for development. So this isn't actually that special of a thing that SecureDrop does. But what's interesting is that at every step we actually put a lot of effort into all these things that we know that we can run end to end a complete submission set through the browser at every single pull request, every single check-in. And we do have to have some automated testing or manual testing because we're using a bunch of different USBs plugged into a bunch of physical hardware. But in the end, this isn't that interesting of a development process. But what we're trying to do is keep each of those things as high functioning of a level as possible during the whole development. That said, it's kind of like fighting little fires all the time. So as software developers, there's an update to something on the operating system and now you have to patch it because this library is no longer compatible with this library or the operating system becomes EOLed. Every couple of years you have to change your operating system. So that slide before about our development process, like keeping that so that it's very easy to go through the entire process each time really helps us with actually developing secure features because we're not getting bogged down saying, okay, we had to update one Python library and we spent nine hours debugging what broke. Now, we try to really say that library change should be quick. We just push off the CI and half an hour later we get a green light or a red light if that feature worked or not. So, you know, I forgot what I was going with this slide. There's something about like things break, but I don't remember where I was going to pretend the poop emoji is not there. But things do break and so since this talk is about improving the architecture, we're going to actually focus on the things that haven't gone well in hopes that this might help you guys in future projects not make some of the design decisions or mistakes that we've made in the past. So one of them was this is a tweet from Kevin Polson and over the summer or early fall, he got a submission to secure drop that when he untart it, it turned into a desktop file. Nautilus executed the desktop file. The desktop file ran arbitrary Python scrapes the GPG keys and then posted a signed message back to some server somewhere. And he posted this to Twitter, which is how we all found out about it. And we had to then go through and figure out what happened. And so really the root cause was that Nautilus allows an executable file with the, allows a file with the executable bit set to run any arbitrary code. And this allowed the attacker to run Python that scrapes the keys, put them in a document with a QR code and that QR code when somebody scanned it, ping the server with the data from the air gap. And so this is a very complicated attack, but this does happen. And so what really went wrong though is that the secure viewing station isn't a true air gap. We're allowing people to move USBs off of it and move data off of it. So we do know that at some point data will be leaving it. But we didn't tell journalists, by the way, be very careful about what you're taking off of it because that actually might be a trick. That's roughly equally as hard of a problem as telling people not to get phished. Like if anybody here is a security person, no. Well, your users are going to get phished all the time if you work for a big company and there's basically nothing you can do to realistically train every user to not be phished. Like it's inevitable. And so in that case, like training users not to scan certain QR codes, like it's just a losing battle. And so the issue we have with this is imperfect isolation. So we allowed arbitrary code to execute in the same environment that contained all of our GPG keys. And so the future development of this would be finding a way to sandbox the decryption code and the viewing of document code from where the actual keys are stored. And like this is something we've kind of known that that workstation doesn't have amazing isolation. It's pretty good and it's hard to get information off it, but stuff does happen. So like if this is the current workflow where you have like the source connects via Tor, the secure drop, the journalist connects to Tor again or connects to secure drop via Tor and then moves these USB sticks around. It's a lot of manual work and it's a little bit frustrating to work with if you're constantly having to like move encrypted USBs back and forth between devices. So our future design decision that we think will decrease like the manual processes that journalists and admins could make mistakes on and also increase the isolation of our system is moving to cubes to have the decryption and the document viewing and the keys all stored in separate systems so that we're not running the risk of random malware files executing in the same VM as the actual decryption keys. So this was brought up earlier in another presentation but loosely like this would be sort of the workflow so that when the journalist connects they would connect through Hoonix to have like their nice Tor circuit to get out to the secure drop application server, pull the stuff down and then each document would be opened in a disposable VM. We could then clean the documents and say okay this PDF has really just been rendered to a whole bunch of screenshots which means we can go forward and hand these PDFs out to the entire rest of the company and know that we didn't just either like malware the whole company or pull information off of the air gap. So this is we were trying to keep things clean so that when documents do leave the secure drop area of a news organization we're not giving infected files to the entire trusted internal network of the company. So another thing that went a little bit wrong is Tor did it made an update and this thing that went wrong was not Tor's fault, it was our fault. We were running a custom app armor set of rules against Tor, the Tor binaries and Tor files like files and when they shipped this it broke the app armor rules and it knocked every single secure drop off of the network simultaneously because they all updated to new Tor and we got kicked off. So update app armor rules and push those out to everybody to fix this and so that's one of the things is that we have a lot of dependencies and every system has a lot of dependencies and the more you can control your dependencies the less likely something will move under your feet and break part of your system and what we've done to fix this actually is we no longer use upstream Tor directly we actually mirror the entire Tor repository to a local from the press foundation control secure app repository and we point our secure drop servers to this new Tor mirror and actually Connor in the back over there is the one who I believe did all this Am I right? Yeah? Yeah. So Connor implemented this and this pull request again that lets us now have a little more control of our system so that external factors like our external dependencies changing still have to go through QA so that we know that we can go end to end on the system we can provision systems we know that the entire process still works before we push this out to some 60 news organizations around the world. The next major usability thing for security that we did is like a UX improvement which was we translated secure drop into a whole bunch of different languages and right now this is the handful there's a lot more done and this is up on the web and we're actively working on transiting more and more of secure drop and this is actually might not sound like a secure architecture thing but from the sense that like the system as a whole does rely a lot on administrators and journalists and sources doing correct behavior at every step along the way like this helps sources you know if we tell them memorize your password don't share your password like only connect with Tor browser if they're doing that in a language they don't really know which was only English for the longest time it increases the chances that they're going to make a mistake and in a sense because we are the stewards of this project it's on us to ensure that their behavior like is nudge along in the correct direction by a good user experience so like people like to talk about like the coolest encryption algorithms they use like for us if we say like if we tell them you know do X and they can't do that one X thing they immediately anonymize themselves that's like that's not any good if we use the sweetest latest crypto so making these good UX changes also does impact the design and turn of the program like there's little small things like every time we have a string we have to rethink about how we actually assemble these strings because you know you can't just drop random nouns the end of a sentence because in some other languages you know you have like inflection and declination and what not and so like this is only a small example but like going through the code there were a whole bunch of places where we did things like pluralize things by string concatenation by adding plus S to the end and it doesn't really work when you're internationalizing so on top of that there was a whole bunch of changes we had to make to the development process and one of them was developing a workflow that allows us to like submit all these strings out to somewhere where we can then solicit translations from the community building these back in the application and then also knowing that any feature that requires a string change we have to go through a longer process or have a like a longer timetable because we do require outside contributors to do these especially as the number of languages grow so you know making a quick patch into a release to quickly get something out sometimes can't happen anymore so we try to think about how we're going to develop in a way that gets these fixes in and allows people to ship things out without you know constantly saying oh we broke a string, we broke a string, we broke a string and then you know bothering the community another thing on the UX side is this is the old landing page and it's a little hard to read at the top in this font but it says we recommend using Tor browser access secure drop learn how to install it or ignore this warning to continue and this warning is flashed via CSS if the or no view the user agent in the HTML when somebody like hits the page for the first time and so we did some user testing and I say we as in the engineers which is to say we are probably not qualified to be doing user testing and they said oh that sounds scary like I did something wrong I'm just fizzing this for the first time so after a little bit of talking to people we switch this to being purple say okay purple is a less scary color now when users see this they won't think they did something wrong and then the downside was immediately I'm going to ignore this because it's just part of the design it looks like a purple banner and nobody listened to this anymore which again says we're engineers maybe we shouldn't be doing this sort of thing we should go out and we should find people experts in the field of user experience and user interfaces and have them really do an audit of the app say okay us as engineers we think we had this nice secure pattern but do we? what are the hidden things that are going wrong maybe this text looks pretty normal to us but to the average user it turns them away it encourages them to do bad behavior so again secure architecture might mean a lot of things but it also does include the user experience side of this another related architecture thing was we had at least one dedicated admin who would again get emails every time there was an OS sec alert and they were getting fatigued by this we don't know exactly how they were but we know that it was a complaint that there's too many emails and if somebody's getting an email every 30 minutes that says like oh this happened and none of it is actionable from a security perspective we're going to wear them down so that the day that there actually is an intrusion they just delete it with their other things or they set up a filter so that there's one inbox with 10,000 unread OS sec messages they're going to miss it and so we had to go through and really work on our rules for OS sec to say okay cool this is not necessary to alert we should try to avoid this and we're trying to shrink this down as a team so that people who are doing the day-to-day operations of secure drop aren't being overburdened and then being incapacitated in their ability to be effective admins so those are some things we've worked on in just like the past I don't know eight months or so and still working on some of these things right now but one of the big changes of the future is moving away from SQLite to Postgres so this kind of falls under the principle of least privilege we have applications that are when they interact with SQLite they can just do anything to it that means the source interface can read journalist, passwords, and user names which is there's no reason to do that and that's the limitation of SQLite right now so we have to we're going to move to a bigger database that has better access controls and it would also be interesting in the future because if we do want to do something like three application servers that all sit behind onion balance they would need to share a database and you don't want to do that on a file system so you didn't want Postgres available via PCP sitting behind the firewall and then you can have many application servers interacting with one database server oh that slide is not done anyways, so two of the major things we've done recently were we also had to get to the point where we could even add Postgres we didn't first had to add Alimbic which is the Python library and command line tool for applying database migrations and before we could do that we also depended on Flask SQL Alchemy which was not implemented in the original Flask application we also have to do a pretty large refactor of our tests and before we could even do either of those two things we'd also do a large refactor of the source and journalist applications and so even to make this one change of preventing the source application from being able to read the data for maybe like the password hashes and usernames of journalists like it's in many steps to get away from this and so it's not actually terribly interesting of a task like this is what I do day to day and it's actually quite boring but like that's a really important thing with secure software is there's just like tons and tons and tons of tedious work you need to do every step along the way a well-formatted application that's well modular that allows you to quickly add tests and verify that the behavior is correct so this isn't a fun presentation slide there's crappy work you have to do to make secure software sometimes and that's kind of part of being an engineer like talking to Connor last night about this and his analogy was that a big old skyscraper is really cool it's pouring concrete and pouring concrete is a pretty boring task so like us as software developers need to do more concrete pouring another hopefully improvement is right now we have started to dockerize the development environment to make it a little bit easier for everybody to set things up which isn't strictly a security fix at the moment but we're potentially moving from the current app server design which is just everything piled onto the application server and running all in the same user space into this is also up for debate so anybody on the team who doesn't like this idea you know you can yell at me later but this is something we're considering doing which is splitting off the different parts of the applications into their own containers and then restricting who can talk to what so the source application doesn't actually need to talk to the internet ever it needs to listen on or make itself available to nginx or the source application also doesn't have any reason to need to talk to even be able to like hit the IP addresses of the Postgres database and so by kind of removing things we get the benefit of separation of responsibilities but also we know that in the containers we can be a little more creative with the dependencies that we need across the different ones and not worry about system dependency collisions or things pinning versions and holding us back from updates we need because of dependencies and it's not a huge issue but it is a consideration that we can then control a little bit more how each of the different pieces fit together and on top of those few advancements that we're looking at which are not massive but somewhat big one thing that came up this summer or fall was there was a research paper about fingerprinting onion services which was looking at if you sit between the user and the guard node or you are a bad guard node and you watch tour traffic go in and you say okay cool I can see an encrypted stream coming in can I guess what onion service that is and can I guess what they were doing and it turns out that yes you can in a lot of cases use because there's not that many onion services out there in the world and there's not that many that get a lot of traffic so if you're trying to make a guess who's using time's up what okay anyways this is something that is an interesting research question but we don't have enough bandwidth to be able to effectively deploy mitigations against this because we have to write mitigations then test them and verify that you can't fingerprint secure drop as an onion service after these have been applied so if anybody is a machine learning expert and wants to help us or if any of these people in the audience you know come talk to us afterwards and we'll have a chat I guess maybe time for an honor to your questions cool yeah this is picture of this if you want to come join us and yeah yeah so when for a long time was kind of oh yeah so the question was do we see different threats across different languages and across different regions and for a long time one of the things on secure drop was that it was assumed that people would know like the NSA would know that some news organization has secure drop running in their office at this location and we were relying on the US legal system to protect the servers but running say secure drop in a country with a hostile government might be different so like in that sense like using Tor becomes more risky like then you want maybe your onion services to actually be more anonymous and the threat model does change as you move from around different news organizations around the world okay do we have what time is it do we have time for more questions maybe one more question anybody else can you show again the GPG yeah so oh yeah that's actually going so this is the GPG key at the beginning if anybody could take a picture oh my god so many animations cool this one if you guys could sign that one that would help us ensure the integrity of this admins use that when they install secure drop so if you want to help admins correctly verify that they have the real authentic secure drop software signing this would help us cool one more question yeah I just wanted to ask how many languages have you got your documentation in at the moment the documentation is only in English because it's pretty extensive and we don't have a plan at the moment to implement that we have only implemented the documentation on the source and journalist interfaces the translation yes yeah so I you can talk to us after that that one's a complicated question because we don't know this internationalization thing which is actually the work of Loic over here who's did pretty much all of but she asked about the documentation yeah yeah cool do we have more time maybe who is the next presenter here next presenter is not here maybe more questions cool in the grey pardon my English I just wanted to know if you have plan to integrate like provider to avoid stenographic attack like when the source is publishing pdf document maybe the hidden fingerprint in it and yeah so I think another some things like tails comes with the metadata which will strip out some metadata but if you do something like you have like printer dots or there's like some way that images are suddenly manipulated when they download them from a server like a very small watermark on them like we don't know about that so that would fall more under the training of journalists about how to redact information or release documents selectively and how to assess that themselves it kind of falls a little bit outside the scope of the secure drop software but it is relevant and I don't have the best answer for that one is the next presenter here for the next talk okay yeah just rotate cool thanks everyone