 It is absolutely my pleasure to introduce to you from the Council of Nine. Sam Erb. Hello. So this is going to be a bit of a different talk. It's definitely more blue team than most of what's normally presented at Defcon. So apologize for that in advance, but it's something that I really wanted to give a presentation on. I think it's important to share with the wider community what we're doing at Akamai. So with that in mind, this is Defense in Depth, the path to Shiax and Akamai. So who am I? I am a software engineer at Akamai. At Defcon, I have won two black badges in the badge challenge with the Council of Nine. Yeah, if you want to reach me, probably the best way right now is to find me on Twitter. So disclaimer, generic disclaimer, don't poke and prod at Akamai please. And everything presented here is the work of many individuals, some of whom are actually in the room today, which is exciting. This wasn't just me. So get started, Defense in Depth. I'm talking about Defense in Depth, and what I'm really talking about is the classical computing term, Defense in Depth, where we're adding additional security controls or layers to mitigate or lower the impact of vulnerability in our product. What I'm talking about my definition though, what I want to do is I want to avoid the next sales Heartbleed at Akamai and build systems that prevent this or otherwise mitigate the impact of that. So I'm going to ask the question, what is Heartbleed? So OpenSSL users were affected by Heartbleed. We have a blog post on how Akamai was impacted by this. In general, this doesn't turn out very well on the slide there, but this is a wire shark output that shows a bad pack payload length, and what really Heartbleed was, was you would request a certain length, and you'd say, hey, like, give me five bytes, and it would return five bytes back to you. And it was supposed to be the same five bytes that you sent to it. That's the RFC 6520, which actually defines this, where the receiver must send the corresponding Heartbleed response back to the requester. The implementers neglected to verify that the payload length matches the length in the field itself and would send you back arbitrary memory from the server itself. So this allowed you to read memory from the server and then subsequently potentially get access to the TLS private keys, which we're seeing on that server, or other information for that matter. Just go louder. So what is TLS? So TLS is HTTPS. If you haven't had a chance to, I would really recommend just going to read in the RFC. It's super approachable as far as RFCs go. This is straight out of the TLS 1.2 RFC, and what I'm really going to be talking about today more than anything else is the server key exchange message. So the way TLS works in general is you have a hollow exchange, and then you have a blue there, and then in yellow you have this key exchange period. And then in red is where you're actually sending your bulk crypto data. And I'm really going to be zeroing in on the key exchange, and in particular the server key exchange. And what that does is proves that the server owns a copy of the certificate by signing certain parameters that get used with the certificate's private key. So the server itself has to somehow have access to the private key, which traditionally has meant that it's just sitting in memory on the server, which is why Harplead was so dangerous. TLS 1.3 makes some slight modifications to this, but that's also something that's worth going and actually reading the RFC. So with that in mind, and getting all that out of the way, I'm going to be walking us through a TLS security journey. It's not the most exciting name, but our first stop is we're going to make secrets harder to find in memory. And to accomplish this, we're actually going to use OpenSSL itself. And in particular, we're going to use one of OpenSSL, one of Occamized Contributions to OpenSSL, which is the OpenSSL secure heap. And this is just standard Linux memory protection features to get the keys into a more protected area of memory and to prevent them from appearing in core dumps, et cetera. And this also applies to other key material, which can be used to derive the private keys as well. I believe the function name that you're going to need to look for is OpenSSL secure malloc. It's also externally visible or externally exposed so that your application can actually make use of this as well. So, yeah. So our second stop is we're going to make the, we're going to move the secrets entirely out of process. The reason we're doing this is that when you could dump the memory of the process, you could potentially dump the secrets. So if they're not in the process memory, you can't access the secrets. And to do that, we're actually going to use the OpenSSL engine interface. I posted some example code here. This is a OpenSSL RSA engine. And really the important thing here is that RSA underscore method. And what that will do is it will pass the RSA, or what needs to be signed by the server into a function. And once it's within that function, it's really up to you to make the signature happen or return it back to OpenSSL. And so this gives you a hook into OpenSSL to accomplish this. And so using this hook, we're able to go outside of the process, you know, using either shared memory or some other inter-process channel to get to the TLS private key that we need. So what this actually looks like in terms of the protocol, I'm defining a web server here and that special RSA sign function that I defined previously. I'm going to use that to send client random server random and key parameters as well as a certificate identifier off to some external key process. And this external key process is actually going to make the signature and then return it back to the process itself. So if we had had this in place, when Heartbleed rolled around, we wouldn't have been as vulnerable at least. The attacker still would have got access to private memory, which could have contained sensitive information, but at least we wouldn't have had to potentially rotate this key. So what we did here was nothing, or what we did here was nothing particularly special, so we decided to take this to the next step and we moved the secrets, not only out of process, but also out of the machine. This means that if you had got a, if you have some sort of bad exploit, like a remote code execution, you have some confidence that this key wasn't leaked. And so what that looks like is we just put the internet between our web server and the external key server. The protocol is identical, only you probably want to add some additional security to this channel. And so for some background here, Akamai has, I think we're say over 200,000 servers at this point, distributed internationally. We have a lot of these web servers. So that also means, that implies that we're going to need a lot of these external key servers. And obviously we don't want slow handshakes. This only impacts the initial TLS handshake, subsequent ones will use session keys, which allow for faster session resumption. And also means you don't have to have access to the server private key as well. But this puts this problem squarely into the engineering space and not into security space. This is strictly an engineering problem and it's also a performance security trade-off here, at least on this initial TLS handshake. Yeah, it really creates a many, many writing problem, something that Akamai is luckily good at solving. We solve that every day with our end-users. So in this case, we're trying that problem on its head and we're having web servers connect to these external key servers. So this involves a lot of planning. So this, one of my colleagues put this together, I believe it was Richard Wiley. This is a map, a heat map that shows connectivity between different parts of the world. It was really fascinating to learn about how poorly connected parts of the world are while doing this. So in general, you don't want to be crossing between not just national, but in some cases, continent boundaries. And that will lead to slow connectivity, slow handshakes, and slow performance. And this is just a visualization of this. We actually use the raw data to figure out where to best put these key servers. And this was actually included in our State of the Internet report in 2017. So in practice, this is actually an internal visualization of our network. It's a little bit hard to see from further back, but the blue dots on there represent locations. We have web servers, and the red dots represent locations that we actually put key servers, I should say, because each dot represents more than one machine. And then the yellow and green lines just represent traffic levels, and this is actually a real-time dashboard that I pulled this off of. So we're actually visualizing our web servers connecting to our key processors in real-time at scale. And Europe is honestly the most exciting one. It has the most lines as closely bunched together, simply due to geographical constraints. Yeah, I've always found the visualization extremely cool. So that's great that we've moved our servers now to another machine. Let's get a bit of a paranoid, and we're going to put them into an HSM. For those of you not familiar with an HSM, this would be something like a UB key, for example, where it's a hardware device and hardware security module, I believe that's what it stands for. It's a hardware device that holds your private key and you're able to send it a request to use that private key, and it will give you back a signed response of some form. And so to accomplish this, I'm going to use Intel SGX. And I'm going to explain a lot more about what Intel SGX actually is in subsequent slides. So this is a little bit paranoid, honestly. This was kind of taking us to the next level as an experiment, but we were actually able to make this work and deploy it, which is very exciting. So we put an SGX key store into each of these external key servers. So what is SGX? So this is an Intel technology. We have it in our certain chips that we deploy in our network. It's not available in all Intel chips yet. I think they're working on that still. And most importantly, it's a new set of CPU instructions for handling enclaves. So enclaves can be thought of as a shared object library, where in the untrusted library, you make a function call and it drops you into this trusted state where you can't really do a whole lot, to be honest. You can do memory operations. But you can't ever leave the chip. And anything that leaves the chip itself is either going to be encrypted or you're not going to have access to. And so what that means is that you can only drop into the state and come back out of the state, which on a first pass doesn't really seem very useful. But you can securely get data into this, which means it can function as an HSM if you code it correctly. So it can be best thought of as a reverse sandbox. Enclaves can read and write to external memory of a process, not to any external memory, just of a single process. And applications at any level cannot read or write memory within that enclave. So it's kind of when you actually call the enclave, it really will then reach out back to you, fetch the information it needs, do something with it, and then return that data back to you. And one of the great things about this, about SGX, it has roughly the same performance as the CPU itself. It's running on the same hardware. The hardware just simply drops into this SGX state and comes back out of it. So that's why I think about this. I'm not the first one to use that term. It was originally termed coined out at 2016 Black Hat Talk. So enclaves, these are the shared object libraries that I had in a previous slide. Enclaves must be signed. And the root of trust is actually traced back to your BIOS. So if you control your BIOS, you control who can run enclaves on your system. And it really roots the chains of root of trust to a very low level. Enclaves have to be statically compiled, all of the code is signed. So that kind of implies that SG statically compiled. It requires BIOS and kernel support just to make sure everything works correctly. And remote localization are possible. And this is one of the cooler features of SGX and is what allows you to securely get your secrets into these enclaves and trust that they're going to not only the right machine but the right enclave itself. So there are a bunch of new instructions and most of them aren't really that exciting. They're for memory operations and launching and stopping enclaves. The important two are going to be eReport and you get to key. eReport allows you to tell another enclave on the system that you're on the same system as it and you generate a report and you sign it in a special way and then another enclave can take that data and verify it even if it passed through an untrusted boundary. eGetKey, which is a little bit more interesting allows you to actually generate a unique key within a given enclave. And this allows for sealing, which I'll talk about a little bit more later. But the important thing here is that you can generate a unique key on that machine and then subsequently decrypt data based on that key. I should say that that is actually a symmetric key. So SGX sealing so that eGetKey parameter allows you to generate a key that's shared between all enclaves that are on the same system and signed by the same key. And this key is persistent so as long as you call this function the same way, you'll always get the same key back from it which is awesome between OS installs you don't have to maintain any state on that machine. You can wipe it completely and the key will still come back the same way the next time. They also have counters in there to handle if their vulnerability is found in the process there are counters in there that will increment, auto increment that will allow you to create certain actions based on the key. That's a little bit too much to tell honestly. So we can actually use this SGX sealing system and per machine what we ended up doing was we created a public private key pair and within an enclave and then we had the private key encrypted using the sealing key and then once it's encrypted using the sealing key we can then export this public key sealed private key and that becomes a unique public private key pair tied to that machine that will survive OS installs. And so we actually were able to use this to do key provisioning at scale. This is across a large number of machines where we had each one of these SGX key stores issue this sealed private key public key combination. We would then, we have a massive advantage because we control our machine deployment and our pipeline to get machines deployed. And so we were able to export that key we then were able to encrypt a encrypted key to that public key and then send that and a bunch of other data back down to the enclave and assuming that this whole process works only that enclave will actually be able to decrypt the secrets that are encrypted to that public key. I should say this key pair is an RSA key pair which allows you to encrypt things to the public key such that they can only be decrypted using the private key. Yeah, so at a high level that's kind of how that all works. So there are a bunch of complexity with this SGX that I didn't get into. You will spend all of your time worrying about how to rotate every single key you create. Initial trust establishment is a little complicated which I described previously especially if you don't control your machine build process or pipeline. Secure handling of multiple enclaves, keys and most importantly replay attack prevention are all things that I didn't talk about but you should worry about if you're actually deploying this. You will spend most of your time trying to figure out how this should work and handle key rotation and all these really complex cases and not a whole lot of time writing code which to some extent is a good thing. So I should add all of this was deployed within Akamai. If you connect to Akamai, there's a chance that your TLS handshake is actually at some point actually traversing a SGX enclave which I think is pretty cool. So yeah, thank you.