 I have a warm DEF CON welcome for Ian Foster and Dylan Avery, or Erie? Erie. Erie. Sorry. Lost and found certificates. So, just as a quick introduction, as mentioned, this is Ian. He's a motorcycle hobbyist and DNS nerd. And this is Dylan, let's go bike enthusiast and cat owner. Did you say cat owner? Yeah. It's true though. So, basically the problem that we're going to be talking about today is, what happens when an SSL certificate from a previous owner is still valid for the next owner of the domain. So, the previous owner gets a real SSL certificate that they're allowed to do for their domain. Domain either expires or transfers ownership. And then the new owner of the domain, they maybe don't realize that the previous owner still has a valid SSL certificate. And prior to 2013, there was really no way whatsoever to get any visibility of this. And really, like when you think about it, it seems kind of silly that this is even an issue. Yes. An example we have here at the bottom, we say Alice registers a domain name in this case food.com for one year, and gets an SSL certificate in this case three years more than the one-year domain name. Our registration food.com is unregistered because Alice doesn't renew it. And then Bob later registers food.com, gets his own SSL certificate, completely unaware that Alice's SSL certificate is still valid for her domain name, and she can encrypt traffic to it. So in order to help try to identify these certificates, we have certificate transparency. This was created as a publicly auditable log of all SSL certificates to help catch misbehaving certificate authorities. It's currently about half a billion certificates and growing. And basically, we can use certificate transparency to identify this issue, which gives us way more visibility than we had before. So basically, if the previous owner has an SSL certificate and it gets logged in CT, we can see it and we know from that that there are still certificates for our domain that we don't have control of. So we look for really notable examples of this, and we were able to find some. One particular example is for Stripe. So if you're not familiar with Stripe, they're a really big online payment processor and a large percentage of transactions that you may have conducted online on e-commerce websites may have actually used Stripe under the hood. What we found was that the previous owner of Stripe.com still had a valid SSL certificate when Stripe bought it. And now the crossover was relatively short, but it didn't have to be. It could have been a five-year cert. It could have been all the way up to today. So we tried to set out to figure out how big is this issue. So we searched for certificate transparency to look for certificates that overlapped multiple domain ownerships or abbreviations. And the hard part of this is trying to figure out when a domain name changed hands. There's no easy way of doing this. The best way we could look at was historical who is, historical name servers, and the way back machine. And we surveyed three million random domain names from the internet and their associated 7.7 million certificates, which makes up just about one percent of the entire internet. It's a fairly small study which should give an idea of what the whole entire internet looks like. And we looked for changes like expression date, email contact, registrar changes to try to determine whether or not a domain name has changed hands or not. And this type of analysis is not perfect. There are both false negatives and false pauses. But it should still give us a good idea of what we're looking at. And we found that in our data set, about 40.45 percent of the domain names had previously seen SSL certificates, which strapped to the internet is about 1.5 million domain names. And of the certificates for these 1.5 million domain names, about 25 percent have not expired and are still valid right now. And so we are calling this problem by gone SSL, which is we are defining as an SSL certificate created before and supersedes its domain's current registration date. This is just another way of saying that the SSL certificate spans multiple domain name registrations. So could this be any worse? So as many of you might know, a certificate can be valid for multiple domain names. In this case, we have a certificate for both Food.com and Bar.com. And a certificate can be good for domain names may have changed ownership and some of the domain names may have not changed ownership. In this example, Bar.com has always had the same owner and Food.com may have changed owners one or more times during the existence of this certificate. So if it's not immediately clear, a lot of CDNs will actually shove a whole bunch of customers on the same cert. And in this kind of extreme case, we found one instance of the CDN putting 700 of their customers on a single certificate. And we've made the decision to blur this out because this is still a valid issue. But if we were to show the domains used on the certificate, I promise everyone in this audience would recognize some of them. And what we found was the one unblurred domain there was actually currently unregistered and expired. And it was available for purchase. So basically we have two options here of whether or not we can revoke these certificates. You know, the question is like should we have the right to revoke these certificates? If you say no, then you can imagine shelling out a whole bunch of money for a new domain, maybe for your new startup. And you find that there's an old SSL certificate in CT that's going to last for another five years. And you just can't do anything about it. You just have to live with that fact. You can imagine a scenario where a bad guy might squat on a whole bunch of desirable looking domains and just selectively only give them out to e-commerce websites so that they could maintain SSL certificates in the middle of that traffic and steal financial data. On the other hand, if we say yes, if you can revoke these certificates, you get these situations where certificates have both bygone domains on them and non-bygone domains on them. A lot of cases are still used in production. And if you give the new owner the right to revoke these certificates, then all of a sudden they have the power to denial of service the websites that are still using that certificate for legitimate reasons. So it's really a lose-lose scenario. So we dug a little deeper and we learned about the CA in browser form. And this is a group of people made up of representatives from both web browsers and certificate authorities who create the baseline requirements for the issuance and management of public certificate certificates, which is just a long document that specifies how CA's and browsers should operate so that the certificate authorities can be trusted by the web browsers. And inside this very long document in section 963, subsection 5, it talks about revocation and the reporting of revocation, and it does specify that. If any information in a certificate becomes inaccurate or incorrect, that the certificate authority must revoke the certificate. So on the previous side, inaccurate, that's kind of an ambiguous term. We found this other section, though, 4.9.1.1, and this one, in my opinion, is a lot more explicit. It basically says if the domain name registrant fails to renew the domain name, then if the certificate authority becomes made aware of that, that certificate authority has to revoke the certificate within 24 hours. Now, we're caveatting us with the fact that we were warned by CAB members that this document is very ambiguous and sometimes self-contradictory. The people that draft this document oftentimes lawyers, not actually engineers. That being said, to the best of our knowledge, we have the right and the power to revoke these certificates and we have the power to do so within 24 hours. So going back to the previous example of the certificate, certificate for food.com, bar.com, where one domain name is changed owners and one is not, this now means that if bar.com is still using the certificate that was available for food.com, bar.com, the new owner of food.com could reach out to bar.com certificate authority and revoke the certificate and cause a download service on them, or at least their users will get an SSL warning. And so looking at cases of this where domains share a certificate with other domains that change ownership, that number of skyrockets or four times increase up to 7 million domains on the internet are potentially affected, about 2%. And of this sample set, about 41% just under half still haven't expired and are still valid and potentially still being used. So I mean at this point Ian and I are realizing we can break stuff, like a lot of stuff. Ian and I had the power to revoke something like 3 million certificates and the CAs had a legal obligation to do so within 24 hours. And most of those certificates, well I shouldn't say most, most of those certificates, we're still being used in production. So it would have broken all kinds of websites, all kind of places. A lot of the websites are definitely ones that you all are familiar with. And so we thought that this basically, there's no real clean fix to this, so we have to just introduce a new class of vulnerability. And there's two flavors to this. The first flavor is the one that we first introduced. It's when you get a new domain and you notice or don't notice, you see in the CT log that the previous owner still happens to have SSL certificate. We're calling that variant the bygone SSL man in the middle variant. And the other type is the type I was just talking about, which is the bygone SSL denial service where you share a certificate and are still using it with an alt name which has expired or been acquired by someone else. And that gives them the power to revoke the certificate that you're using. If they were to do that, that would take down your site. So if you remember back from a couple slides ago, there was that CDN that showed 700 customers on one cert. We now realize that if we bought that domain and reached out to the certificate authority, we could say, hey, the previous domain owner failed to renew. You have to revoke this thing. And we now have the power to revoke it and break 700 customers' websites. Now for our demo, I'm going to be tabbing over to an email client and in real time revoking the cert. Not really. But we totally have the power to do that. So for our prerecorded demo, we have a couple of certs here that we did reach out to the certificate authorities just to see whether or not they would actually revoke them. And so for the first example here on the left for Digisert, I should mention this cert had a subject alt name on it and the subject alt name was using that certificate in production. We reached out. Hey, one of the domains on this certificate belongs to us. Can you please revoke the cert? And the way we framed the email, it seemed like we didn't own the other domain. And within 24 hours, Digisert verified that we did own the one domain, didn't verify that we owned the other domain. They had no problem revoking the cert. So another example here, we reached out to Amazon and I should mention also, like, usually certificates have a little field on them called the CPS contact. And the CPS contact is basically the email contact that you want to reach out when you want to revoke the certificate. That's the contact that starts the 24 hour timer when you reach out to that. So we did that for Digisert. For Amazon, they actually didn't have a CPS contact on their certificate and it's kind of hard to reach out to AWS if you're not like a customer of AWS. So we tried to do this the best we can without actually creating an AWS account. And we found this public email ec2abuse at amazon.com and that's the one we reached out to. It took a little bit of back and forth but initially that got us to the right certificate people. Took a couple of weeks but again, they had no problem revoking the certificate for us. So this is our pre-recorded demo that failed. We reached out to Komodo and the same thing, we reached out to the CPS contact, asked them to revoke the certificate and to this day, they still haven't revoked the certificate. They basically said, you're not the original owner. Like, you didn't generate that certificate. We're not going to pay attention to you at all. And on the bottom there, you can see they actually tried to use it as an upselling opportunity saying, forget about the old certificate. We'll sell you a shiny new one. You just forget about the one that the previous owner still has. So next, we tried the revoke certificate using Let's Encrypt which is a slightly different type of certificate authority. They use the ACME protocol to try to automate as much as they can with no human interaction. And it's currently fused the ACME protocol to try to revoke a certificate. They require you to prove ownership of every domain in the certificate in order to actually have the revocation happen. So we reached out to their CPS contact and asked them about this and they recognized that or specified that this is their current policy that they require you to prove ownership of all the domains in the certificate to revoke and recognize that it is a slight conflict with the CA browser form and that they are currently considering a change of policy to prevent this. There might exist a case where someone has a certificate for a domain name that you might have the domain name for and lost the domain and required it from them. So currently with Let's Encrypt, we were not able to revoke this certificate but it could be possible in the future. And if Let's Encrypt implements that, it's also worth pointing out that will be real-time replication. It won't be like 24 hours while the CA reaches out kind of thing. Because their whole system is real-time automated, if you use Let's Encrypt and one of your domains on your cert becomes bygone, instantly somebody on the internet will be able to revoke your certificate. Very true. This is a tool I made a while ago. It's an open-source intelligence gathering tool that crawls the graph of the certificate alt names. It works by you give it a domain name, it finds all the cell certificates for that domain name, goes through, gets all the alt names of all the certs, gets all the certificates for all those alt names, and continues to grow and grow until you get a complete graph of all the domain names and certificates for a given domain name or property. It was built the idea of domain name enumeration for red team targets but it's very useful trying to find vulnerable domain names for bygone cell denial service as well. This is an example of the graph generated with this tool on the FS website. So, one particular example where we found this was Salesforce. So we ran this in Salesforce.com, it created this nice graph and we ended up at Squarespace.com, which is nothing new to Salesforce. And this was because Salesforce.com used to own do.com, lost that domain name somehow, Squarespace got it, not sure it expired, they transferred, but they kept in the Salesforce.com as a cell certificate, the one that red dot in the center circle of green on the left, that cell certificate was good for do.com and it was good for do.com far after they owned it and after Squarespace had it. So Salesforce had a valid cell certificate for another company at this time. And this gives one Salesforce ability to potentially run the millsquarespace.com and also Squarespace.com could have revoked Salesforce as a cell certificate taking down their site. And if you look on the left side, all those like green encircled domains around that certificate, those are all of the websites that would all of a sudden break if Squarespace decided to revoke this thing within 24 hours. Yeah. So this is digging a little deeper into the do.com example. We can see on the right is the certificate in question. On the top part, we have read the dates that was valid before and after. You can see those between the two different owners and all the domain names that was valid for Salesforce and do.com. And then the left we have historical NS lookups for this domain name. The screenshot was taken after the domain name had been transferred to where it currently is right now, I believe it's like a for sale auction site. Some of us can go by that domain name if they want. But on the circled red part is when Salesforce had the domain name in the past, you can see historical name services being hosted at Microsoft, AWS and a few other places as well. So we wanted to also introduce the tools specifically crafted for finding this. So circraft is a nice visualization tool. We created a new tool that uses the Facebook certificate transparency API to both detect by gone SSL denial of service and by gone SSL man in the middle. Now let's caveat it with the fact that you have to give it accurate information. So to detect by gone SSL man in the middle, you need to give it a complete list of all them or you need to give it the list of all domains you own with an accurate date on when you first registered it. So maybe you renewed it a couple times and the created date on the who is record doesn't actually reflect the first time you created it. You need to make sure that actually reflects the first time you bought the domain and then to detect by gone SSL man in the middle, you give it a list of all domains that you own and it will go through and see if any of those domains share certificates with domains that you don't own. So for this to be accurate, you need to give it a complete list of all domains you own. As long as you do those two things, have an accurate registration date and a complete list of all domains you own, you won't get any false positives and it can very accurately detect both flavors of the problem. The two downsides to using this tool first, you have to create a Facebook developer account and give them a token and the second downside is Facebook actually will rate limit you if you hit it too aggressively. And lastly we created another tool, the by gone SSL certificate transparency log monitor and this actually has a certificate transparency actual log. We used an SSL search bar tool and added in by SSL detection to that so you create a config file for the domain names for the certificates you want to be alerted for, specify a valid update which is the date that you acquired the domain name and then whenever it finds the matching shirt it will print out or notify you the following patient like in the screenshot right here and if it happens to notice that the certificate is valid both before and after the date that you acquired it, it puts this little bike on a SSL flag and those changes have been accepted upstream into SL made search spotters so you can grab that from them if you want to run your own certificate transparency log monitor. Although you can look through all the half a billion certificates in CT and it will take some time but if you want to complete a self-hosted solution this will work great. So if you want to try to protect yourself from being vulnerable from by gone SSL or anyone else having a SSL certificate for your domain name or potentially revoking one that you're using, the best you can do right now is to use the expect CT header with the enforce flag on your HTTP server. This will protect you from someone else having a SSL certificate for your domain names that's not in CT and using it on you. However this will not protect you with a case with a certificate that has already been logged in CT. There is no current protection against that. The best thing you can do is monitor CT logs using our tools or others for all SL certificates for your own domain names and if you see a certificate for your domain name either issue before you got it or after reach out to the CA and ask them to revoke it and it should also point out that not all certificates will be in CT logs. CT was only required for browsers like Chrome as of April this year for non-EP certificates. Certificates issue before then a lot of them are in there. I've seen certificates as early as 2009 and before in certificate transparency logs but it's not a requirement to be trusted by browsers. So now we've got some requests for the general internet as well. You realize that this issue isn't going away. We have some basically band-aid asks. For example registrars could notify when you first register a domain name the registrar gives you a heads up and say hey we checked CT and the previous owner still has a valid certificate. They might even also consider auto revocation in certain cases. Another ask is if these certificates are much shorter-lived like the let's encrypt 90-day policy the issue is still there but it's there for a shorter period of time. We don't have to deal with the 5-year issue before. When you do your revocation it's kind of courteous to give a heads up to the other people using certificates with subject alt names so you don't accidentally break production websites but as mentioned before there's no mechanism stopping you from breaking production websites. And CAs could also not issue certificates that go on beyond the expiration date of a domain. So if you do who is on the domain you see the expiration date. There's no reason why the certificate should ever outlive that. So CAs could just not issue certificates that outlive the lifetime of the domain. And then the last bullet point here is really targeted at CDN specifically but also just generally companies you gotta be really careful with the way you use subject alt names. So if you use custom demands on the same certificate in kind of the worst case there are 700 one of them might become expired and that's going to open yourself up to a stranger revoking your certificate. So a recommendation there is you can do what github is doing with github pages they have every single customer on their own let's encrypts or if you use custom demand but if you share customers on the same server you're opening yourself up. Thank you for your talk here are the links to the tools that we talked about and our insecure design website where we have more information on this topic. Thank you all very much. We'll take questions outside.