 Hey folks, thanks for being here at the Crypto and Privacy Village. We have another fantastic talk lined up. Our very own Jake Williams, malware Jake, from Rendition Infosec. He's going to be talking about the semantic SSL debacle. Lessons learned in future steps. So please give it up for Jake. And now you're going to give me a minute to readjust this mic because apparently we only have one. So which is cool. And it's not cordless. So if I trip on this, somebody prepared, send me no first aid. No? Okay. I can't stand behind a podium for half an hour or an hour or whatever we're going to do here. I think we've got like 45 minutes or something. So this is something that was really, really interesting to me when it first hit. I'm really, really interested in SSL in the first place. I think SSL is a dumpster fire. And hopefully by the end of this talk you share some of those same opinions. And if not, you at least understand why I think SSL is a dumpster fire and kind of what we need to be thinking about as Infosec people. And I look around here and I'm guessing that maybe I've got 10-ish percent decision makers and a lot of influencers. A lot of technical influencers, right? So my goal here is for the decision makers to make sure you understand what's going on and for the influencers to give you some of the arguments to go back and talk to folks and communicate effectively with your decision makers so they can make the right decisions. So I want to talk about the Symantec SSL issue. Originally I wasn't going to do a lot of this. I was just going to kind of assume everybody knew. And over the last couple of weeks I've been teaching SANS around the world actually in three different countries over the last month. And I found out that by pulling Infosec people I found out that almost nobody understands this. So I revamped the slide deck a lot. Then this week Google pulled a nasty on me. And they reached a tentative agreement with Symantec on how they're going to move forward. So then I had to overhaul a bunch of stuff in the slide deck because obviously then I've got, you know, can't talk about like, well we don't know how this is going to end. We actually have some idea how it's going to end. Of course we'll have to see how that all plays out. But we at least have an idea, right? So I've been pulling data to you. I'm a data guy. I really, really like data. I'll make my data sets publicly available later. If anybody's really interested, follow me on Twitter. I promise you I'll tweet out links to the data sets when I get them up in GitHub. But basically over the last several months we've been pulling certificates. And I'll talk more about some of our methods here. We've been going out and pulling SSL certificates from the Alexa Top 1 Million. Now I know that this is not necessarily the best data set, but it's a consistent data set. So back in April this whole thing started in March. Back in April we went ahead and grabbed, early April grabbed the Alexa Top 1 Million. We haven't updated that since because I want a consistent set of sites. And I want to be able to watch the drift across basically how certificates change and what are people doing with certificates that are different. Because I would think that this industry wide announcement of intent to deprecate trust in Symantec would force change in people. For those that don't know me, I'm a big psychology guy too. I actually minored and psyched back in the day in my undergrad. And I love psychology. So I'm really interested in the psychological aspect of this, right? Some of these messages came out and people are renewing certificates. Did they do stuff differently? And unfortunately the answer there is not a lot actually. It's kind of scary. I want to talk about some of the stuff that we're dealing with here. This is not the first time we've dealt with the intent to deprecate a cert. And it's definitely not the first time that we've dealt with not only an intent to deprecate a cert, but the need for mass cert revocation. And yet we're treating it like it's the first time, right? And someone have a couple closing thoughts and kind of fire away on a couple of ideas. I'll give you a little trigger warning here. If you think at the end, I've got a solution for this. I do not, right? I like that. I tend to want to provide a solution, right? Provide a problem, talk about the problem space and then show up with a solution. I am not smart enough apparently because I've been racking my brain. I've been talking to a lot of other really smart people and nobody that I've talked to has a good solution for this, right? So the issue, why are we even having the talk in the first place? The start of this is that Symantec, their subordinate CAs, Symantec and or their subordinate CAs, they misissued a number of certificates. And originally this was found through an audit and originally it was about 12 certificates and now we know that it's somewhere on the order of 30,000 certificates. And part of the problem that Google had was that over time the number kept growing and growing and growing and there were addendums upon addendums upon addendums to the final report, right? Obviously this is never a good thing, right? You have a final report and again, we're not talking about a final report. We all work in infosec, right? I presume most of you work in infosec somewhere and you know that when you perform an audit, for goodness sakes, there's always data that skeletons in the closet, stuff that was swept under the rug, stuff that you find during an audit that you didn't find otherwise. But it's rare that I ever go back and I have to issue an addendum to my final report and it's rare that I have to issue an addendum to the addendum to the final report and I've never issued an addendum to the addendum to the addendum to the final report, which is what Symantec continued to do here, right? Separately from this, this is not a regular audit, right? But part of this key here is that when you operate a certificate authority, you are bound by a number of regulations to track data. This would be like going into a bank audit and asking them, for instance, how much money is in the vault and they're like, hold on a second while we go count that stuff. That's not okay, right? And it's not okay here either, right? We're talking about the web of trust for the entire internet. What else? Symantec failed to track the activities of its affiliated CAs and the really cool part here is if you read into this or read through this a little bit, they have this geo, basically this geo trust or geo root initiative and basically in this geo root initiative, they are signing basically certificates for what they call registration authorities, RAs. And these RAs, basically their job is to validate that you are who you say you are. So picture for a moment that you have this whole proxy arrangement, right? I can vouch for you but I might vouch for you without even knowing you and it's the registration authority that's actually validating that you are who you say you are and they say, hey Jake, right? Of course I know this person. I say, hey Jake, go vouch for this guy and I just do it. Now here's where things get interesting because over the years Symantec has bought a lot of RAs, they bought a lot of other certificate authorities, Verisign for instance, right? Is one that they bought, Thought is another one that they bought that used to be just absolutely huge, right? What's really, really interesting here is that Symantec in one of their audits failed to disclose one of their registration agents, one of their RAs. And in the later audit, they found out as Google challenged them and said, hey, you forgot to put this in your transparency report. They were like, that's not ours. And Google's like, oh no, you're signing search for these people. Now when you step back and you think how bad is this, right? Now I'm paraphrasing a lot here. Go read the docs. I understand that stuff is dry but it's important and it's dry. Hopefully I'll get enough out there that you can speak intelligently to this anyway. Symantec cross-signed a root certificate used by the U.S. federal government. And look, initially this is probably not bad. If you live in the U.S. and 100% trust the U.S. government, of course I do, right? I mean I live in the U.S. and I don't trust the U.S. government. It's ridiculous, right? Symantec issues test certificates and search for unregistered domains. Now some folks and particularly Symantec have tried to move the narrative along that one of the reasons Google is so mad about this is that one of the test certificates that they issued, I don't even know what a test certificate, well, what Symantec says a test certificate is, is a certificate they legitimately signed for a domain that they do not own that they use for testing. One of them that they signed was for Google. And you can imagine this made Google a little annoyed because what would one do with this? Who knows, right? And so they, now Symantec comes back and says look, these certs never ever made it out of the wild and yet Google figured it out. So anyway, there's like some, some interesting crosstalk going on there and some, I will tell you that as you read this, you cannot come as a rational person, as a, as a rational analyst. You cannot come to any other conclusion reading Symantec's responses than they were 100% asleep at the wheel. And rather than saying, hey guys, we were asleep at the wheel, we don't know what we're gonna do now, help us out. At every turn they deny, deny, deny. It's, it's like watching politicians debate. It's absolutely nuts. So our C.A. is officially too big to fail. Look, the issues with Symantec were serious. Google earlier dumped Woe Sign and Startcom. They said, look, we're not gonna trust these guys anymore. And they gave us about a month to knock this stuff out, a month and a half to knock out Woe Sign and Startcom. Admittedly, Woe Sign and Startcom are relatively small. And if you read the Google document, basically from Chrome, or the Chrome team, basically saying that they're ready to dump support for Symantec, but we're gonna phase it out over a period of more than a year. You can see throughout that they talk about how big Symantec is, what the impact is gonna be if they just go bam and drop it. And I step back and I have to say, look, Symantec is apparently too big to fail in the context of, in the context, at least as far as Google and then Mozilla, which is following suit with Google, is concerned. I think this is a big issue because if it's a security issue, it's a security issue no matter how big the CA is. I might actually argue that if the CA is that large, the certificate authority is that large, the security issue is actually more serious because they're signing more certificates. And hence, we need to be ready to move faster with large certificate authorities than we need to with small certificate authorities that sign a very, very small number of certs. I'll finally say that in infosec, as we're talking about this, when I do security audits and I'm looking at best practices for organizations, I rate organizations that, let's say, have a $10 billion dollar year market cap differently than somebody that has a $2 billion dollar year market cap. You have a different level of resources. Here, Symantec, my friends, has no excuse. And I'm gonna be very careful so I don't get sued because I'm sure there's someone from Symantec in here taking notes and ready to go. That's cool, no problem. But look, the originally proposed solution. Basically, there was the goal to go phase out over a number of releases and basically phase out trust for the Symantec certs that were signed during what we'll call the trouble period. So by the beginning of the year, by about February, it seems at least, and there's some missing transparency I think here from Google as well as Symantec, but it seems that everybody's confident that whatever the issues were, well not whatever they were, but among the many, many issues, and I don't have all of them listed on the original slide there, there are many, many more, those are the most serious. Based on the many issues that they found, they said, hey, we think we're good going forward, but that leaves us in a position where during the trouble period, we know we misissued around 30,000 certs. Well, at the time it was 100 certs and then it eventually was 30,000 certs. And so the question then becomes, what do you do with those? Because some of those have lifetimes of five years easily, right? You can go in and you can select, if you're willing to pay the $15 a year or whatever it happens to be, depending on who your reseller is, you can go ahead and pay that and you can get a ridiculously long certificate life. Should you do that? That's a whole different security question, whole different security discussion. Google appeared to be interested then in even as they reissued certs, they wanted to lower the validity. Symantec came back and said, look, this is punitive because nobody else is going to be bound by these timelines. And they're right. Google to their credit came back and said, hey, we get it totally, but you guys have been negligent, I think as well, wait, wait, wait. If there's a lawyer in the crowd for Symantec, I didn't use the word negligent. I was in, I was paraphrasing there, maybe negligent isn't the word. You, you have been demonstrably careless in, that sounds better. I'm not a lawyer, but mine reviewed this talk. So anyway, so the basically demonstrably careless in doing this, we think you're going to do it again, right, kind of like I've got a dieting problem, I ate too many cookies and odds are good I'm going to do it tonight too. So look, seriously, that kind of thing going, and so they dropped us down and they said look, you know, consistently you're going to have a lower and lower validity periods here and you're going to have to live with that. Symantec then says look, we've got predatory vendors jumping up and down, basically contacting customers and, and look, if I'm a vendor, I totally would be doing this, right. If I'm an SSL vendor, right, I would totally be contacting Symantec customers for two reasons. One, the majority of Symantec customers have no idea this is going on, right, or very limited idea this is going on, at least initially, and to be fair, if the proposed solution rolls through, it's going to be difficult for you to, more difficult for you to stay with Symantec than it is to go with literally anybody else, right, and if I'm a vendor that seems like a really good market differentiation and I would totally be pitching that to, to people there. Look, this is either dirty pool or good business depending on where you sit and how you feel and I don't know if I'm Symantec I call foul, right, if I'm every other vendor I'm like, heck yeah, I'm totally getting on that game, right. Regardless, know that it happened, know that it's still happening because the proposed solution proposed now, new proposed solution was just ratified this week, right, so know that it's nowhere near done and I would still expect other vendors to capitalize on this and try to capitalize. If you are a Symantec customer, be very, very careful about what you do next, make a good informed decision. My personal take, and I'll go ahead and, you know, in case anybody needs to cut out quickly and you are a Symantec customer, my personal take is I just wouldn't continue with them and this has nothing to do with whether or not I believe they're doing good CA work today. My honest to goodness opinion there is no they're not but that's a whole separate issue. The reality is that they're already on probation, right, and I feel like everything else that happens from here is probably going to be viewed with a, or responded to basically with a sledgehammer rather than a, rather than a coddling like they've had so far, right. Regardless, just so you know it's happening, let's talk about the cross-sign federal route because this is cool. I like this, right. Well no, actually I don't like this at all but Symantec cross-signed a bridge certificate for the US federal government. So the period of nearly five years, by the way, Symantec when they sign a route, this is not a certificate that they signed. I want to be very clear about this so everybody understands this. It's not a regular certificate. It's not like they, for DOJ.gov, they signed a certificate so you go to the website and get a little trusted icon there. They signed a route for the federal government, cross-signed a route for the federal government and failed to report it. Now any CA that's trusted is required, absolutely required, by the CA standards, by the browser alliance, actually that sounds like the force or something, but you get the idea there, right. You know, so Mozilla Opera, Mozilla Apple, Google and Microsoft to report when they signed new routes. They signed a route for the federal government, for the US federal government and this certificate could be used without any further interaction from Symantec to sign certificates for anything and you have to understand the severity of that because as much as I trust the federal government in the US, I think looking out here, the majority of you are probably from the US, that's fine. Step back and ask yourself, what would you be doing if this was China? If Symantec had signed a bridge certificate for China, how would you feel about that? So hold on a second, China has routes, the Chinese government doesn't have, now you're arguing with me about it, is that China, now I'm getting there, so you're jumping ahead with me here. So, okay, look, if it was the Chinese government or the Somali government, they don't have a CA, the Somali government were owned a, and then we had certificate pirates running around, sounds actually like a new whiskey pirates kind of thing, but regardless, look, bottom line, try to put your tinfoil hats on here, but this is serious, because we're talking about the same government that oversees NSA, CIA, and lots of organizations you probably don't trust, and when I step back and ask realistically, if you got redirected to a website that now had a brand new certificate signed by a trusted CA, would you know? I mean I know like one guy that would know, because he constantly is looking at certs and doing research on certs, and that's like the one guy, I brother son, and that's about it, right? I can't imagine any, I wouldn't notice, I'll be the first to tell you, most of organizations wouldn't notice either, and these can facilitate man and middle attacks, and so once the transgression was discovered, and again this is important, and others ran for five years, and once it was discovered they didn't fix it, they waited, and they went back and forth about like the hey, we're working on revoking this, this is too easy, it's literally you just revoke it, there's a process in place for this, it's called a CRL, right? So it took them five months to correct the issue, so I've got a couple of ideas here, right? Why did it happen? General laziness? I don't know, give the feds time to react to a PKI change, I don't know why this is necessary in the first place, the federal government on the browsers that they are, sorry on the sites that they use, they issue their own certs anyway, if you've ever been in the military or worked with any of those folks, you know you have to go install Root, which is weird anyway, but you've got to go install the Root that they didn't need to deal with this, right? So give the feds time to react to the PKI change, maybe Symantec just doesn't want to be a C.A., and the five months seemed like a great way to kind of bridge that, you know, I'd be like, I can't say, it's kind of like a polite no bid, somebody comes up, asks you to work a job and your normal rate is like you know here, and you're like I'm gonna go bid it way up here, kind of thing, you don't want to tell no, but maybe Symantec's like that, they're like ah, we're done with the C.A. business, but we'll piss off all our customers, so we'll just, we'll just get beat, you know, beat down, I don't know, or possibly it's to give feds time to abuse the soon to be lost capability, and I'm not going to tell you which one I think it is, right, because I don't want to put a tin foil hat on up here, but, but if you knew you had a capability and you had a very, very limited time left to use it, what would you do with it? Anyway, about the issue of test certificates, right, so Symantec issued test certificates for legitimate in the wild domains, they also did it for thousands of domains that were unregistered, and this is a huge separate issue, because if I'm able to get a cert for a site that you may register later, a domain that you may register later, and then I already have the valid cert, you can see this could be abused for, for man in the middle activity, I might actually have a cert for, for a domain that you own, and you don't have any cert for the domain, right, imagine that, that's wow. Symantec says these test certificates were never actively in use, Google of course was able to discover it, there's no legitimate purpose for this, and particularly for a site as big as Google.com, right, there's no legitimate use for this, there's no legitimate use for this, by the way, I feel it's probably appropriate to mention that Symantec, the owner of a very, very large root CA, who has cross-signed certificates against guidance without reporting it previously, now also owns some very interesting HTTPS interceptions software stuff, anyway, I'll just throw that out there, I think they reached a deal with Bluecoat last year, wow, gosh, that just hit me, anyway, so improperly revoking certificates, right, so Symantec, even though, you know, these weren't related to Google, Symantec, this month, right, so I want to put this in perspective, because if you don't, if you don't already feel like Symantec is a dumpster fire, right, and again, lawyer in the room, I know you're here, I know you're here, I'm not calling Symantec a dumpster fire, I'm just saying if these people aren't convinced that it's a dumpster fire, right, by now, maybe this will convince you, because while Symantec's already on probation, because this whole thing kicks off in February, in early July, a gentleman named Hannah Bach, I think Bach, I don't know, sounds German, anyway, he went and bought some certificates for a couple of domains that he owns, he went out and registered a couple of crazy named domains, and then went and bought some certificates, and so there are four domains that he owns, and he bought some through Symantec, and he bought some through Komodo, and I think there was one other registrant or one other certificate provider he bought them through, and then he did something cool, he went and he revoked the certificate, he said, hey, my keys have been compromised, I need you to revoke the certificate, he used a different email to contact them, basically he said, hey, here's my private keys, I need you to revoke these because they've been compromised, now a CA, according to the browser alliance, has 24 hours upon notification of compromise of a private key to revoke the certificate, the kicker is they're supposed to validate the private key and see if you can figure out where this is going, oh, oh sorry, a gap was identified, this is Symantec's response, a gap was identified in the public and private key matching process, also known as we forgot to validate the key, and what this means is that while Symantec is unaware of anybody having done this in the wild, that up until mid July, up until two weeks ago, you could have sent them some random binary string and said, hey, this is the private key for blah domain, it's been compromised, I need you to revoke the cert, now there's math that you can do to validate this, Symantec knows all about this, that's how they create the certs in the first, I presume they know about this, that's how they create the certs, actually you know what, forget it, I don't even presume they know this anymore, but look, that's how they create the certs in the first place, and you can mathematically validate that this private key is the real private key, and so picture the power that you had in your hands to shut down operations, because suppose for a minute that we take a large provider like Amazon or something, and we go and validate their cert, their private key, do it a private key compromise, now I think Symantec would have been smarter, not to do it for Symantec, but you can picture other, or sorry Symantec wouldn't do it for Amazon, they seem pretty big, seems like somebody might have gone really Amazon, seriously they don't want a key reissue or anything, or I think they'd have been smarter to see that, but there are some mid-range sites that I think you could do some really serious damage with here, obviously nobody's gone back and tested this again, I'm not encouraging you to test it again because it sounds all like fraudulent activity to me, but if you're so inclined who might have stopped you, so no joke though by the way, this is a neat attack vector and it's something that you ought to try with whoever, wait you should talk to your general counsel and then you should try this with whoever your SSL providers are because this is serious because once they revoke the cert everybody gets an SSL warning, you know this is a brand damage issue or possibly a business loss issue, so again it's something that you should check, something you should know, so I want to talk about trust issues because I have lots of trust issues, this gentleman on the front row does as well, and I want to talk about who is your browser trusting anyway because the entire SSL model is based on trust in a lot of certs that I think if you looked at the people who own those certs, I think you would look and go for instance, and I want to put this into perspective, when I consult with clients and feel free to steal my analogies here, when I consult with clients, I open up the list, we talk about this a lot for the for the browser trust and we open up the list of certificates and we look in Firefox and we look in Chrome and we just say look, would you hand them a thousand dollars and trust them in six months you're going to get it back and they're like, no I wouldn't, I'm like good, then pull trust because what you're asking for is something way worse than that, you are asserting that these people can sign trusted certificates for literally anything, many nation states as this gentleman correctly pointed out have virtual control over businesses operating on their borders, this does not align well with very complex threat models that most organizations have. Now I know what you're thinking, if you're a Chinese telecom, I'm just going to use China as an example because it's always China, actually recently it's been always Russia, but whatever, Chinese telecom and you go sign a certificate, how does my traffic, if I'm just a US company, how does it even get there in the first place? Before we talk about that, let's talk about a couple concerning ones, because Hong Kong Post does not look like somebody that I want to have for, why does the Hong Kong Post Office operate a CA that Firefox in the United States trusts and is this necessary, and should you leave this talk and remove this from your browser, from your set of accepted certificates, right, totally, and I feel fine saying that if there's a lawyer from the Hong Kong Post in here, game on, right, so look, I'm just not, I'm not down with this, is it being used? Yeah, absolutely, 38 of the top 1 million sites are secured with these, that's not the issue, if you're looking at that and you're hearing only 38 and 1, that's not the issue, it's not that it's 38 and 1 million, they can sign certs for anybody, that's the whole point of the trust, right, you are trusting them to sign a cert for literally anyone, right, and so of course then if we get that man in the middle, well, you get the idea, okay, what about Shanghua Telecom in Taiwan, right, again this is another CA trusted by Mozilla and all the major browsers, Microsoft actually goes a step further because they're cool and they trust the China Financial Certification Authority, the CFCA, I don't even know who these people are but they sound bad and I don't mean bad from a, I don't like China, just China's part of my threat model because like half the intrusions I do are China, right, that's just a reality, I operate here and I operate in Southeast Asia where, as you can imagine, China's got a lot of vested interest in what's going on and yeah, I mean, so is it fear mongering, right, no, no this isn't fear mongering, people who accuse me of this, it's definitely not, I'm a low fud if you're uncertainty and doubt kind of person, you can mind this trusted CA with the ability to do man in the middle, if you're in a foreign country, ISP, man in the middle should definitely be a thing you are concerned with, DNS cash poisoning, I don't know if Dan Kaminski's here, I haven't seen him tweeting yet since he's been here, if he is, don't fall asleep near Dan, that's a losing move, or pass out near Dan, it's a losing move, Dan's an awesome guy though and Dan walked through a number of years ago DNS cash poisoning and of course vendors rushed out to fix that over the last several years, right, and so he presented it here like in what, it was a 2008, 2007, right, so it's been nearly a decade since Dan presented how to do DNS cash poisoning and trust me it works, right and so as we look at this with DNS cash poisoning they can route traffic basically from you in the US out to wherever it is they need to or realistically they don't even need to route it outside of the US, they just pick up a server in Amazon, the Amazon cloud or in Azure and throw one of their trusted search there and game on but if you want to get into the really really scary stuff take a look my friends at BGP hijacking because this is where stuff gets crazy scary and you talk about the SSL trust thing being an absolute dumpster fire, BGP is far worse and if you're not looking at this go take a look I don't know how I missed the E there that's probably the British spelling anyway, BGP route high or root BGP route hijacking, route hijacking is probably the easiest of these to track, right, if you if you're not following BGP stream on twitter go check this out it is an absolutely awesome account and it tracks when somebody either intentionally or accidentally steals prefixes in the BGP system basically when some router somewhere says oh hey I own these go and route that stuff over here now look sometimes I'm sure that that's a mistype sometimes I'm sure that that's you know it's it's a an actual error it's not a typo it's just a flat out an error and typos happen right Amazon took down a big chunk of S3 earlier this year with a typo you know that was obviously them having a typo but there's also some really interesting stuff we saw all the traffic in Europe Germany is a big fiber switching hub we saw basically all the traffic that goes through their route through St. Petersburg for about an hour earlier this year that was just accident I'm sure is no big deal and so we've seen stuff route from Japan or Korea into China we've seen a lot of really weird temporary temporary disruptions in the force if you will and again it's based on a trust model when you combine this with certificate authorities all right you have the ultimate power for man in the middle because a lot of our when we talk about controls for this stuff for BGP route hijack and for DNS cache poisoning it's that you don't own a legitimate cert but here you do own a legitimate cert or the equivalent of one because again we're trusting some of the wrong people in my opinion to to sign this so they could steal secure only cookies harvest credentials explore user trust serve malicious updates to software we've seen that recently yeah we have that's right that petia so look tentative solutions right this is where I had to rework a huge chunk of my presentation because freaking google anyway uh so they came to an agreement and the tentative solution says that basically first off it's important because it allows both sides to claim victory semantech can say you didn't totally shut us down we can still issue sort issue certs sort of right and google can say we found issues we corrected the issues there were some punitive action taken uh everybody else in the room you're definitely not going to get away with this don't try this if you're another certificate authority this is ridiculous right uh they they're letting semantech off with at best a slap on the wrist the game on the proposed solution has three phases first off is registered subordinate c a then there's a partial followed by a complete distrust of all semantech certificates so semantech's going to partner with another c a no later than december first 2017 this original timeline uh we were supposed to be done trusting semantech certs so as of february we're supposed to be done by december uh here we've negotiated a lot uh and look over time between february or sorry between march and now things have only gotten worse meaning we only know more and more and more about mis issuance and other issues with with semantech so the picture's gotten worse and then simultaneously we've moved the timeline back and and this does not sound like a winning piece for security this sounds like a lot when we talk about like security plus and c isp where we've compromised basically convenience for security and that's effectively what we've done but cool uh regardless semantech's going to partner with another c a and the subordinate c a is going to be entirely responsive for issuing certificates and i do give google mad props here because they say look if semantech gets out of line and starts issuing certificates or if they press a button there and and they issue certificates on your behalf it's your butt right we'll delist you too right so that's good i like that right um and they basically say hey look you're entirely accountable now i don't know what this actually means right i mean i say accountability for the win but i don't know what accountability means in this case because realistically semantech isn't being held accountable for this they're being stripped of their ability to issue sorts on their own but they're being given a way out to continue making money issuing certificates and i don't i don't see this as a real punishment uh you know compared to uh you know compared to the crime uh so uh and by the way lawyers it's not a crime it was a phrase of speech right uh so tentative solution phase two partial distrust april 2018 semantech certificates issued before june 1st are going to start displaying ssl errors the thought is that by 2018 most of these certificates are going to expire anyway and so this shouldn't be an issue i'll talk a little about the reality of this a little bit later in october 2018 semantech certificates issued before december uh first 2017 uh now we'll start showing ssl errors so the idea here is phase one comes in we've got the older piece and then phase two comes in and we basically completely distrust semantech certs except for the one signed by or issued by that new subordinate c a so i did some analysis here and i started with the elux of top one million sites and we went and retrieved ssl certificates and we can start to look at certificate changes over time uh we grabbed the full certificates we wanted to allow the analysis of the data uh that we didn't think was relevant before and i'm happy i did this because what i thought was relevant at first was not the full set of data that i wanted there's a number of limitations of the approach though and number one is that i didn't make any attempts to discover uh htps on anything but 443 all right so if you're running on some non-standard port i got nothing uh only htps certs were examined uh there are certainly other certs that exist uh secure smdp you name it websites of the ssl certs that aren't in the top uh the elux of top one million i know a lot of these exist right my mom registered something on go daddy recently and she has an htps cert i don't have it because she's not in the elux of top one million she's not that interesting uh semantech's the most popular in the u.s uh or is most popular in the u.s but the elux of top one million is international and i think they're probably semantech is probably underrepresented in my data set but i don't know how to fix this all right so uh i kind of kicked around some ideas originally and then we finally said forget it i will just go with what we know is an imperfect solution like all academics document the limitations and go go go have prevalence ssl over the top one million 60 percent all right so 625 thousand had ssl certs uh as of uh two days ago three days ago and we've been continuously updating this as we go uh 112 thousand of those were signed with semantech certs right so originally the thought was uh and i think the uh hyperbole that we saw initially was one in six certs and the internet is signed by uh semantech it turns out that's actually close close to true right uh and uh this may be the case that uh it's even disproportionately the top one million because that's better than one in six semantech's actually not the biggest in the alexa top one million i i thought they would be uh komodo actually accounts for slightly more certs global sign and go daddy i cannot fathom getting a go daddy cert but okay um i guess danica patrick comes in and signs those herself uh regardless uh when we took a look at the uh the web uh basically uh net trust i think or netcraft ssl survey you can see our numbers are kind of falling in line give or take and netcraft doesn't use the alexa top one i mean i'm sure they do use the alexa top one million but they've got a ton of other sites that they're indexing as well meh i didn't do that uh because because that was lazy um but you can see our semantech has a bigger slice of the pie and according to what they're reporting uh than what we're reporting we're showing about one and six they're showing about uh it looks like one in one and three ish right give or take uh so we talk about how prevalent is semantech and how do they break out because semantech has a lot of ca's right it's not just semantech this is one of the problems if you take nothing else away from this talk go back and talk to your business not just your business but your business partners right all the people that you that you coordinate with and i know some of you in here are working for fortune 500s and you're coordinating with these little mom and pop shops and you probably have this taken care of and they do not right it's your job as good internet citizens and oh for your security uh to actually go and and kind of follow up on this semantech operates a lot of ca's it's something like 15 or 16 different ca's that we found as we started doing this research that are signed by semantech roots this is a huge deal because all of them are going to be impacted right as we look at this probably the biggest two are geotrust and rapid ssl right neither of which bear the semantech name directly right uh so there are a lot of people here you can see semantech is way down on the list and we've talked to several clients who said no problem thanks for bringing this to my attention we don't use semantech and we're like haha uh actually you do as it turns out and they're like we use rapid ssl I'm like let me explain to you how pki works let's come come over to the white board here real quick and we kind of draw out and show them how indeed this is signed by semantech and it is going away too at exactly the same pace uh so my question my my theory here is is that google was really going to shake semantech customers because if I'm in the fortune 500 if I'm in a forget fortune 500 if I'm in any business I'm not buying a semantech sort of this point because of uncertainty I don't like uncertainty if I can go with somebody else that I have a good feeling about who's not already in the penalty box uh or I can go with semantech I would go with somebody else that's just me that's uh it's not advice by the way it's just me and we did some analysis here we did our first polls uh in the very beginning of april uh we look at 112,000 as of two days ago 115,000 in april it's 2.3 percent market shift and initially I thought wow that's huge right 2.3 percent of your business is is actually pretty significant I'm going to tell you that I thought it was going to be more I thought by this time we'd be looking at somewhere in the 10 to 20 percent market shift to be to be honest I thought 2.3 percent and it turns out I was extra extra disappointed because it's not really 2.3 percent I'll talk about that in just a second the numbers are kind of lying here I'll tell you this is less than I would have predicted I don't have data on normal SSL certificate market shifts I don't think anybody does well some of the CA is probably do but I don't know over time how do people change certificate authorities I mean the lazy thing to do is literally just to pay them again and get a brand new cert issued you don't have to change anything it just kind of kind of works it's the easiest possible thing to do it's it's why I think so many people on domains register with GoDaddy right they went they registered once and the easiest possible thing to do is let them charge your credit card again game on right a lot of people moved away from Symantec but other people moved to Symantec and this my friends is absolutely mind-blowing I think although I have no data to support it this is the problem right because the majority of them that moved came in via rapid SSL and I think that's probably what we're seeing there is people are if they're cognizant of all at all of the Symantec issue they don't understand that these are also Symantec so literally you take nothing away from this talk one these are Symantec right and two Symantec signed a federal government cert that they could use for man in the middle for a period of five years then failed to do anything about it for five months right that's probably the other thing that I would take away from this because I don't know anyway so moving away from Symantec we saw 4,400 domains move in that period where they go largely Komodo and let's encrypt I'm actually really excited about let's encrypt they sign a lot of certs for malicious domains I mean that's just a reality a lot by the way if you're looking to do good cyber threat intelligence type stuff early warning go pull the certs that let's encrypt is they they're very transparent about it the domains that they're signing certs for and look because you can see a lot of them like grep for the words PayPal or iPhone and brother there are some crazy numbers of domains there so while let's encrypt does a lot of malicious signing there they do it the way they're supposed to per the browser alliance they're verifying you own the domain the fact that the domain itself you probably should know is a separate issue they don't deal with that so as far as the movement to Symantec where do people leave from right turns out a lot of them left from Komodo a few of them graduated from what's encrypt but overall woe sign right now that they're untrusted some people finally got on the bandwagon between April and July realize they've been running on a untrusted cert for a number of months and said hey where should we go the next untrusted cert right anyway so shouldn't certificate lifetimes you're going down right we saw in the beginning where the initial plan the initial plan as it were was to lower the lifetime right of these certs I would expect then that as people go and renew these certs that Symantec would be like jumping through hoops to tell people like hey don't register for three years right because we can't support that because because the the google plan says so when we started the certificate lifetime was 656 days in the July data set it drops to whopping 652 I am thoroughly disappointed here I really am I'm extremely disappointed that we're not seeing a decrease in the lifetime of the certs and and again I can only ascribe two things to this right one people don't understand they have Symantec certs although I have to say as people are reissuing certs through normal expirations it really irks me that Symantec isn't advising customers and maybe they are I don't know customers could be ignoring them but but it would seem to me that Symantec would be like hey this is on the horizon and we think you might need to deal with the possibility that this is going to happen so no I I know what they're telling their customers so the the comment here is they're telling their customers we're too big to fail we're not going to fail don't worry about it I I've seen that correspondence through my own customers it's yes anyway so so doing this better how can we not screw this up again right look it's easy to ignore the potential impact people come back and say there's not a single example of malicious activity as a result of this that's fine it's fine for you to say you don't have proof of any of this activity but we never will right this is kind of one of these spots where we don't know what the damage is and we likely never will I don't think we can say it's okay nothing happened here because again we don't know if anything actually did happen one and two the risk here is is the entire internet ecosystem and I'm not trying to not trying to do hyperbole here it really is the entire internet ecosystem this is not our first certificate fire drill we dealt with this in heart bleed where we had to invalidate lots of certs and we sucked at it we did all right and here again we're in a position where one of semantics arguments consistently over the last several months has been we can't invalidate and reissue certs the global ecosystem is not set up to do this and they're right that's the one thing I'm gonna back them on they're right our ecosystem is not ready for this we need to plan for dealing with mass certificate revocation and reissue because if it's the ca that's being untrusted you need then to validate yourself with another ca this takes time it takes resources and ca's don't just scale orders of magnitude larger overnight so possible solutions these are admittedly flawed but they're possible solutions we can have a trusted third party pre validate organizations in case a ca shuts down unexpectedly kind of like a global brain trust of ca's we kind of have something ish global browser alliance type thing could we have this too I think that's ripe for abuse as well so I don't necessarily recommend that but it's it's a thought we could buy multiple certificates and have them on standby in case RCA gets shut down so I got buy from Symantec and Komodo I think this is also a bad plan right I've actually had customers recommend or like well to hedge our bets we bought an extra private key from Komodo and I'm like no no no like where is that thing right is it online someplace is it is it an hsm because I'm worried about like keeping control if it's hard to control your keys as it is doubling the number of keys makes it twice as hard right that's just math we could have third party ca SWAT teams that can jump in and run a certificate authority when issues are discovered or surge support is needed so if Symantec or if the browser alliance if there were this magic group that had a SWAT team they could have jumped in here and done this of course realistically all these costs money all of them are going to increase costs I don't have an answer here right somebody smarter than me is probably sitting in this room who does have the answer totally let me know what that is after we're done with the talk so I can be a couple of parting thoughts here it's 2017 we should not be talking about this we should not be talking about this we definitely shouldn't be talking about this months after the initial release Symantec should not be telling their customers it's okay we're too big to fail I would as a personal thing not as a legal thing Mr. Lawyer whoever you are as a personal thing I think it's I think it's important that Symantec is talking about the too big to fail piece and not talking about the very real security issues that have been introduced I think that that's just bad for the community overall I will also say that Google's handling of Symantec is not good either I think Google deserves to be in the penalty box here they identified a security issue they have the power to take action and they are valuing user convenience and customer convenience over security and we in Infosec routinely tell people don't do that there's a hardware hacking village with the voting machines right down the hall here where that's the whole reason it's there is because of this garbage right Symantec should have been rapidly untrusted just like woe sign forget this too big to fail stuff it didn't work for the government didn't work for the banks it's not working for CAs either and then finally review the CAs that you trust in your browsers this is not so most of them don't need to be there and the way you can convince your management of this is run bro bro network security monitor NSM go through you can configure bro with bro scripts to log all of your SSL certificates and find out who's signing those things and if you go through over a 30, 60, 90 day period whatever works for you 30 is fine for me and you look and you find out there's only five or six of those trusted CAs signing the certs that you are actually using get rid of the rest maybe you cause one pop up some place I will tell you and I can't share the stories here but I will tell you and somebody else sitting in the audience saw an example of this where we actually are fairly confident that based on some good net witness data that we got popped like this and it was a big company and it was a big target and they were having trouble getting in and we're pretty sure that's how they got in because something served a certificate that was trusted but not trusted and I'll just leave it there that's it I got like five minutes tops for questions sir speak in the mic because otherwise they can't there's a mic right there I think it's a mic but hello yeah that's good it works so there was a slide that mentioned that as a main tech to perform the modulus comparison between the private key and the public key in the in the third yep to validate but they did not do something else what is that something else do we know oh no that that was it they actually didn't validate the private key right so the question is did it what what else didn't they do besides validating the private key we're talking about here yes this one it's a perform the modulus comparison which I thought was was the whole key of this mathematical comparison between the public and the private key what what did they not do it's my understanding they were doing some mathematical shortcut I know whatever they did is not a full comparison of the key because what he sent them was not the private key yeah so they validated the modulus part of okay so just to preserve this on the video here again basically that what we're thinking here although I don't know that we're 100% sure in this case is that they generated another key with the same modulus they validated the modulus rather than validating the key itself right again you can't fake that if you can I totally want to work with you because that's going to be awesome right we're going to get stuff taken down sir do you know if the revocation of the U.S. federal government government certificate actually had any effect my understanding the revocation process is kind of not working as we wanted it to yeah so did the revocation of the U.S. government sort have any effect the revocation process definitely is not working the way it's supposed to the cert eventually expired so it's less of an issue now than it than it was but no you're right the revocation process is not working the way it's supposed to I don't think it had any effect it eventually expired off anyway but it was a cross signed it was a cross signed route and that was one of the arguments that Symantec made is that a lot of government sites were signed with this and revoking could potentially cause issues there again I don't think they understand the math behind this or some of the people making arguments don't understand the math behind it sir what are your thoughts on cert pinning as a possible solution yeah so what are my thoughts on cert pinning as a possible solution for some of these issues I will say that cert pinning is neat but in you know from a blue team standpoint I need SSL interception all right I need SSL interception and if you're doing cert pinning I consistently then am overriding that in group policy I'm overriding that in browser settings because I need to be able to do SSL interception I can't run a large organization today and do data loss prevention without being able to see into your Gmail I mean this is the reality because if I can't see it the attackers know that you know they can use that John Strand and his team over at Black Hills put together GCaft I don't know if anybody's seen this but it's it's like NetCaft except it does basically data Xfill it's a free framework for doing data Xfill and command and control over Gmail right what's that oh I thought I heard somebody mention GCaft I did Dropsmack a couple of years ago the same thing with Dropbox right and so if you're doing cert pinning that's great it solves one problem but creates a brand new one right because we can't see into that and then attackers are going to use those types of techniques I don't have the right answer here unfortunately so a couple of things with a modulus I'm not sure you can actually generate another key unless you've got the tosion if you've got the tosion you can factor in the first place and get anything right I am not a hundred percent sure on them off right but again I anybody has P and Q they've got they can make any key totally understood right and again this is a spot where his blog post is not 100 percent detailed for obvious attack repeatability reasons but he confirms that he sent them not the private key and they revoked it anyway all right and that's that's the key what regard not to overload the word the key but that's the that's the point is that they revoked a key they shouldn't have Symantec admits as much I think that they were doing some mathematical shortcut there that is not the full is not the full analysis and then you know RSA's security if we get quantum or if we get advances in factoring we have to be better at handling certificate replication generally shouldn't that have been baked in in the beginning yeah sure so if we have advances in factoring kind of repeat this for everybody if we have advances in factoring you know RSA basically a lot of the security kind of rotates around that we have to get better revoking the stuff in the first place so I agree with that wholeheartedly you're right obviously the shawl one apocalypse taught us a lot about that we accelerated the death of shawl one for for certificate signing because of advances in well not factoring specifically there but advances in I think combinatorial mathematics there right we're looking at the collision vectors in shawl one so again I I think you're a hundred percent right on that we have to have a better way to do this and then we don't so okay I think it's a good spot to cut it all hanging out here if anyone wants to talk well like in the back around the hall or whatever thanks for coming appreciate your time see you next year