 Hi, I'm Brian. I'm here from industry and from the ruthless practical world Somewhere on the East Coast. Hello. Thank you all for having me. I am not at least in this room I'm not a cryptographer Maybe I can pass outside and I see some faces here who tried to teach me crypto But I'm not good enough at math so I had to go into computer security instead So now I have a lot of computers and I'm responsible for their security I don't expect you folks to know what Akamai is Akamai is sort of a stagehand for the internet. It succeeds when other people shine So when I say crypto at scale by scale What I mean is a quarter million or so computers like this when you wake up in the morning You want to see how the Olympics went so do a couple billion other people if a billion web requests all hit The Olympics web servers at the same time or NBC comm at the same time NBC operators have a really bad day, right? But folks want to watch the Olympics here at a certain time and they want to go to NBC Maybe folks in London want to watch the Olympics on the BBC and folks on the other side of the world on something else so they can all share capacity in some sort of hierarchy of web caches and If you're using the web, you're probably talking to Akamai computers Not whoever else you might expect to be talking to When I say at scale what I mean is on the order of a quadrillion requests a year about two quadrillion in 2015 and Roughly doubling a little faster than Moore's law and every request is about 60 kilobytes So that's a lot of TLS handshakes right about a quarter of those turn into a TLS handshake So 250 billion right 250 trillion TLS handshakes in a year All done on about a quarter million machines Most of them are web servers the other chunk or some sort of infrastructure behind that of those About 80,000 or one hierarchy like I just showed you And they all share one certificate name so you can reuse handshakes and you can redo session cookies with that The slides will happily be online later if you want to instead of taking pictures, too Why do you why would you want to use this you use this if you're putting image tags in a way? Web page and you don't want to serve all of them so that may be for social media all those profile pictures all those come from this You use this if you want to do streaming video all the olympic stuff is coming back to this something where end users aren't going to see the host name But there's a harder problem if you want to do a whole website delivered over this sort of hierarchy you have this problem because of historical realities about SSL and TLS and so we end up with maybe 50,000 servers for vanity names Right, so if you want example comm to work We put that on this system So this is what I mean when I say scale maybe 5,000 points of presence hosting 10,000 names on about 50,000 machines backed by a big pilot infrastructure a Consequence on that is that we're sitting on at this point between half a percent and 1% of all IPv4 addresses So when you heard a year or so ago that the world had run out of IPv4 addresses That was sort of a peak oil moment for us of realizing that well We've been doing something for 10 years, and we're gonna have to do something else for the next 10 but all of that together is maybe 20% of web traffic and This sort of graphic is the kind of thing that we have to use to steer and explain it to ourselves Each one of those little candles is a place where we have servers And you can see they're kind of concentrated in places where there's a lot of people and maybe there's bars Or in places where you don't see so much IP use Okay, that's scale. What's crypto crypto in general is a tool for making things not work Now when I give this talk in a room that isn't entirely cryptographers. I say that's a cryptographers definition Maybe maybe not in this room TLS in particular as a tool for making the web not work If that's me and that's a web server that I want to talk to I want one arrow to work And I want the other one to not work. What do I use for that? That's crypto Crypto is the worst possible tool for all of the things that we use it for and I mean that very seriously right because of all the expense And all of the impracticality of multi-party computations and and public key crypto and all of that If we could possibly get away with using another tool if I could just air gap things to keep them separate Why I'd be delighted to use an air gap instead If I could have a safe in my office and put my secrets in there and I lock the door I have much more faith in that safe and in the Shear physicality of its walls and the geography to access it instead of encrypting something and posting it to Dropbox Right, so we only use crypto when we have to and as a result. It's the worst possible tool for everything that we use it for My marketing people also don't seem to like this explanation somehow when I say we want to to market the new encrypted secure websites and they're not going to work nearly as Much as the other ones did the product people tell me that I'm not allowed to talk to the customers anymore So we have that in common. All right, I would like to tell you some stories Crypto in its function of having things not work has over the last five years or so had a number of really bad days Mostly days were either something worked that absolutely should not have worked Should have worked or now I'm confused right something shouldn't have worked and did And so we get things like Heartbleed or something should have worked and crypto got in its way and didn't and things failed that should have worked and That's some of the other ones up here We're gonna look at this bottom row of the big icons And I'd like to tell you three stories and out of those three stories I'd like to bring some Challenges from industry to say as well as we understand you and that's limited Here's what we'd love to have from you and some ideas that we'd love to work with you on on how to take stuff from crypto Research and bring it out to where the rest of the world can use it right places where we have no tool And so we want the worst possible tool because at least that's a possible tool So let's start with the big one from 2014 right as of the middle of 2014 People thought that the world might be ending for practical computer security that we were gonna have years going on like this Now since this is not a sysadmin room. Let me remind you. What was Heartbleed? Heartbleed was a bug in a popular crypto implementation called open SSL It was a common sort of bug the sort of bug you might expect Static analysis programs from all over the world to find and when consulted they in fact did find it So maybe we've learned something there The important thing to know about it is that it as a result of this bug There was information flow from server memory to clients that shouldn't have been What was conveyed in this information flow? Well, we're not sure really anything that the server knew about Might have gotten conveyed to some clients long-term TLS secrets other web requests that were being handled if this was a mail server somebody else's email or Whatever the sysadmin was typing at the console right pretty much anything that might have been sitting around in memory Might get leaked out to the clients No one ever saw Heartbleed used in anger But we all knew that it might have been and it was so obvious that even though it It had been live for a couple years before it got discovered It was so obvious that surely somebody else might have or must have discovered this and used it So this gives us a natural experiment if you tell the world that your keys and secrets are compromised And everybody knows it right it's really easy to tell from outside who was exposed to Heartbleed So tens of thousands of organizations got told your keys might have leaked probably leaked and Everybody knows that they did it isn't just that you like you accidentally lost a disc on the subway that had your TLS Private keys on it Everything's out there and so you should probably rotate everything so we get to see what happens when we tell people that the answer is The answer is alarming so this is a graph over time of How many of those 10,000 or so vanity name certs get rotated and what you see here Let me use the laser pointer instead Because who could turn up the chance to use a laser like this All right, what you see is a couple days are nothing happening That's nothing happening because we thought maybe we don't need to rotate everybody's certificates after all Nobody saw Heartbleed used in anger and we had certificate authorities calling us up and saying look We didn't build in search capacity We can issue at maybe our normal rate maybe twice our normal rate There's no way we can reissue all of the certificates used by all public websites in a matter of days We just don't have the people for it. There's physical protocols involved of carrying keys between machines can't be done Okay, then news stories started hitting and pundits started talking And so within about a week or two all the keys that we can rotate all by our lonesome have been rotated Right, so that's a little more than half the population Over the next couple of weeks There's a lot of phone calls and a lot of explanations of customers who need to rotate things But sometime around mid-May certainly by 45 days after the event It slows down and this rate here is the ordinary keys expiring after two years Right, so what that means is the by using track that out the rate the real update rate ends around the end of May From April 14th initiation So 45 days after if you didn't rotate by then you weren't going to you're just gonna wait for your keys to expire And move on from there and you can even imagine the face of somebody who realizes let's say they were on an extended vacation And they got back after 90 days away. They left April 13th a Friday bed choice They get back 90 days later. They're in here and they're looking at well now. What do I do? Well, maybe his certificate only expires over here, so he says he'll wait it's been 90 days already What's another 10 or 20 or 60? But here we have about 25 percent of the population Don't usefully rotate at least 15 percent Don't respond to public notice that their long-term keys are compromised by doing anything about it Isn't that neat right? I always used to think that the sort of protocols that said we'll use a long-term key as a identifier with somebody and we'll have some number of Compromised parties in the network. I wondered about how realistic that was and this is a new Translation of that model that at least 15 percent of people in the world who've heard that their keys were compromised didn't do anything about it right, they're not actively malicious they're just passively malicious and This is the best case This is for people where you could look from outside and see that their keys were compromised And they knew it and they knew that recursively right if you look at a protocol on the back end There we go like Streaming video you've been watching the Olympics this week right the way streaming video works in the modern world is all over web protocols There's a camera pointed in an athlete and that camera is connected by a big thick cable to a machine called an encoder And an encoder is a translator from the sort of protocols that a video camera speaks to Internet-like protocols and the encoders are terrible Right, so if you're a camera person and you're a streaming video person you care about your lenses You care about the sensors in your camera You don't really care about the encoder the encoder is there as a sort of a piece of magic the encoders make that video available to something like Akamai and They do it over HTTP with basic authentication Maybe maybe they use TLS and if they do maybe they check the certs Maybe not but so you should think about this as there's a password embedded in that and if you Looked around on paste bin in the dark corners of the web You can find the URLs for those encoders or that the encoders used to talk to Akamai And I devoutly hope you can't find the password because if you can find the password You can have your video stream go out instead of Simone Piles And that'll be exciting for everybody especially me We had some hard attacks about that this year right people who set up new encoders and it wasn't working right and Whole countries were getting the wrong thing instead of the Olympic stream all kinds of excitement not NBC. Thank goodness But there's a problem if you want to rotate that password Let's say I accidentally tell you that the password for NBC for the Olympics is actually the word NBC password Please don't try that And now I've got to change the thing right so there's a relatively ordinary three-step process to go from All right client and server agree that the password is a the server starts accepting two passwords The client starts sending the new one the server stops accepting the old one right and just straightforward three steps to rotate a password This should be normal There's a problem the encoders are terrible the encoders only have one password field right many of these encoders were literally purchased in The 90s and set up there's some sort of appliance You may not be able to get a replacement the people who made it or out of business There's no software for them you could buy a new one right if this will only went one way if it were just a matter of getting Right the client here only ever has to know one password at a time right so that could be the solution Akamai can accept two passwords, and you'll just set up a new encoder or you'll change it over doesn't work that way typically the encoders the server and the What you would think of as the web servers the Akamai stuff is all pulling from there So you can't rotate it most of these that are used for live video around the clock are feeding into things like satellite installations and into local television and Because of consequences of the protocol. I don't want to go into here if you rotate that password You have to restart the stream on the encoder and that requires someone at every other endpoint to push a button Just one button In order to start the stream working again That means to change that password if I actually do tell you the NBC Olympics password Someone at every local tele NBC affiliate television station in the United States needs to push a button to say try again Right, so we considered sending out a great number of undergraduate interns or something to do with like a drinky bird that could push the button But what actually happens in reality is these passwords have not been rotated in typically more than 15 years And cannot be right anyone who adopted live streaming video in the early days of streaming video on the internet is Pretty much stuck with this the empty string of course And I see you all wrote it down so This is all of these those passwords were of course in memory when a heart bleed happened so all of those folks we also caught in contact with and Said maybe you want to do something like if you were thinking about deploying new encoder hardware Maybe now is the time it's April the summer sports season is starting up You maybe want to do this before the World Cup because you can't do it during the World Cup You could advertise it as you're offering new ultra HD whatever other features You're gonna pack in and the answer was we can't do it. We're not gonna do it Nobody's rotating passwords on this they're stuck So I could show you a graph for this for uptake of rotated passwords for streaming But it's just a flat line nobody did Literally nobody and there's about 300 other secret classes that we were tracking Over the the months after heart bleed to see what happened when you tell the world Basically, everybody who's using open SSL you have to change your secrets You should rotate them even if you were using something like Windows s channel that didn't get compromised your counterparty's resuming open SSL So you should change that and the answer is at the best case 15% of people ignore you in the more typical case nearly everybody ignores you and sometimes Absolutely, everybody ignores you So much for long-term secrets as identities next crisis Some of the people I demoed this on said they wanted to know why I was putting a duck astronomer on the slides This is not a duck astronomer as anyone can tell this is a poodle carefully hand-drawn. I don't know why he's looking at the stars Poodle was another event that happened later in 2014 and poodles really interesting because poodle in a lot of ways isn't a technical failure poodle was the world finding out that SSL 3 from the 90s was broken Now everybody here knew that SSL 3 from the 90s is broken. That's why we built TLS 1 and 1 1 and 1 2 and so on right But this was a case where there was a crisis of faith Crypto is about belief and about faith and that faith and that belief around SSL 3 were suddenly shaken and so everybody's got to run off to the revival tent and Make absolution somehow and sacrifice a goat or a poodle and try to get proper security again But it gives us another natural experiment What happens when the whole world has a crisis of belief and they suddenly believe they've got to turn off SSL 3 in like a weekend? I did come out I think of Friday night And I have dark memories of walking the wrong way into the office building against the tide of people leaving because we were gonna Go try to deal with this What we saw I can't quite show you I have this image in my head, but it was from our live telemetry systems And I didn't think to take a screenshot So I've sketched what I was looking at the night that the world found out SSL 3 is broken Here's a specific way that it's broken and we think we're going to see exploitation of this at national borders within three days Right our threat intel teams went and made some phone calls and they came back to us And they said we think that SSL 3 across national borders by the end of the weekend by Monday morning The the Americans will have something up the Chinese will have something up the Russians the French whatever And so all the national border tapping points are going to be seeing poodle exploitation in like 72 hours So you have to move now now now now now now now that didn't happen And there's a separate operational talk about why the threat intel predictions were wrong there But this is what I saw that night trying to make that decision So I want you to ask yourself what you would do if you got that message. This is a locus of a population And it's log log Mostly because that lets you see a big line across it. Otherwise. It just looks like a great big letter you That this says nearly all of those 10,000 vanity domain names those 10,000 websites call them roughly the biggest 10,000 websites on the internet nearly all of them get basically no SSL v3 traffic they get a little bit Right, there's a there's little tiny fractions that come in from somebody writing a script in like Java 6 or something and Running that and it only speaks SSL 3 or somebody who's behind some horrible academic proxy at a horrible University And it only does SSL 3 going forward some abandoned hardware whatever so there's a little bit But essentially they get none and they wouldn't notice or care if we turn it off maybe 100 customers or so have 1% of their traffic as SSL v3 and then Basically nobody has more than 10% of their traffic as SSL v3 Except for six or seven hundred stalwart diehards who thought that SSL 3 got it right and they're not okay with ever switching to TLS And so they're still running their entire website is running on SSL v3 And so they're using a 16 year old dead protocol, but maybe they too have now had a crisis of belief So you might imagine we want to break this into chunks and sort of handle these by parts Right, maybe you do a different thing for that set of people than for this set of people And what we can do to try to help that bulk of population is we can see where can we turn off SSL v3? Right, how what percentage law at a point do I draw the line, right? Anyone think we should draw to 10% right? Can I see hands? No, all right. How about 1% Someone bold good couple people 10th of a percent Couple one a million the rest of you I put you to sleep God. I'm so sorry. No, no, wait good 100% oh, I'm sorry. You're right. Just turn it off for everybody. All right So what we found was there's few enough people over in that camp that they can all get a phone call right that night and be told Hey, this poodle thing is happening You need to do something about SSL v3 and your entire website seems to run on SSL v3 Maybe you should do something about that right because to fix it is From the server point of view is a couple checkboxes in a web portal and then you fix whatever your clients are And of course a number of them said to us things like I can't do that My clients are deployed hardware and we said well, what are they and they said oh, they're automated teller machines. Oh Okay We can't fix that why what are yours? Oh city buses that are reporting in where they are. Oh, okay So what we said is we're gonna draw the line at 1% We looked in it some of the data was and we saw that a lot of what was hitting those 1% and less sites was Web spiders right sort of competitors to Google and duck duck go and Bing and that sort of thing who were crawling the web But they were like little tiny ones that nobody had ever heard of and as far as we can tell We had no way to contact them the couple of customers We'd spot checked in with had no idea who they were and said that's undesirable traffic So go ahead and turn it off. So we set the line at 1% and we turned it off and I almost ended up on the job market Because it took about 90 minutes from pulling that that lever over at about 2 in the morning Right when not a lot of people were using it, but more importantly when none of the operators were watching So about 90 minutes later we start getting phone calls. Hey, something broke. You have to turn it back on and what a list they went through of what had broken of I Need SSL 3 why you only we saw you only have one SSL 3 transaction a month Yeah, but that one a month is how I do payroll for my company. Oh Okay, or I use SSL v3 came in Several of the national banks from Venezuela and like big big commercial and retail banks From South America came in and they said we need SSL v3 Why our client population often has devices that can only speak SSL v3 They might be banking from like an old mobile phone or something like a feature phone says no key on it So how do we help those people? Well, you got to get them new phones and until then we have to have SSL 3 on for banking Even if that hurts the people who were using modern protocols to talk to you too. Yeah, even so I can't turn off 5-10 percent of my user population. Oh okay, or the best ones I video game company called in that makes both set top boxes and Small portable devices and they said our entire infrastructure all of our little handheld gaming platforms and all of the the set top boxes All the whole there's a big web store. You can put in a credit card number and download apps and so on and play the great games of the 80s all of those use SSL 3 and they can't speak anything else and We don't have the engineering resources to put out something new We said, okay, we've got to turn it back on for them So we're turning back on spot checks here or there and there and then the world's second largest router Manufacturer called back and said you have to turn it back on and we said what how could you possibly and they said well? We missed something Okay, that the update mechanism to update firmware on their routers used SSL 3 It also had end-to-end checks But it would only they said it's okay for us to not upgrade this because it's fragile. Why break it if we break this We have to ship discs to a billion people So we'll just leave it in place after all there's an end-to-end signature So there's an end-to-end integrity check so we didn't care so much that it was SSL 3 But now we needed to work over SSL 3 so that we can continue pushing updates We swear we'll get another update out and then you can turn it off. Oh, okay. How long will that take 18 months? Okay, and that's again from mass consciousness within the computer security community that said this has to go this weekend and We ended up with a Swiss cheese array of turned off and turned on and pulling people aside to try to fix this and the core issue that Made this so dangerous for the world was that the same keys are used for SL 3 and teal as one and one one and one Two right so the next time that you write a paper in your introduction You say an uncompromise here means that these keys are used only in accordance with the protocol Remember that on my side of the world that never happens because the same keys get used across many iterations Okay, one more disaster and this one should be news for most of the folks in the room. That's a fossil Fossils an acronym for fixed origin SSL This is one that now I can talk about in public The issue is if there's end-users who are talking to some Intermediate proxy whether that's Akamai or a local blue coat proxy or whatever's on the Resnet things that I see messing with my connection from the dormitories to get out to the internet then There's a man in the middle right an approved one an acceptable one maybe Who's between the end users and the origins and there's been this wonderful set of work? From people like Ivan Rish take it qualis who did SSL labs this for those who don't run a web server this is a website you can go to when you tell it a server name and it examines it and it does all of the Automated checks you'd want an expert to do to tell you how good that TLS endpoint is this is fantastic because it tells people by the way You're terrible. Oh, I didn't know I was terrible Click here for instructions to not be terrible. So I do and it says we saw here's the server version. You're running Here's a set of configuration in what you should put in I do that and I run it again And it says you're great. I love being told that right so this is something that they really moved the bar for operators but what we found was that very many of The customers who were using services like us or other proxies had configured the proxies to not check their origin SSL configuration at all So some a tool like Ivan's like like the quality SSL lab scanner can't see what's behind Akamai It sees the Akamai edge and that looks wonderful. It tells us we're great. I love it. I check it every morning But when you go to look behind the Akamai system back to those origins Nobody was looking at that and so the sys admins of that population had not had that social feedback cycle of a customer Calling them out and saying you're terrible fix it. So they had baked in crypto decisions often in the 90s early 2000s We're still living with them even better. They'd often told Akamai Don't ever break a connection going forward to the origin Right. There's a little checkbox you can use when you're configuring a proxy like Akamai You can either set the switch to check the origin cert in which case it'll break when the origin cert expires It'll break if the name is wrong. It'll break if it'll break the connection if it should be broken Or you can set the switch to don't do that make it always work All right, we're gonna do a show of hands again. Who thinks that everybody had that one set to where crypto will break things? Good very well. Really? All right, who thinks that it was maybe about half and half All right, do I hear in 9010? 99% five Not five percent five. There were five customers who did get it, right? One of them was Akamai zone infrastructure the other four are wonderful wonderful businesses that care a lot about end-user security But let's just say that your bank and your newspaper and your email provider Maybe could have looked at this setting a little more carefully and thought harder Right statistically everybody got it wrong and we hadn't known right We shipped the setting we we shipped a manual for how to use the setting and we discovered that basically everybody given that choice Got it wrong. So we called some of them up. We flew out we visited We said what were you thinking like to you know major banks, right? So we walked down Wall Street and just crossed the street back and forth through the banks said excuse me We want to talk to you we in a routine review of your security configuration We discovered that you don't actually check anything with TLS when it crosses most of the internet, right? So an end user is getting to if you go to access your bank from on campus here There's Akamai servers on UCSB property So you're talking to those with TLS and then the long haul back to New York is happening with no authentication and no integrity Whoops So We found this happens. We'd like to fix it and they said oh they talked to us They told us stories about how they put it this way in test mode to set up Right, they wanted to make sure that the site even could work because they think of crypto as a tool to break things and Then after that it tested successfully and they never got around to shutting it off. We said it's been 12 years and they said oh, yeah, alright So it took four and a half years to move everybody across to where that's fixed and to where now Essentially all of those forward connections use TLS, right? Overwhelmingly they can pin a key. There's no need for PKI in this It's a much simpler problem than the major web crypto because there's only two parties, right? If if example bank comes to us and says they want to be a customer they show us what key they're gonna use They never have to rotate it. There's no need for certificates and dates. It just here's the key I'm gonna use look for that. So it's just plain old public key crypto and still Four years and more to get folks to adopt it So what do we learn from that if you imagine a stereotypical cryptographic protocol and and forgive me I'm from a sort of more more algebraic world of Someone's gonna send a message and there's a little proof and you send a message and there's a little proof before you send the message Right and those proofs may be buying some variables that you're gonna put in those messages If at all possible some implementer will leave out all the proofs and they'll just send the syntactic messages And they'll see that it worked right so simulation based proofs of security are a lot closer to reality than we thought You can in fact operate as a cryptographic endpoint on the internet with no brain as we prove So that's an issue and where we're starting to see some bending in that and some flex is around algorithms where you can't forget to check It right where you're gonna we're gonna give you something for secrecy and integrity and you can't possibly Decrypt the secret without calculating the integrity checks them along the way That's a way to get these endpoints to actually do an integrity check some if they can skip it They will right even if they have to calculate it and send it back, but they don't have to compare themselves Someone will find a way to skip it We heard the most amazing Explanations about why people didn't want to upgrade their origins to actually do the crypto and to actually check Provide certificates and get them checked some of us said well I can't put SSL there some of them even insisted you have to downgrade SSL comes in we want you to go forward over plan old HTTP. I can't I'd have to pay for it Who here operates a web server of some sort? Okay, yeah, all of you are using either open SSL or you're using Microsoft s channel, right? Anyone else? Okay, so if you said to me, I can't turn on SSL. I'd have to pay for the SSL software module That's weird right like if you go talk to operators That sounds bizarre until you hit the largest end of the scale and it turns out that a huge number of them We're using defunct web servers And they baked in things right like they had like AOL web server AOL server with tickle baked in And so in order to switch web servers to something that supports modern crypto they'd have to port all of their code from tickle to pearl or something else or They had a caching a load balancer and the load balancer is a T last man in the middle just like our caching proxies and They didn't have a service contract for it, right? They bought one they took it off the shelf the vendor says you shouldn't use this unless you have a service contract There's some software patches will happen, right? It's a complex machine that speaks like four different TLS eight machines going at once there are bugs in it Well, we didn't we decided not to buy any of those so we've just had it running since 2002 and They went out of business and we need a new one. So we have to pay for it We saw a huge number of folks from regulated industries who came in and said I don't want to have to do this You can't make me do this. I'm regulated and we said what does that mean? And they said well, they check me out for my security and we said how did you get this past them? And they said well, we did my assessor says it has to be SSL. They didn't say it has to be good SSL And so one of them told us my assessor says it has to be SSL. So I can't change to this TLS thing We had a conversation with that assessor and managed to help them smooth that one out We heard my next change window was in 2018 and remember in some cases we were hearing this in 2012 In some cases we heard yeah, we fixed it switch it and we'd switch it And they'd say oh my god switch it back Why well we said we discovered we had another data center. We lost it We'd wondered where that operating budget was going it turns out we had another 10 machines in Chicago And they haven't been updated in ages and they can't speak modern protocols. So we're gonna have to work on that Oh, but but they have access to all of your this was a bank, but they have access like all the customer data Oh, yeah, I Close gonna count after that one and the best ones were I'm paying you to deal with the security I shouldn't have to do anything And and a number of those explained they they couldn't change it. They did not control their TLS stack So this was people operating some of the biggest content sites on the web a major media organization said I can't change my TLS Configuration of my origin why not I have no control over it. Well, who does the IT department? What well, so apparently their IT department is staffed with Morlocks, right? And so that's them over there and they have an IT department that in some cases even ran a CDN of their own Right, they deployed thousands of machines around the world or in the service area, but they're paying us and we're not cheap They're paying us piles of money to use us as a CDN instead Why because if you're the marketing department or the content department one of these then it's much cheaper to pay us in Money instead of you and you get to talk to some of our C care people instead of using the free service which is run by Morlocks and There's a message here for the enterprise IT groups of learning something about when they when they cut staffing and they let the Morlocks Talk to the content department then what happens is the content department interprets that is damage and rounds around it That one I don't have a solution for and I don't think there's a crypto solution for that right that one's gonna be a social problem Okay, I said I would tell three stories, but this tells me I have ten minutes left. So I have room for one more This is a story about something called SNI and if you read carefully that says hello, your name is insert value here SNI is an extension to TLS and Here's what it's for. This is a really basic cryptographic principle that says you should mention the names of the parties you're talking about Original SSL and TLS client sends to the server. Hello to some IP address the server says back. Hello Here's a certificate proving. I'm example calm and the client says example calm. I don't want to talk to them I wanted yakker.org, huh? So that broke a lot And in the same way as zillions of other protocols from Needham, Schroeder to Kerberos and everything else get fixed by mentioning The name a little bit more the client should say to the server. Hello IP address I want a particular site and the server can respond with oh, yeah, sure. I've got that certificate available here I'll be that persona and I'll talk to you All right, so all of the places you know that are using many many host names on a small number of servers Like folks like us who are sitting on two to the 24th IP addresses We're doing that because most clients because some clients don't do this server name indication feature. They don't support SNI Okay, how bad is it? Well SNI got designed and introduced this problem got recognized in 2001. It's still not commercially viable Why not? It takes about five years before most clients actually start shipping SNI it takes another five years before Every browser you would plausibly buy and use has it when Android ships it in 2010 Three years after that the standard Python and Java implementations still didn't support SNI Although they both did ship it near the tail end of 2014 First weeks of 2015 it never made it in Windows XP and it never made it this one's going to be key into Android 2 2 Now Android 2 2 is old right if you're carrying an Android device and you're in this room The operating system probably starts with like a H or a K or something right? It's way later than Froyo But in huge chunks of the world if you go to buy a smartphone You are probably getting a device that still runs to this day Android 2 2 Why because in 2 3 the licensing terms changed and Google gets more control over what's on them So 2 2 is in some cases the last free Android right says a lot of folks out there still buying those phones This is what we saw across the middle of 2014 Now what happened right around the end of April in 2014 anybody know No, not harplead actually this is a couple weeks after harplead That's not harplead. That is Microsoft killing off Windows XP That is the expiration of service for Windows XP. It was courageous It really was if you look before that time like I fit a line over it But it's sort of an unhappy looking line with a lot of outliers, right? You see these things binging up and down all over the place and if anything SNI adoption is trending downwards How the hell can that be? Well That's that there were more Android 2 2 machines showing up on the internet then there were machines getting fixed But then Windows XP service ends and within seven days after that Huge numbers of Windows XP machines vanish from the internet now. I can't tell where they went right? I can just tell they stopped talking to me. So it might be someone pulled the power cable I hope this is what happened it might be someone pulled the network cable and they're still running But they're not on the internet and so maybe they're safe It might be that they got stuck behind some proxy So they're doing like TLS 1.0 locally and then they're doing something more modern outbound Could be maybe that's not the end of the world, but they vanished. What's also notable is that the variance drops enormously Why is that the variance before when you see these big outliers? That's where my sampling technology caught things on weekends and What we observed before that date is that people had wonderful cryptographic tools available to them on Saturday and Sunday and in the evenings But Monday to Friday their cryptographic tools suck. That's because they were using Windows XP often and that was the best tool they had often paired with really atrocious intermediate proxies Right and then after that we see the variance dropping enormously. There's still some And when I looked at this I pulled this data around October 2014 and I said, okay Thank God that the Microsoft actually did that now I need to get someone to do that to Android 2.2 which is open-source software. So who do I find to be the actor's the vendor for that? So we'd started some conversations talking about it those conversations went on long enough that we got more data And this is what we see now So you can still see that jump that jump that we just looked at is this jump Right and so before that all the way back to 2013. There's some sort of vaguely sinusoidal I don't know what this is. There's some annual cycle related to like summer vacations, maybe and Then even that damps out and sometime around January 2015 You start seeing something that you can at least pretend is exponential fall-off in the number of old machines that don't support SNI So we're winning it's up around 97 and a half 98 percent right now And and trending smaller and most of what's left there are the same things that I made fun of on the ssl v3 slide Right, so if we hadn't had the poodle experience, I'd say great We're almost down to 1% when it's hit hits 1% just turn off all the old ones. No one will care Maybe a little wiser now we've learned alright, we shouldn't do that But the neat thing here is this one's more server-controlled so this so you can do it Server by server customer by customer right so if you wanted to stand up an e-commerce site now You could have a perfectly profitable e-commerce site that only used SNI today right in the US now I hear that there are some user populations say those where it only comes in to upgrade the computers every one or two years Maybe they don't have this world yet right in that world. You might see 10% of the population can't do SNI The browser vendors have a slightly different view because they get telemetry from the browsers that tells them What what's happening, but big enterprises turn that off and don't send the telemetry right so there is probably a hidden Population out there of people who are behind proxy, so I can't see them not sending telemetry so the browser vendors can't see them and Taking no updates so they're exactly the ones who really need help for them. I advocate prayer Or replace hardware Okay from that What can we learn and what can we try to take forward? First problem Overwhelmingly people don't actually rotate compromised secrets even if the fact of the compromise becomes public So the mathematical model that says we're gonna associate a person with a long-term key and that's the identity Doesn't Hold right it's a useful model. It tells you some useful things But we're hitting its limits when we try to look at scale at how people are actually using crypto You knew people shared passwords. I'm here to tell you maybe you've done it yourself very naughty I'm here to tell you that there are people who are sharing cryptographic keys in exactly the ways that we don't want them to Second thing to learn protocols are gonna live for 10 to 15 years It took SNI and Heartbleed and poodle to kill off SSLv2 and v3, right? That was The worst crisis we thought we could imagine at the time So bad that even if someone walks in at one of the post-quantum sessions and says by the way I brought my three kilobit quantum computer. Let me tell you your RSA keys factors It will be 10 years from that date Before we can get RSA off the public internet at least Right because we had similar crises for these other protocols and we couldn't move we have better infrastructure now it's true, but we have more people and we have more weird old devices too, so the best model for the future is the recent past and I'm here from industry to say that the post-quantum Crypto world we need from you a decision. Is it super singular isogenes? Is it lattice-based? Is it code-based? We need something that we can start shipping and working with It's wonderful to see that Google and Mozilla and mine handful of others are playing at the edges But we need from the theory world to know now Because we have this terrible feeling the quantum computer people tell us that they're 10 15 years out Which means the time to start shipping mainstream browsers and mainstream crypto endpoints with post-quantum crypto in it is now It would be really good if we could do that this summer, so you're late Moreover keys will be shared between versions of the same protocol now the social hacks we can do on that right? That's not the next version of TLS. That's called quick instead. So that should use different keys Have worked half didn't work people actually shared keys between quick and TLS a bit too There's a Common meaning of uncompromised is used only in accordance with this protocol We have these composability frameworks that tell us where you can bend around that we could really use some more Theoretical work to explain to industry. What does it mean when I share keys between TLS 1.0 1 1 1 2 1 3 What does it mean when I'm speaking HCP over that most of the time, but sometimes something else And we're mixing those keys together. We could use some help in understanding that And the last lesson is Implementers will skip all imaginable checks and some that are even not imaginable We have this model of honest, but curious I'd like to ask for some help on people who are honest but flaky right people who are simulating a Protocol participant, but they want to send every possible message. They can and they haven't really done their homework first Cool Relatedly what checks could we make unskippable right in the same way that some of the a ad constructions do How could we make it so that you can't possibly get the secret unless the integrity check works and the more that we can build Protocols that way the safer we can feel about what some of these horrible horrible implementers like me are doing Thank you all very much for your time. I'll be here through Wednesday night I'd love to talk more and I'm really happy to take questions and then we'll go have some lunch