 Okay, good morning 25 minutes, right? Okay. Okay. I can be reached either the addresses below one forwards to the other You can guess which way it goes To the main point as in it's not the crypto that's going to be the weak point of your security if you're an open source crypto project It's not the crypto that's going to kill you or the project The project will die for other reasons My main point here in the talk is to talk about how open SSL came back from the dead And what some of the reasons were and what some of the pitfalls we encountered and hopefully how the rest of you can avoid them The history of open SSL can be divided into a few historical eras Pleostocene, Mastodene, whatever and I'll go over each of these in the next couple of slides SSL SSL EAY There's various creation legends about how it came about These both happen to be true here Eric young EAY hence SSL EAY. Hey, I've got a DES package It was a lot of parts of it were written in assembler on the most common x86 platforms It was the fastest one around it was adopted into Kerberos or other people's Kerberos implementations And I've got this big number math package. What should I do with it? His colleague Tim said hey, let's do SSL is this new security protocol out of Netscape And that's how it happened. It was two guys in the garage more importantly. It was two guys in Australia Which was free of the long arm of US government export regulations and patent office There were a small set of folks sending in patches Groups I worked with at the open software foundation OSF now stands for something else Did it there were a handful of tests a minimal functionality sort of had a run on by hand But it worked The next year it was open SSL the rise of open SSL Tim and Eric got day jobs So they were gonna have to leave the package for various reasons a new group started up There were about many as a dozen members folks some key with Apache Major motivation was export control again stay away from the US We have seen So this is another instance where you short-sighted US rules Hurt security right At one point the team was scared to even land on US soil Okay Active mailing list they still took patches The quote on the bottom here was typical of what you'd see in the in the real world or the IHF world The actual quote I had started out with the letter f dash dash dash But we have a code of conduct and I was advised not to use profanity We open SSL the fall of open SSL the project was more abundant Releases were not pre announced. There were no documented policies. The source code was complex and arcane If you looked at it, it would make your eyes hurt the braces I Thought nobody else coded this way It turns out there's a it's a style called white smiths. I think Wikipedia claims the code was complex and arcane Everything was done via tables of dispatch function pointers a lot of it still is not all of it but a lot of it still is and You just looked at this code and it looked like nothing else we were ever used to it was hard to maintain the policy of SSLE a y and open SSL was For a long time if somebody gave me a patch and it didn't break my system we'd take it There were seeds for a big failure in that philosophy which we will talk about on the next page It was hard to contribute us export regulations again interesting story asked me at lunch sometime how I My contribution to open SSL ended with me up being on an ISIS kill list I'm sorry The main developers were overworked and overcommitted. This was over course of a period of years It the team dwindled down some folks went on to other projects is to be expected some folks day jobs got more involved as this to be expected and the two core developers In order to make money They did a lot of FIPS work Being able to get FIPS certification. I guess it's actually called a validation Again, it doesn't matter They had to take on this work FIPS validation is important to a large number of people They want to be able to use crypto and they would like an open source package The open source code Was a nice the first open source source validation. It was a key thing It took a lot of changing of people's mindsets to understand how I could validate a package as a Validate a clump of software as opposed to validating a hardware token But it happened. It's still going on It's just stopped right now. It'll come back the project donations were minimal I Don't know who wrote that expression in talent that I know who wrote that expression parentheses at the bottom of the page it was my native English speaking Aussie friend Under $2,000 a year right as opposed to sub-USD 2000 per annum So me very very minimal commute very very minimal communication. So we have you know the or the origin the rise of fall This is what the thing looked like This is all the his the top line is the history of all commits with an open SSL from the beginning To now Yeah, and the highlighted graph there shows the two years before the event I Oh, my open SSL his career to the event and You look at it and Steve Henson 448 commits over a two-year period Andy Polyakov Steve Was the main developer of and took over of the maintenance of the protocol code the asm one code all sorts of things The FIPS architecture Andy Polyakov dot asm Is an assembler whiz? People from Intel would send him Optimized and tuned versions for various crypto algorithms using the latest Intel assembly language Instructions and he'd send them back optimized versions He would also You know cause we were all in awe and nobody could read the code if you look at the pearl asm stuff again Talk to me catch me during the week You'd see what looked like a pearl script generating assembly code And then you looked and saw that each operand was actually a subroutine call to actually generate the real code It's pretty obscure. That's but there were occasional bugs But anyhow if you look at that that averages out to you know Barely a commit a day So what happened? Why was this this fall? There was a long learning cycle to understand the code Everything was done via an extra level of indirection for example the Threads package originally done by my team in OSF Was ultimately implemented as a runtime callback thing It wasn't part of open SSL, but you couldn't say you could create all these callbacks and at runtime say here's The callbacks map to P threads Which and that persisted for a long time kind of stupid or the error system Here's the callbacks that implement the error codes And if anybody else wanted to implement their own package and storage of error codes They could do that at runtime similarly for Malik, right? So everything that's how I talked about every those tables of function pointers It was way too much to understand you'd sit there and trace through all the way and you go Well, I think SSL 3 server handshake is the function I want but now I got to find where the bird is in the right place The group needed to get consulting dollars FIPS work to keep the project alive Otherwise these guys would be out on the street or mother's basement or or whatever it was, right? No time was spent on building the community Not very little. There was no time There was no ability to make an ounce and to keep to any kind of plans, right? Here notice that says okay. We got the latest release of open SSL is out it fixes these vulnerabilities With absolutely no notice. No idea. These are important. You have to go track down and read everything No idea when the next release is coming out those in the early days Pardon me who waited for a year for SSLE why oh nine eight to become open SSL one dot oh dot oh But all of these things added up to a state arc attitude If I don't tell you anything then I can't or promise anything Then I can't be held to account if I don't make the deadline And also because I don't know where my next paycheck is coming from. I don't know what the deadline is going to be Okay, the CDE that must not be named Right Any other times when I've given earlier versions we've given earlier versions of these talk We ask people who knows what 2014 oh one six oh means a couple people might raise their hands, right? Because we're an OCD kind of numbers oriented field People do know what April 3 means, right? rekeeting entire Internet There are a couple things that were noticeable about Hartley one it was a ridiculously simple bug right You sent the ping packet And you said how many bytes to send and it should have been how many bytes were sent but it wasn't so sort of We're just send whatever data happen to be there It was the first real Security CV well first real CVE that had a logo a name The and it made the mainstream press. I don't know who Riva is But I think the industry she's probably moved on but the industry has not yet fully moved on from Hartley, right? This is the daily mirror Fine fleet street publication. Okay. Oh, she was a song. Okay. She was shot Okay shot and killed. Okay. So all right. All right. Well, I'm glad. I hope she finds peace No, seriously, man That would stink If you had to see your daughter anyhow But the fact that it made the mainstream press made everybody more aware people were getting calls I'll change my passwords. There were plenty of other front-page headlines The last front-page headline I think was when the New York Times above the fold talked about the Blikenbacker attack But more importantly the fact that it had a name was really useful Because you could say part bleed shell shock Poodle and It's a lot easier than you know having to translate to a name if I said see me 2014 015 oh Maybe I had it off by one hour, but maybe I meant something else. So that was really useful It also gets really tedious because then you got to get the domain name And you got to get a graphic artist, you know the or or Microsoft Word art to create your logo But the name is useful Okay, I encourage researchers to come up with names not websites and logos And the folks at Inria are really good at coming up with names or use Okay, we're covering So now we enter the post-heartbleed phase php This the core infrastructure initiative was created it was created both because of the open SSL heart bleed bug And it is also funded and supported open SSL very well So it's sort of this mutual synergistic thing We are both the progenitor and the poster child for what can happen with things are done, right? Donations jumped so we had four funded staff members now They didn't have to do FIPS work. They didn't have to go out and stay on the street corner go crypto go Six months later So about four months after heart bleed, I was invited to join the dev team In October we had a face-to-face meeting funded by the CII This was really really important and useful. And so one of the lessons is try to get your dev team together We're still geographically dispersed Australia Boston New York area and then scattered all over Europe right now. I'm American. Sorry Europe is just one place But we did a lot of important things a lot of us centered around governance We wrote a release we came up with a release policy. We came with security policies. We posted them on the website We came up with a coding style At this point, you can't quite tell in the room there, but we had like 12 people in the room for three days and we managed to come up with a consensus as to where to put the curly braces That's pretty amazing. We socialized we went out for beers We sat at the table next to lunace Torvald who always kept his back to us We don't know if that meant anything poodle helped at the third day of our face-to-face the poodle vulnerability was disclosed because someone Notification that there was this thing and someone else reverse engineered and figured out what it was So it went out a week before we were expecting it to and so if you got a bunch of people in a room It becomes all of a sudden now a war room We were working and trying to get the secure fix out and make it as good as possible that also helps so having a beer helps Having a crisis to address helps Just being there in the same area helps I'd recommend the beer over the other two Going to meetings is part of recovery right anyone who's been involved in a 12-step program knows you got to keep going to the Meetings so keeping up the cadence. We met again last October two years later Now there was a lot of discussion. We had a much bigger agenda Still haven't published all of our notes and conclusions some of them are on the blog We had we went over spent a lot of time going over the CVE notification process If you have your own list of who to notify or pre-notify for CVE's Everybody will want to be on that list It's really really hard to come up with this a A bright white line to separate those who get to know from those who don't We empathize with those who don't get to know We use the Linux distro mailing list All right maintained by Sonic wall Because what we want to do is feed the top stream and then feed out we also allow in many cases employers of of Team members to have advanced knowledge not not always Okay, we had a lot of discussions about and with the CIA the Linux foundation the Linux foundation Is the secretariat and runs the CIA? We discussed how to grow the team. It's really important to us that we get more you know We seem to have a sense of vibrancy Going on now and it's alive. We want to grow the team. How can we get more testing? That's actually now that the code seems to be a in good shape and be we have a plan for keeping How can we get a better testing scene? Nobody wants the right tests? The one place I've seen it is in Japanese companies where they they feel happy when they found a bug because their test found a bug Yeah, well nobody we've updated the roadmap and platform box and we're working on a regular release cadence That's informed by the release process and calendar of what we think large enterprises and various other organizations do Part of the reason for doing this is we want to get people off the old releases We want to be able to make it a planned thing so more transparency Have more beers and try to make things predictable, right? Predictable expected Not surprising Boring is a good word Okay Transparency as I said before for a number of years you post to the mailing list and maybe another mailing list poster would respond Maybe somebody you know if you've been around for a while, maybe somebody on the team would respond But transparency is really really really important Building community Document what we do how we do it why we do it the website was overhauled It's still got way too much words on it because the initial authors of that content were Swiss and German And their language is much more formal particularly when they're writing English than say mine They won't let me put F-bombs on the website But you know I would the mailing list were moved. Why does this interesting because we were previously using something called major? Domo, which was created in the 80s not mailman, which everybody else uses We sped up the RT approval process We have multiple moderators and then ultimately we killed RT because nobody really uses it and if you But we started I think a virtuous cycle When a project isn't a black hole people contribute when people contribute They see the project isn't a black hole and that you get this really nice feedback loop code quality Another part of the recovery Appearances count we have almost repeatable code reformatting Amelia on our team is trying to get that to happen where every check-in could be run the reformer and you'd get the same thing Mandatory review by second team member the team the reviews vary from yeah, okay, like if somebody posts Assembler code. I'm like, yeah, sure for IBM said it works on their platform great. Here's this I don't know systems the assembler. I'm never gonna know systems the assembler, but sure So we have sometimes where it's done pro forma for process sometimes where people get really down and into it Now that we're doing everything on github We are starting to see more people from the outside community involved in the teams Pull requests, which is good We're still we're still working to improve the quality of our reviews isn't great Isn't as good as we want it to be That would have caught Hartleid Right that will actually take it. That's the first part that would have caught Hartleid We're doing more tests. We run coveralls As part of a regular continuous integration. Oh, we have continuous integration now We had a build form maintained by a guy at Cisco. We have Travis. They've upped our Quota so we can do more builds one of the nice things about open SSL is when you go to a provider And you say hey, would you give us free something? They tend to say yes So our thanks page has a lot of people, you know Akamai CDN rsync back rsync.net backup Travis increase build things Amazon Web Services CPUs to generate more fuzzing data And so on so if you have something that you think could help open SSL see me If you don't have hard gold, you know sacks of gold give it the RWC computing facilities talk to me These are all modern practices, but remember open SSL is old right it goes back to the 90s It's no longer dumping ground for everything as I said the policy was oh if you gave me a patch and It didn't break the system that I was building on Okay, we would we would take the patch don't do that anymore We are Removing old week ciphers. We've moved the gossed Russian cipher out to a separately maintained engine and so on most structures are opaque I'm gonna have to talk even faster. I'll skip the recovery the one thing the test tools tools are not as great as they should be Somebody changed the shell script to remove 10 lines and the code coverage went down a Config shell script. I don't get that We went through all of our tick art RT tickets This is post-heart bleed the first big drop is when we closed off a whole bunch of bugs The second big drop there is when we said look anything more than two years old We're just not going to get to we sent mail to everyone told them we're close each person told them we're closing it Reopen it. Everything's done on github now So what's happened on github? Here's the metrics that Kenny had asked for when I first submitted Almost 4,000 commits this year. We have about 430 users thousands of forks. I think like 2,000 a Lot of new issues. We've put out eight Three releases on the mainstream one was a bug that we didn't check We introduced a bug in fixing B. So we had to put out C. I'm pardoning put out C 12 releases and 102 101 we put out the last release and it is done We understand long-term support issues. We've created one or two was a long-term support issue But we had We got paid to fix some CVs in 098 But they're never gonna appear on our website CVs we had nine high a high root a high one forces a release a 20 medium might force a release when you get enough mediums and then put out a release 28 low Or we just fix it the notification process seems to be working. It doesn't meet everybody's needs There are infrastructure. There are internet companies that could reasonably claim that they should know Sorry we mostly met the disclosure and fixed deadlines We slipped on one time and that wasn't because we were trying to get the right fix It was because of manpower allocations And when you're working with open source Stuff you tend to have interesting personalities, right? And so someone will be they'll just go dark Pounding and pounding and pounding on them and then finally they'll wake up and go. Yeah, here's the fix Here's the test. Let's do it. Thank you. We've had no critical yet critical is receding to that The current GitHub activity. I'm just gonna point out the highlighted thing at the end They would chose, you know more in the past year Year and a half then almost the entire previous history Excelsior the New York State model onward and upward Everything is done on GitHub now. Everyone has a CLA. We don't take contributions other than that and the major technical debt is Has been and is being addressed Native thread support there's no reason to make it runtime switchable The state machine that handles the protocol has greatly revamped We don't do open coding of the TLS messages anymore. If you look at the current code particularly in master It'll look like function a series of function calls that basically resemble the TLS presentation language It also looks like the crypto bytes And I forget the other out the other data type in boring The CLA flags have improved the help has been improved all the documents are improved Our goal is to Within the next year document every SSL function. Here's what's coming. The FIPS work is funded, but it's put on hold For TLS 1 3 TLS 1 3 is the focus of the next release. It is the main deliverable in the next release It will be binary compatible with the 110 release We have a contract we open a cell have a contract also we Akamai Committing open SSL to a fixed delivery date part of the No surprises We'll have the details announced soon, but First quarter where first quarter may be four months long. We'll have TLS 1 3 interoperable with Firefox Chrome Facebook pick bunch of servers We're moving to Apache v2 license hence need for CLA's we're doing more testing. We can already run the boring SSL suite We're planning on adding the Kerberos test suite and the Python PY crypto test suite We can run external test as part of our stuff Other things that should come these are not in the public roadmap, but they need to be fixed All the as I mentioned documentation the random number generator anyone on Perry's crypto list saw a number of threads swan by like What can open a cell do in the real world? I don't care what the definition of entropy is I just want to be able to generate stream of random bytes so that I can generate key material There's almost an answer in there We'll see Generic store facilities so that you can actually build a portable open SSL trust store and portably use it in all of your applications We want to do that With that I'm just like two minutes over. So thank you for your time you