 So hi everyone, we're going to speak about the errors in certificates. This is my PhD and virtual thesis of Pavel and we are from Masarik University in Burno, and it's done in cooperation with Red Hat. Certificates are basically ubiquitous today, basically whatever site you see including the DEFCOM website is secured, TLS certificates are in place. So even if you don't handle certificates daily, you've definitely met certificates and you've worked with them. And usually it's the case that everything is okay, like the website displayed, but sometimes you get across something like this. There's a problem with the certificates, which you may or may not understand, depending of your level of proficiency and on the website that you are connecting to, whether you know that there should be a problem or it should be okay, but it's suddenly not. More often from your positions, you will not see certificate errors like these, but done in command line interface. So imagine we need to verify a certificate that happens once in a while. You're testing a system, you're a QA or a tester, or the system just crashed and you have got a certificate and you are kind of suspicious that the certificate is to blame. So we want to check the certificate yourself. So we have assert that we want to verify. If we try to do that simple as that command line, we may get something like this. Error 34 at zero depth lookup, unhandled critical extension. I mean if you're handling certificates daily, you probably know at this point what's going on, but more probably you don't have a clue. So what you do, copy and paste, you go to Google, open SSL, unhandled critical extension, you get to the open SSL documentation and you learn unhandled critical extension. Well, thank you documentation. It can be better, it can be worse. Open SSL is definitely not the only one to blame and the guys at open SSL are really trying and they've improved stuff on their own and on our prompts and if we start picking other documentations, we also find other funny stuff. In GNU TLS you can learn that the certificate presented isn't the one expected and then it mentions TOEFL, though it probably isn't the one TOEFL that came to your mind at the beginning. And the thing is that there's quite a lot of errors that can happen just when validating a single certificate. This is a list from the open SSL manual page and it's not complete. If we went further on, it would end somewhere over here, but I was just unwilling to go with a smaller font because then you would have no idea what the errors looked like. You might think that, well, certificates doesn't matter that much. Well, it does. Recently you might have noticed there was a CVE for Microsoft error because a product of the Microsoft crypto API was validating elliptic curve certificates in a slightly incorrect way and the slightly incorrect meant that if you delicately crafted a certificate, you were able to create a certificate officially trusted by the Microsoft API on whatever you liked. And the example shows basically you can do ransomware that's digitally officially signed by Microsoft and Windows totally accepts that. So validating certificates gets kind of crucial at times and we should really be able to get it right. Now, we don't want to just keep complaining about all the things that Martin has just said. So we decided why not improve the situation and make it better, make it more usable. So the obvious way to make it better would be to make some pull requests to the official documentations to keep adding the missing documentation and we agreed that that's the obvious way but we decided a bit different approach. You might have seen our booth here at DefConf and we have created a website called x509errors.org and it was actually, you could actually see it here at DefConf at our booth. So what's it about? We decided to list, we decided to create a web page that lists all the possible certificate validation errors across different libraries and make websites to create documentation that could be more usable for all. So we needed something to base it upon. So we took all OpenSSL certificate validation errors and because OpenSSL is by far the most widely used TL library today so we based it on it. And then we split the error messages into a few different categories. So for example, one category could be the trust or chain related errors. Each category has some description, for example, trust or chain related errors are errors that happen when the building of the chain up to the root fails somewhere. So you can find this description here with the category. Also some relevant links to some useful resources such as RFCs. And then you get to the specific errors. So when you click on a specific error, you get some detailed information about it. What we managed to do is that for many of the OpenSSL errors, we've created an example certificate which means that we replicated the error message in such a way that we have a certificate or better said a certificate chain that upon validation gives exactly the one error that we want. And that can help you as a developer for testing purposes but it also helps us because when we have this error and this certificate that gives this error upon validation we can use it to run validation in different libraries and see what error messages are equivalent to this one error message in different libraries like in TLS, Bolton, NSS and so on. And so we started mapping together the error messages from these libraries to see what are the equivalents to help us do that. We created a table like this. We call it a mapping table. And on the most left column you see the list of all OpenSSL errors. Now the ones that are green mean that we have already managed to replicate these ones so that means that we have the example certificate for them and we can compare it to the other libraries. Now the other columns represent other different libraries and when the error message is green that means that we've successfully managed to map the error message to its equivalent in OpenSSL. Now from this graph you can see a few things. The first thing is that we still have a lot of work to do. We've managed to replicate almost the half of the OpenSSL errors. There is currently around 77 errors in OpenSSL. Around 10 of those are marked as deprecated in the official documentation so they can't really happen. And we suspect that there are still a few of the errors that are not marked as deprecated but we suspect that they can't really occur in the wild. So we are more like close to the two thirds with the green line but I agree that there's still a lot of work to do. Now the second thing you can see in here that there is a massive difference between the total amount of errors between the different libraries. So for instance in GNU TLS there is like a quarter of the amount of errors as compared to the OpenSSL which will mean two things. Either GNU TLS doesn't do that much over control. For example it doesn't control all the things that OpenSSL does so that might be one issue. Or the second thing could be that OpenSSL has a difference between these errors in a much finer grain so more OpenSSL errors could be mapped against one GNU TLS error. Anyway we are still improving on this and we are adding new libraries like NSS, MBIT, TLS and so on. And this. Now back to the scenario that Martin has started. You found yourself in some documentation. It doesn't matter what library it is but it doesn't help you much. So imagine an alternate world where there will be another one line into documentation linking you to some website. You click on it and you find yourself somewhere here. And there's a detailed explanation of what the error message means and what should you do. Now this might not only help you as a developer but it might also help the developers of the TLS libraries themselves because they don't have to write the documentation themselves. But what else should such a web page contain for the developer to find it usable? Basically going for the next steps that we are planning. Of course the obvious one is to when their website is presentable and contains enough information to reach back to the developers of the libraries and offer them, oh, we've tried to comprehend all the documentation and map them together and we see that GNU TLS has a good bit in their documentation on unhandled critical extension and we would like to add that to your documentation and basically in an ideal world of course. Kind of unified system. The pity is that the changes would be compatibility breaking but imagine the world where all the libraries would give you this very same certificate error message so no matter what library you are using, you just know the errors even if you switched from the previous one. Which is actually something that we plan to do. We would be reaching to the developers of OpenSL, GNU TLS and SS roughly in February. I've already talked with the main developer of GNU TLS and we've communicated with the OpenSL guys previously and in the meantime to improve the website we are adding more libraries as Pavel said adding more example certificates. Other than that, in parallel we are trying to get some sort of statistics for the individual errors. Running an analysis of the IPv4 scan basically all the certificates that you can reasonably get looking at the internet, things like census data sets and rapid seven scans to find out how frequent these errors are because as you've seen OpenSL has a lot of them compared to let's say GNU TLS and maybe some of them really do occur often and those are the ones where focusing on them would be a priority and maybe you find others that occur almost never at which point the sheer volume of the number of errors might be confusing so maybe if we have similar errors that never occur or almost never we could merge them into something that occurs a little more often and does not create the volume of about 80 different error messages that you need to understand if something happens. So this is a point that we want to add to add some sort of a statistical dimension to how much this happens in the wild. A second thing as mentioned earlier we would like to create a better documentation for the individual errors because it's still better to have one thing that tells you what's the problem what should you do, what are the security consequences directly instead of having the six, one, two liners of the existing documentations that we are compiling at the moment on the single place. That is also an improvement from the current state of the world. We would like to pose this better documentation just on the existing documentation as it is in different libraries because it turns out that sometimes in... I'm not sure whether this is not the case really sometimes a documentation in GNU TLS on the issue says a rather different thing and explicitly offers you that this might be the problem because you have a version one certificate instead of a now commonplace version three certificate while OpenSSL documentation just talks about the extension and doesn't offer you this is usually the case directly in the documentation. Furthermore, we have set up a research booth that many of you might have seen at the beginning of the eBuilding where we are asking the developers what sort of documentation would they like how long should it be should it have a dedicated section on the next steps should it have a dedicated section on what are the security consequences and that's why we created about a 15 minute long questionnaire that's still available, the link will also be on the last slide where the developers are shown different variants of documentation for chosen certificate issues that we've selected and they are asked whether they like the first or the second one more what they would add, what they would omit, what they would reformulate. Apart from the questionnaire and the existing documentation there may be a cooperation with the Red Hat Technical Writers Team to polish it up in a way that the developers are kind of comfortable with so that it's not a couple of us academic guys basically saying oh, this is a batch of documentation but we have it underlined with data from real developers thinking that actually they do want a longer documentation not just the two-liner even though that would be intuitive to me we have learned in academia and research that intuitive solutions are not necessarily the right ones and we should kind of ask the people actually using it to find out basically summarizing the goals of the whole project that we have would be to provide a useful resource for the IT professionals with examples that can be used for testing and for getting a hang of the error and the documentation so if something happens to you and you get an error and you don't understand what it is or whether it's severe or not you have a single nice place that you can look to and one of the previous researchers of ours has shown us that when we try the little heretical thing that the error message in the command line itself set O error at zero depth something, something, something comma CX509errors.org for more details at least to me surprisingly three quarters of the developers clicked the link that they were given in the command line or in the manual page which means that if you are the developer of the library and you provide the user with a quick link to a good resource they at least try whether the resource is good so there's a huge opportunity to kind of direct the developers to a place where a good documentation will be to be found which basically brings me to the second and the third point as said secondly we ideally want to create a better documentation because we will ask all sorts of people and all sorts of libraries what they would expect for a particular issue and then include all those things and thirdly we would save the time of the developers of the libraries itself because they would not have to write the documentation for the individual errors word of mouth says that developers rarely like writing documentation and explaining the stuff so they might actually appreciate the thing now the talk went a little bit faster than we expected that basically concludes the talk the project is online at x509errors.org by now already for a couple of months if you're interested and the survey that I mentioned that was administered at the booth again still available at survey.fy.muni.cz.fconf it will be open till the end of today till midnight though if you manage to fill it in in the next about an hour roughly till four o'clock when the last session starts we are still at the booth and you will be able to come and claim your reward of the red hat and the faculty of Informatic Merchandise that we are left with we have been giving away for the completed questionnaires which still leaves us with a couple of minutes of questions so general questions or any comments for how should it work better or why won't it work or any feedback is welcome by now yes okay glad to hear that yeah there's definitely that that's a good point when you use a link as prominently as this there's definitely a question of who is actually in charge of the website so far as the current situation is I wouldn't recommend putting the links in there right away because it's us a couple of students and the faculty of Informatics University administering what's there which is not necessarily a respectable source maybe if the if the libraries and the developers like the idea and the the sub project or something like this goes under the wings of red hat or whatever of other company it would be a better option but as you said again phishing is a problem or whatever sort of spoofing when you're going for the link basically telling do avoid the certificate validations all together as 90% of the Stack Overflow posts tell you that's definitely a good point that was I mentioned the link because we tried it in the previous experiments two years ago here at Defconn with the research booth I was quite curious myself because I kind of expected the command line suited Linux guys not to click on things that are displayed in the command line and I was rather surprised myself that they gave the link a try and since they saw in the first case that it seemed useful they also tried in the latter cases though I do perfectly agree with you that I'm not readily available to convince other people to put links everywhere though doing interviews with the developers as we did two years ago it was interesting to hear that opinions very quite a lot there were developers saying oh there should be a YouTube link video in the manual page that would explain it to me while the developer setting next to him at the same time was like kidding me I would never play a video explaining an error to me even if it was linked in the manual page so we've heard all sorts of ideas thanks for the comment any more comments questions yes so if I get it correctly you kind of aim at broadening the scope to the TLS connections as itself not just the certificate validation part that could also work and of course if we are going this kind of unifying standardizing direction we could have whatever breath of the scope we want ideal case let's standardize the whole API for TLS connection across all the libraries that would be a nice world though probably unachievable we were considering that though in the beginning we decided to start focusing more narrow and then if it turned out viable if it turned out that the developers liked it and it made sense to them maybe then broaden up because we were ensured that the proof of concept of this sort when it was like a year or two ago would work and the developers would find it useful and it would make sense but still focusing on the certificate validation sub world was sort of self contained so basically I do agree that that would totally work and we just decided not to do it yet so as not to have a too bigger portion of the pie so that we would be unable to eat it yes, do you mean doing the static analysis of the TLS libraries like OpenHistagno TLS or of the client software that uses them? the ladder that is also something that we have been considering though only slightly because it's a rather different sort of research with the requiring the expertise in the static analysis and all sorts of stuff like that we may come to that starting from the IPv dataset scan analysis if it turns out interesting and there are students interested in that basically since it was me in the project for quite some time and now it's like three to four of us that was not a priority yet we've been considering that at the beginning but we thought that this would be a simpler and more viable start since we wanted to do something that we can then ask the developers to ideally push upstream rather than starting a more complex thing with the risk of not getting the changes upstream and not getting the changes used so basically a similar answer that was to the guy on the left of mine but definitely thanks for the tip because it works to me as a signal that something like this might be usable and might be interesting for the developers with that I would probably close even the QA section as I'm told that I'm out of time thank you all for your time and appreciation