 Hi everyone. This is a talk of the academic cooperation of Red Hat Research and Masary University in Bono. I'm doing my PhD in cooperation with Red Hat and Pavel is doing his bachelor's helping with the research. And we'll be talking about TLS certificates, the errors of TLS validation and the associated ecosystem. The talk will be, or we'll try to do it a little bit interactive. So there will be three polls in it and we will want you to vote during the talk so that we have some inputs from your side as well. So let's start. You're browsing the web. Oh, sorry. Did it work? Yep. You're browsing the web. And most likely you're doing a secure connection to the website. When you click the lock icon, you see that the connection is secure. You can see the details of the certificate. All the signatures valid and stuff like that. What would it be like if the connection was not secure? Well, you would see something like this connection not being private with all sorts of information and an error. This can happen not only in browsers, but anywhere when you handle certificates. If you have a certificate and try to validate it using OpenSSR in a command line, you can get something like this. Error 34 at zero depth lookup, unhandled critical extension. And I guess you know what that is. You probably don't. Well, believe me, I've asked a lot of developers and it turns out that not many people know from the top of their head what unhandled critical extension is. But we don't worry. We have Google for that. So, Googling, unhandled critical extension, OpenSSL, what do we learn? Well, unhandled critical extension. Thank you, Mr. Recommendation. That wasn't particularly helpful. But I don't want to be picking on OpenSSL because other libraries are not ideal either. For example, BlueTLS has an error saying the certificate presented isn't the one that you expected tofu. And it doesn't mean this kind of tofu. So again, you're left with wondering what the tofu means if it's not the piece of food that you will not have afterwards. Which brings us to the first question that we want to ask you. If you were to guess how many errors related just to certificate validation, do you think there are in OpenSSL? Try to vote in the poll section on the right of the hopping screen. I see four votes so far. Those 30 people in the room, so we'll give it a couple more seconds. Mostly going for 20 to 40 or 60 to 80. Though I see that nobody thinks that there's at most 20 errors. I can definitely say that there is more than 20 errors. The reality is that, well, one person thought that now. The reality is that there's 78 errors just related to certificate validation in OpenSSL altogether. And that's quite a lot if you want to browse through them or they need to find something. Or if you are debugging or testing a software of yours that uses certificates and needs to handle all these errors properly. The situation is rather different in different libraries. OpenSSL is one of the more error-based library with GnuTLS and MBAT being about 20 and NSS having actually over 100. Though the precise number is complicated to guess because the errors overlap. So I've kept it at 100. We've done experiments here at DEF CONF in 2017, 2018 and 2020 regarding the usability of OpenSSL and the certificate ecosystem. It turned out that even though people think they succeed in using OpenSSL, they were tasked to just create a self-signed certificate and 87% of them thought that they managed to do it successfully. It turned out that only 45% of them did. And the general usability of OpenSSL was seen as rather poor. Next year in our experiment we have shown them errors of certificates of five different errors and we asked them how much they trust the certificate with the particular errors. And it turned out that the opinions were not what we would have hoped for as security engineers. That brought us to thinking what can we do about it. We have decided that there is a way for us to help in this equation. Click. As a first part of the process, we have created a website. We have made it x509 errors. And the whole idea of the website is that it's supposed to be a guide for developers working with TLS validation. I will now briefly cover the key features of the website. Click. The website lists errors from four different TLS libraries. Those are OpenSSL and the TLS button and new TLS. So whenever you work with certificate validation in any other way, and you encounter some errors, then you can find the error here. The errors are also grouped by categories depending on how the specific error can occur. So we have, for example, trust or chain-related errors that happen during the building of the path. We provide relevant links to various documents. For example, RFCs or manual pages or the official documentation, as we think that obviously we cannot show you everything, but we can point the developers to the right direction. Click. Each error on the website is clickable. When you click on it, it shows you some additional information. First, it shows you the original documentation that you can also find in the original library documentation, but it makes sense to list it here as well. Next, for many of the certificate validation errors, we have created an example certificate or rather an example certificate chain, which is a specially crafted mouse-form certificate chain that yields this specific error when the certificate validation code is run on it. And this can have various use cases. First, when you encounter such an error, and you might not know what it actually means. So looking, for example, on the way we generated that error could help you where the actual issue lies. Next, for us, this has an advantage that if we know that some of our mouse-form certificates yield, for example, in OpenSSL, the error unable to get issued locally, then we can try and run the other library's certificate validation code to see what error the other library yields and click. With having this, we can create kind of a mapping between the errors among the different libraries. Now, the mapping in many cases is really obvious. So, for example, if you have a revoked certificate, then arguably you will have some kind of an error message in each library telling you that the certificate is revoked. But it is not so in for many other errors. So because of the fact that some library has more errors than others, then it must mean that when we have some error in one library, it contains a set of errors in our library. And this can help, for example, when you transition from one library to another. Say you are working with OpenSSL and you know the error parameter violation, and then you switch to embed TLS and encounter an error called that search not trusted. You have no idea what it is. So, you can see that one such error corresponding to this error could be the parameter violation. So, I have yet not mentioned the maybe the most important feature of our website, and Martin will tell you more about that. Moving on from having a comprehensive resource about certificates and all the errors that can happen in four different libraries, we are thinking about the documentation. Now, a second poll comes. How long do you think the average documentation for a certificate validation error is? And I mean, in general, we took the commonly used libraries, OpenSSL, the TLS button, embed TLS, WALT, Polar, MS, crypto API and stuff like that. Try to estimate how long documentation is. Again, we will leave a couple of seconds. The poll is on the right in the Hopin interface. So, there seems to be some engagement. The majority of you think that it's between five and 10 words, even though there are people who think it's five words or less, and those who think it's 25 words or less. Well, the reality looks a bit like this. It differs in between the libraries again, but roughly we can say it's 10 words or less. The only positive exception is the Microsoft crypto API, whose average documentation is astounding 16 words, with the lighter blue being average and the darker blue being median. So, the median is roughly slightly lower, which means that there are certain errors for which the documentation is much longer because they are probably much more complicated. And coming from this idea, because we find 10 words to be insufficient to understand and to be able to explain what, for example, Unhandled Critical Extension means, we started to draft a new documentation. And at DEF CONF 2020, we did another experiment when we asked 180 participants that stopped at our booth to evaluate two documentation versions. One of them was the original version, as you've seen previously, for the Unhandled Critical Extension, the very, very, very short version saying just Unhandled Critical Extension. And our documentation that was maybe even overly long, but we wanted the second extreme so that we could discuss where in between these two, the ideal documentation in the mind of the developers would be. This is the actual Unhandled Critical Extension version that we proposed them next to the original one. As for the results, the new documentations, the versions that we created, we tested it on three different errors, was shown to decrease incompleteness, ambiguity, and inconsistency from the self-reported opinions of the developers, though slightly increased in bloat in Tango. This was kind of expectable because it's much longer than the original one, but I have to stress that they have seen incompleteness and ambiguity as very much decreased while only a couple of them thought that it slightly bloated. The new documentation increased the understanding and the subjective satisfaction and the perceived helpfulness of the documentation as well. Asking about the form, it turned out that the developers wanted just slightly shorter documentation as the ideal length compared to what we had. Ours was roughly 27 lines. They wanted about 22 to 24. But they put a focus on the structure of the documentation, that if it's well-structured, it can be quite long. Overall, they were generally satisfied with the structure that we provided. They looked like this. The error called a very short summary and then four sections of explanation, security perspective, what to do and consequences. Although in the end, analyzing the textual qualitative opinions of the developers, we decided to merge security perspective and consequences into a single section for a slightly easier and lighter structure. Then for an overall comparison, 89% of the participants in the Compt 2020 study preferred our design to the original one, to the short one. Which brought us to creating these new versions of the documentation and putting them into our comprehensive website that lists all the kind of knowledge that we have in the ecosystem of certificate validation errors. At the moment, we have the new documentation version just for six errors of OpenSSL, but we are gradually dividing more also as part of the academic attempts at the university students of security, teaching them how to write documentation that explains what is necessary to decide what is necessary to include what is necessary to exclude from the ideal so to speak documentation. Lastly, I will present what we are planning to do in the future. Click. As Marcin has already said, we plan to include the redesigned documentation for as many errors as possible because it is the key feature of the website. We also plan to add more libraries into the mix because we still don't cover many commonly used libraries, for example, Java libraries that are commonly used in number of devices or NSS as well. And we also need to work on the example certificates as well as we still don't cover all of the errors with them and it's a bit tricky replicating each of the errors. Click. Recently, we have also started research on analyzing huge datasets of certificate chains that are available on the internet and analyzing basically the prevalence of the individual errors on the internet and the expected outcome of this is that some of the errors are much more prevalent than others. On the other hand some are extremely rare or even non-existent in the wild. So there is a question arising whether it could make sense to merge the extremely rare error messages. Next. Lastly we are still talking about certificate validation errors but we can all agree that the most likely source of X509 errors is when implementing some kind of an application that connects using TLS to some server that detailed up-to-date source of information on this topic is kind of missing on the internet. There have been some attempts, for example by the Federal Security they have written very well written useful documentation precisely on this topic but that was like in seven years ago and it is now outdated and doesn't cover all the necessary features needed to provide a secure TLS collection for example revocation checking quick. So we aim to make this resource we aim to create such a resource for various libraries and we aim to make it as detailed as possible covering all the necessary aspects of secure TLS. We also plan to link the all the relevant documents here because if I'm sure the developers can find additional info there we also plan to link the all the relevant documents here because if I'm sure the developers can find additional info there quick. And in many problems in some problems in TLS there may be various alternative approaches so for example the mentioned revocation checking can be done multiple ways by the traditional certificate revocation lists or by OCSP or OCSP statement and each of these approaches has its own pros and cons and generally revocation checking is very tricky to implement correctly so we would like to also list the alternatives with the implementations list the pros and cons and explain when it is useful to try one approach and when to use the other approach and that's all from me. In the next section what did we have here we presented you an industry academic cooperation helping PhD students and bachelor students and master students to do their work and do something that's academically interesting but also which has outputs that the industry benefits from and we are aiming to improve the certificate validation ecosystem or the situation by consolidating all the errors libraries in one place we created a website for that website is already online actually for more than a year we are gradually adding stuff in it and now adding better and redesigned documentation for the particular errors with example certificates and mapping of the errors across multiple libraries in the future we would like to add more examples and more improved documentation pieces and include more libraries as well as client programming guides error occurrence statistics there are many more ideas that will hopefully come if the project continues as the academic-industrial relationship which brings us to the last poll question that we have and would be interested in what feature of the website that we describe do you find the most useful either the developer guides presented the last those still upcoming not yet present online the cross library mapping showing what errors are what in a different library the new rewritten documentation prolonged that are just on the website for the moment or the example certificate that is testing and can make deterministic well testing and QA on all of the errors that are possible there again a couple of seconds for voting in the polls panel on the right I see that so far the new extended documentation is leading with six votes with the others having a single person voting for them and with this question and basically this contributions of ours we'd like to end our talk thank you for your attention and welcome any feedback or Q&A if there is any will be available or at least I'll be available in the discord after the session I'm not sure about Pavel we are able to take Q&A now if you have any I have a question so you said there were 78 errors in OpenSSL right? have you taken a look at the new OpenSSL I think it's OpenSSL 3.0 that's coming out pretty much now I think we haven't checked the 3.0 there were changes from definitely pre 100 to 100 and some compatibility breaking changes very nice ones from 110 to 111 I think we've been in touch with the developers regarding the previous experiment as well and we've been happy to see some of the documentation and manual pages think upgrading nicely and actually improving the stuff that we've shown the developers have complained about but we haven't focused on the 3.0 yet I've been looking into it recently but it's not yet at the website