 Hi everyone, my name is Martin, I'm doing a PhD here at Masa University, but not, which is not this university, but the other one. And I'm going to present to you a research that we did on my PhD on last year. Actually there was a booth, maybe you've attended the booth yourself, maybe not, I'll ask later on. It's a PhD cooperation with Red Hat Tech, and it will involve most clear TLS certificate, certificate elimination, error messages and stuff like that. But before we dive into TLS and into the hardcore developer stuff like that, let's step one or two steps sooner to see why we are going there and how we came to get that thing. And for that we'll go to Hawaii first. So imagine you are in Hawaii, it's the 13th of January 2018, you may know today. You're in Hawaii on your day at the beach, sun is nice and the sea is warm, and suddenly 8.03 in the morning a text comes and the text says ballistic missile threat inbound to Hawaii, seek immediate shelter. This is not the drill. And then things start to happen. For 38 minutes something happened. I don't want to say I don't really know what happened, whether it was many havoc or just peaceful contemplation on the Hawaiian beach. But then after the 38 minutes by some means you probably learned about that missile alert is in error and there is no threat. And basically what happened, the guy that was in charge of doing the test or the drill alert for the whole Hawaiian archival, then they just, well, misclicked and issued an alert for the whole island. You may blame the guy for being careless, but after a pressure from the public it was suddenly revealed what the user interface of the software was like. And in something like this, the guy selected the PECOM state only, which is the live red alert, while he was supposed to go for a drill PECOM demo state only in a combo box somewhere in the system initiating all sorts of alerts and errors. And this symptom is because everyone at the island to get the text, all the traffic signs, everything, just a shift to get the alert. And now you may say, yes, this is a problem of a bad user interface, this is nothing to do with security. I claim that this talk is going to be about security. So let's move one step closer to security. Let's look at an encrypted email. There's a very known beeper, at least in academia, that's quite old from 1999. And it's called Why JolliCon Begrip. A bunch of academia guys decided to produce a study on the PGP at the point, PGP 5, looked something like this. They don't really remember or maybe they'll use it. The study basically shows that out of the 20 people, or recruiters from the university, but not IT related, about three of them were able to send a PGP signed email, one given an hour of time. And out of the rest, about half managed to send their private key by the email that they send. About a third of them managed to upload everything online, both private and private parts. And most of them had trouble understanding that they need to keep the key on the recipient, not the key on their own. Basically a usability failure from PGP 5.9, sorry 5.0. Then, hopefully, the world will move down. So in 2006, someone did another study. Now starting the tradition in academia of the JolliCon articles. There are articles about JolliCon doing all sorts of things. And PGP 9 combined with Outlook Express basically did not get any better results than previous ones. Still a little later on, the world still hasn't moved on. Because in 2015, MailVelo, which is a plugin for browsers, doing PGP basically still has the same usability flaws and people are still kind of unable to send signed and or encrypted email. It's not just the academia, kind of everyone knows that on the internet. And we're not even pretending to PGP or encrypted email as a thing, as it's implemented now with particular users. But still, this is only the end users that are not necessarily IT educated and are not working in IT. So it's kind of expectable that they may not have to screw up the security. But I daresay this is not the only area of people having such problems. So let's finally move to the GLS and to the certificates now. So imagine you need to validate the certificates. You called for open SSL, you found the appropriate sub-command and the command line argument and you get an error message that looks something like this. Well, you see the details of the certificate and then you mistake that error from 7 happened and the zero depth of the lookup and that permitted subtree is violated. Whatever that means. I mean, I know some of you in the room, so I know that some of you in the room are perfectly aware of what this means and what are all the intricate details. But I daresay there are people amongst you who have no idea what kind of a subtree is there and why it's permitted, but it's been violated, but it's permitted. So it's okay, isn't it? And now I'm kind of starting to go into quotes that people reacted when we showed them this error message. The first option that you can have when you run into an error message like this, well, do a pop up for the end user because they are trying to connect somewhere so they probably know what they are trying to do. It's not that much of your problem. You delegate the security decision to the end user. This, well, works from your point of view because you get the problem dealt with. But I would argue that you should go for the second solution, which is investigating yourself, trying to understand what's the problem and trying to either do the security decision yourself because you are much more capable and much more knowledgeable in the area than the potential end user or at least recommend an action for the end user that they should take and not just ask them whatever you think about the particular error. The problem with solution two is that there are many possible errors that can happen when we are only validating certificates, not speaking about other security stuff that you would be able to do. This is a list from OpenSSL. It's not full, it continues below. Down there, there's roughly about 80 errors only related to validating certificates in OpenSSL, which brings us to actually stating the problem. So we were interested how do people in IT, mostly employed in IT, educated in IT, perceive certificate flaws? Do they understand what caused these flaws or what is the actual point of the error? Do they see the security consequences that the particular error might have? Do they know whether the error is serious or less serious? What further complicates the matter is that sometimes invalid certificates, for example in TLS, are deployed deliberately and even contact the admins of the servers. They say, well, it was too difficult to obtain a real certificate, so I just did so signed. Or it's just for testing purpose and nobody should see the server, but you know, it hasn't public IP, so people kind of start and learn accessing it and using it for if it's some sort of a service. Or mostly in the past, they were just unwilling to pay a certificate authority for a certificate because let's say it wasn't that common or it didn't exist at all. A second part of the problem, or what we are interested in, is how do error messages themselves influence, slash help, slash worsen the comprehension of the problem? Do they actually measure if we change the single line errors that happened when you validate the certificate? Does that change anything? Or is it really out of the scope and it just says it's an error and you'll Google the error anyway? Well, to investigate these things, we did, as I mentioned at the beginning, an experiment here at DEF CONF last year. We did a booth back there, you might remember. We had stands of IT jokes trying to lure people in to stop by, get interested so that we can convince them to participate in the experiment. Now a brief question. Any of you in the room participated in the experiment last year, hands up please, so that I have a rough estimate. So four, which means mostly not. I guess the four then came to see the results or to see what the other people thought about the same problems. So when people stopped by and went for the experiment, the task was as follows. You will hear at DEF CONF and imagine that you'd love to register for DEF CONF, signing it with Google or Facebook or your Microsoft account that surely everybody has here. And since it's an open source conference, you decided to help the conference yourselves to fulfill the dream of yours. And you started to write a patch. You wrote a sort of small testing program for the beginning to try connection to the different servers of the identity providers to find out whether the connection works. And you find out that it doesn't because some of the providers' servers have certificates that don't validate at some point. So you get an error, for example, the already mentioned permitted subtree violation error. So this was the setup. And what we did was we asked the people to try to understand what the problem is and what the associated risks are, doing whatever they were to do if this was a real case at their work. So googling, reading tutorials, reading manual pages, using local tools. The only prohibited thing was talking to other people. Then asking how much would you trust a server with such a certificate, or with a certificate with such a problem. And then later on, at the end of the tasks, there was an interview report when we let them describe in their own words what was the problem with the certificate and what were the associated risks as they understood them or as they perceived them. In this way, we talked to 75 developers here at DefConf, which took quite a bit of time, but was definitely worth it. Of the 75 participants, sometimes on the slides, I'll use the small pawn symbols not to write participants everywhere because it's sometimes rather long. 67 of which consented with having the interview at the end recorded. So some of the things that will be presented only comes from the 67 participant sample, not the full sample. The vast majority were employed in IT, usually more than just a couple of years, the median was eight. Only two-thirds of those, well, of all of them, had formal education in IT, which meant university degree, bachelor's, master's or post-graduate. Actually, there were about two post-graduates in IT taking part in the experiment. And most of them used OpenSSL as a tool before. I specifically mentioned OpenSSL because the error messages that we used were copy-pasted from the current Fedora OpenSSL version, which was, I think, 110 pips validated. We also used for some other tools just to be sure how much people handle certificates in other areas. And the number wasn't as great as OpenSSL, but they used the network security services, Java tools, GLUTLS, and with the lower percentage also some other tools. Now, the first part of the results. What perceptions do people in IT have with respect to certificate laws? I'll go one by one with the five cases or the five certificate errors that we had, and for each of them, I'll discuss what were the main topics or the themes that repeat it in between the 75 participants. So the first case, kind of a baseline case, was a GitHub or pretending to be a GitHub certificate. That was okay. Well, this is the literal OpenSSL error or a message, actually, that you get when it validates correctly when you do the OpenSSL, verify the standard one without any extra checks. By far the most, this is from the recorded interviews, so 67 is all of them. Almost everyone mentioned that there isn't a problem that the certificate is good or it's okay. It's not 100% because the interviews were unprompted. We didn't ask precise questions like, do you think there's no problem with the certificate? Yes or no? Because that would be rather leading the people to tell us something that we wanted to hear. So we prodded them until they were satisfied with everything that they said, and then we made thematic analysis out of that. I mean, this is not surprising because there was literally nothing wrong with the certificate. 13 people mentioned that they decided to do some kind of an extra check themselves, like saying, oh, I looked into the search and couldn't find anything wrong, so I could probably trust it. Sometimes they examined the validity, sometimes the issuers, depends. Then last notable group is people mentioning that they trust the certificate, but just on the assumption that the program that they are using to do the connection with the server is bug-free. Since we provided them the program to do the connection establishment because it would be too long to ask them to code that, we also asked them in the task to presume that the code itself is bug-free because otherwise people tend to open the code and look for bugs first and only then inspect the certificate and get much more interested in the certificate. So this is a rough idea what it will look like when talking about the cases. The second case is expired. It was a Microsoft certificate. It has been expired for seven days by the time. This is an error message that OpenSSL gives you. Now I'd like to give you a couple of seconds to think of what are the thoughts and associations of yours to an expired certificate. If you were to describe to a friend of yours who has never seen an expired certificate before what are the things that you would tell them and now basically shout them out. I'll try to collect them. What is important about expired certificates? For how long they have been expired? Yeah, it might mature or it might not. Anything else comes to mind? If the local time settings are okay, that's a nice bit. Do you find it common? Does it happen on the internet or is it a rather rare case? I see rather different opinions here. Someone's shake their head that it's not common. Someone said that it's common. Checking who's the issuer might be the case because it happens all the time for less entry. One thing? Yeah, sure. So it depends. It kind of is common, but it is usually not common that the server is still operating and this is the certificate that you were served when you contacted the server. So as expected, the most common answer was basically saying that it's out of date because it's literally what the problem is and the problem is not that difficult to comprehend here. Then 27 people, nearly half, mentioned that this is probably just some kind of a mistake or negligence or operator error or just somebody just forgotten and they are about to do it. They are about to renew it or something like that. Which basically hints that they didn't consider it that much of a problem and they didn't think it's probably a tag, though. I admit that this is interpretation. The third one, the third most common team was that it's common. From what they can see on the internet or giving examples that it happened here and there at that time or this time. The other team was that it was okay before and now it's not okay. So they expressed that it had the certification guarantees that the certificate should give you, but not anymore. And then they mentioned the thing regarding the issue or the subject of the certificate because if it was a small business from my local neighborhood, I would probably trust it anyway. But since it's Microsoft, because the task well said it's a Microsoft identity provider server, they don't trust it much. So that the subject mattered regarding what subject mattered in what trust they gave an expired certificate. The last thing mentioned is that eight people mentioned that it actually might be an attack because we can't guarantee that this is not a stolen previously very certificate that has been revoked or something like that. I just want to say one more thing. Sometimes I mark the codes with an asterisk. Those are the codes that I kind of see as hinting at understanding of the actual technical problem with what there is, not the assumptions or the external experience or stuff like that. It's not that interesting here. It will be interesting in a different case. The third case was a self-signed certificate. Again, we see the standard open SSL message. So what comes to mind when you get a self-signed certificate? What would you tell to other people? Not trust it. Anything else? Just to test purposes. Didn't hear the beginning? Okay. There's some problem with the authentication kind of coincided with the... Well, do not trust that. Anything else? Possibly a man in the middle attack. We're kind of browsing around what's actually the problem because people mentioned that it's signed by itself and there's no certificate authority and stuff like that we'll see in a moment. Basically saying that it's signed by itself or some variation of this was the most common theme. Then saying that there's no certificate authority or an official authority that would have signed this certificate, which are both actually saying that they kind of understood what kind of a certificate that is. The other thing that they mentioned is that well, anyone can create a self-signed certificate. They often explicitly said including themselves and that it's a certificate they could have created themselves. So basically no trust could be given in those. Then people mentioned that if this is expected then they would consider it trustworthy because there are pages that have self-signed certificates that they trust but it's usually their own local server or a local server of their friends or their library or whoever else where they know that there's such a certificate or someone told them or they are kind of expecting a certificate of this sort to be there. The same amount of people mentioned that it's usually used either for internal purposes only so such a certificate should not be got when you're connecting to a public server or for testing purposes which I think no. I don't have the testing as a separate point. The last but not least is saying that it might be an attack. Again, it's not that prominent but eight people which is roughly 10% mentioned that. Now I want to highlight here that in the most common teams there's not the common tag that would say that people think that self-signed certificates are common which are they say if we asked five years ago or 10 years ago that might have been the case more but today's the world of less encrypt. This doesn't happen and people don't consider it common anymore which I think is a very nice outcome that less encrypt has came with. There will be still some more cases but without we are halfway through the cases there will be two more so let's dive into them. The fourth was a host name is match. The identity provided was Facebook. You were connecting to OAuth.facebook.com but the certificate was issued for OAuth.facesbook.com so there was an extra S in the name. The error message doesn't give you the details but you can still look into the certificates as many people did. Again, the most mentioned was that the name is bad because the certificate was issued for a different host name which is literally a very nice description of what the problem actually is. Only about half of the people mentioned that they did they mentioned the other name which means that they must have looked into the certificate and actually saw this is the difference there's the extra S and sometimes they said it explicitly. A little bit more than 27 people actually did look into the certificate because we have the technical logs of what they were doing as well I'll mention that probably on the later slide when we get to resources and their behavior and actions. Then 22 people mentioned that it's probably some kind of efficient site or some attack of the source and each people said that it's a different domain but I'd say it's some kind of typo or something like that operator negligence error because there's just a single character redundant in the name. But compared to other cases the attack code word was much more prevalent than the mistake code word which I personally am quite happy with considering what the error is. The last case is the already infamous permitted subtree violation which is basically now describing what the problem is if you have a certificate authority that issues the certificate for a sub-authority but constrains the sub-authority you're only allowed to issue certificates in these sub-domains or basically in this tree or you're restricted to go for a tree that's why it's either permitted subtree or restricted subtree. And then if the sub-authority issues a certificate outside of the constraints that it was given by the head authority the error message is that the sub-trees of domains where the endpoint certificate was allowed was violated so it's basically a name constraint problem but the error message says nowhere that it concerns the name. The most common code was which is but only about half of the people that they understood that there is some chain and a certain point in the chain is restricting a hostname to something so they got the idea that there's some sort of a constraint somewhere but they were often not able to tell who is setting the constraint and who is the constraint being set on. Nineteen people set something that was literally wrong which means so for example when I opened the certificate I found out that one of the authorities was listed as false but the others were fine which handles the basic constraints extending the certificate which is for certificate authorities compulsory fields fields that says whether the certificate belongs to a certificate authority which means true or not which means false and the endpoint certificate for the actual server not for the chain of course said that it's not a certificate of a certificate authority and when they opened the certificate they saw a big line saying certificate authority column false which about four people independently found and attributed the error to the other of the 19 people saying literally wrong things were saying all the key purposes were set wrong because of the key purpose extension and most often blaming other extensions in the certificate that they found and obviously they didn't understand very well because they found an error in those then 14 people literally said that they don't understand the whole thing and they don't know what's the problem 10 people mentioned that it's probably an attack or that it's bad for example saying that they would not let Google that they probably have a rogue admin that was a person actually kind of understanding because he knew that it was even worse than the endpoint certificate being hacked or something like that it was the problem at the sub-authority level because the sub-authority issued something that it was never allowed to do then 10 people again understood that it's some kind of a CA problem saying that the CA was not allowed to sign that but again they often didn't know the details of who was constraining where and what was the constraint on but they understood that it's a problem of the CA somewhere or the sub-CA and 10 people they may be overlapping of course understood that it was the CA setting the constraints for the other CA 10 people said that it's probably a mistake just an innocent misconfiguration of the kind that happens all the time on the internet just like in the previous cases there usually are people that consider even FaceBook for a Facebook server a typo which I myself consider rather serious a thing but I'm not the authority to judge 10 more people said that when they were trying to Google for details on what the permitted sub-tree violation meant it was rather hard for them to find documentation because OpenSSL documentation on the error code permitted sub-tree violation says permitted sub-tree violation then you Google and those that were able to Google something ended up on a personal blog of some Lithuanian guy who's an IT specialist and wrote a blog about what name constraints extension is good for Wrapping up the understanding we moved to the second part which is asking the people whether they trust the servers with the certificates or with the errors that I described the five we gave them a trust scale to choose from it was 0 to 6 they could have chosen 1, 3 and 5 as well but those were on the numbers the four that are listed were also given a description which 6 out of 6 basically means it's totally okay if it was my bank website I would be logging in 4 roughly meaning probably okay logging it to a library or whatever but I wouldn't like banks or something like that 2 out of 6 probably not logging in on such a website 0 out of 6 totally untrustworthy I'm going away from this website whatever is written on it is not relevant because it's probably spoofed, attacked or whatever else now a comprehensive and maybe a little bit intimidating graph what it says is the grey bars are 0 out of 6 and then it goes up with the base blue being 6 out of 6 I'm totally satisfied what we see that people are mostly totally satisfied would be okay case, no surprise there for the hostname mismatch we see that this is the most distrusted case more than half of the people not trusting it at all and considering it not even trustworthy while for expired self-signed and name constraints it's somewhat similar even if you do the average it turns out somewhat similar which I claim well it's not necessarily a problem per se but could have been better because the expired certificate had the full guarantees when it was some time ago when it was still valid but the self-signed certificate basically never did give you any guarantees because as some of the participants said anybody could have created the self-signed certificate so if it was my decision I would have much more liked if the self-signed case was much more darker and similarly for the name constraints because even though it's debatable whether the problem is that bad or not it's the problem on a CA or a sub-CA level so it's not just the problem of the N certificate which the others are so this to me is a rather bad error because there's trouble with more certificate on a higher level or on an administrative level looking into the expired case in more detail we asked people whether it matters to them how long the certificate is expired and we offered them four choices and gave them the scale as well these are now the averages so you see that if it expired yesterday it's almost like I would be logging in but probably not with my bank and as the expired date grows the trust decreases quite a bit but this illustrates that yes, people kind of expect that expired certificates are common or happen sometimes and that when the certificate is expired it's probably not necessarily that bad a case moving to the third part of the results which is do the error messages actually influence something does it matter whether we write something like this or something different the idea how we tested this was that as I mentioned there were people getting the open SSL error messages but the other half of the people were getting different error messages with a different piece of documentation that we wrote and gave them to use the new error messages were like this in the OK case it explicitly said that all performed checks past kind of hinting that there might be checks that were not performed automatically for example check for the revocation status is not done by open SSL verify unless you explicitly asked for expired offers an option that it may not yet be valid self-signed is a little bit explicit that it's self-signed and not a route hostname is match is a bit more verbiting in a way describing that the hostname does not match the certificate subject name more driving you towards the names so when you open the certificate you kind of know at which part to look for and the name constraint drops the permitted subtree and says that it's a name constraints violation problem or that the name violates the constraint set by the CA furthermore about that the wording was new we kept the error codes and we also tried experimentally to add a link oh see details over at this website because we are well interested ourselves whether developers would click or would visit the website or would Google or go for official documentation instead because the error word usually was the same the new website with documentation that we created had the five error cases in a rather distinctive yet concise structure with the error code and the full verbiting thing that was displayed in the terminal to match the section they were looking for the explanation saying what happened, what's the problem the security perspective trying to explain what might be the security consequences and some kind of a next step section what to check next, whom to contact what would be the right action to do what changed when we did the new error messages for the OK case people did more extra checks or mentioned that they looked into the certificate to check this and that because both the error message and the documentation kind of hinted that those checks were performed past and they sometimes were googling what checks are actually performed and which are not whether their evocation status is excluded automatically or not for the self-signed more people thought and expressed the idea that it's probably an attack or it might be an attack or it might be wrong as opposed to let's say a mistake for the self-signed other than that for the name constraints there were considerably less codes indicating that they don't understand the case or that they would be stating things that were literally untrue for the two cases that I'm not mentioning no observable significant change when we look into the trust differences the graph being even a bit more intimidating than the previous one for the self-signed the hostname is match and name constraints the trust basically decreased you can see that the name constraints the self-signed and the hostname is match case are darker than in the upper bar which is the most seen in the self-signed column where we got nearly 45% of the people saying that it's not trusted at all and you should not even trust the documentation on the website it's probably an effect both of the error message and the documentation which also tried to stress this but as you saw the documentation was rather brief it was usually a paragraph from the security perspective of two to three lines we didn't want to write the labor because nobody would read them really just briefly we also measured all sorts of other things at the booth name constraints case took longer to process than the other two kind of expected because the extension is not known that much people tend to look at the certificates rather than just saying oh okay so I won't look or name constraints problem okay I don't need to look into the certificates actually they do which is a nice thing almost everybody googles not that much surprise there they often use the text code less often than the formulations with their own words and even less often copy paste in text message so they like text codes because they realize it's a unique code that is easy to search for which is great the link that was offered got clicked rather often nearly three quarters of the people that saw the link clicked it or retyped the website to the browser themselves which gives us as the developers of the tools nice opportunity to point users to a useful place that we may know about or the official documentation it can be even more specific than what we did it could be cx509 errors hashtag name the text code which basically gets you to the sub part of the web page that you want to read right now without googling without anything more keep calm there will be no more results I've already covered quite a bit even though there are more details I could have gone into so recap what we did we did a study with 75 deaf confetti in these last year we gave them four certificate errors half of them open cell error messages half of them redesigned error messages and documentation the cell sign and name constraints may be overly trusted I realize that this is up for discussion and I'm no authority to say which error is more secure than the other though that's my opinion the name constraints extension is not much understood even among like people literally working in IT attending deaf conf the expired trust depends quite a bit on the time that's elapsed from the point when the certificate got expired changing error messages documentation matters and links in the error messages seemed to get clicked and propose a way to guide the user somewhere so what can we do next with this well short term when I finish writing the paper in a matter of weeks I plan to submit a couple of pages to open a cell we've done a couple of small pages last year from the two years ago experiment here at deaf conf it'll be mostly like compatibility preserving stuff like the error messages when I have a formulation that I see that is a bit better can be reformulated maybe the error messages the documentation for permitted subject evaluation can say something more than permitted subject evaluation and I know the guys at open a cell are happy to get requests of the sorts and then well publish and share the results to spark the discussion on how IT people see certificate or perceive certificate laws whether people understand name constraints extension so what would be the consequences if it actually got used much more than it is currently nobody really uses it and maybe a bit controversial discuss whether there should be links in error messages in the more long term and maybe a bit more ambitious way I would like to map error messages for now for certificate validation in different libraries to see if similarly looking errors like expired expired or not yet valid in GNU TLS mean completely the same thing or there are nuanced corner cases that are an error in one and a different error in the other and maybe compare or share the documentation in between the library a more long term and ambitious thing you may know that in late 2017 Mozilla Firefox Mozilla Microsoft Google W3C and Samsung decided to merge their browser documentation at a single place and they created a single shared collaborative place at the Mozilla developer network which is a thing that could be done for the error messages if all the libraries were linking to a single place that described the error messages and what the problem with the certificate is then well it wouldn't be the case that developers of every library would have to formulate all the documentation and the explanations themselves and to test whether the formulations get understood. So this is what I can do next. I would also like to propose what you can do next take your product that you're developing or testing and do read the error messages that it produces to find out whether you understand them first and whether the users of the product or the developers if it's tools for developers understand them ideally go to some users and ask them or show them the error messages and see what they think whether they get what they are supposed to get or even make a study if you feel like that and have time or budget for that. And if you like the ideas presented here well spread the word there will be a paper available soon if you want to have it e-mails to you I'll leave here a paper to sign your e-mail in share any feedback in person to me or by e-mail. Closing user security may still be as I found out when I was formulating stuff for the paper and for the abstract because this is a description of the session and Grammarly complains that user security is an unusual word pair and I would probably go for good security or proper security. So it's still not a thing that people would understand as a call-a-quill term. Which concludes my talk if you're interested in the research there's a website with other research of mine. Thanks again I had to help to cooperating with the PhD of mine and I'm happy to take questions. Yes So have you considered that this is this kind of random messages like it might work on 19 e-books but it definitely never will work even if you improve them put links and whatever it won't work on people who don't care for computers and for piece that was the intention at least at some point in browsers to basically say that ant-terrorism is bad and we want to let you go on the side. So do you think like this makes sense for frequent situation courses there are many scenarios which I don't specialize in security in certain cases something like that but in general how do you think about this So recapping a bit long question drawing attention to that this works for error messages towards developers and IT people but it might not work for end user error messages or literally people using IT only as users and that browsers and it's true that a lot of research has been done in discussing browser error messages and there are TLS connection problems and what I'm basically hinting at which is already going into the answer part is that it has been investigating quite a bit for the browsers not much if at all for the developers and the tools and the task that I had was aimed only at developers because if you are an IT layman so to say so we would never see these error messages hopefully if you did then some one somewhere did a wrong promotion of some sort of a product in the browsers it's true that the research recently has moved towards not letting the user decide that much or only letting him decide when the problem is not that serious Firefox for example in many of the TLS problems just prevents you to access the case and only if the problems are lesser then it offers you a way to override with an exception with about five clicks intentionally making it complicated for you to get through though this is the result of the research because in the past even the Firefox browsers tried to explain what the particular error was and it was just way too complicated for the layman so I do agree that for the IT layman deciding yourself as the developer or possibly a security developer is a better option than letting them decide that's kind of returning to the example that I had at the beginning that making a pop-up that there's a problem with the TLS certificate might not be the right option sure it may infuriate some of the users because they are unable to access their friend's server that has a sales sign certificate but I'm not sure if that's not for the better actually did it cover the question yes okay so the question handled key length in certificates it had multiple parts whether we had something regarding key length in the research of ours if so what was it basically and a separate question if I see any advantages or disadvantages of moving to 4k keys as opposed to 2k keys in this particular research there was no part regarding the keys in the two years ago research both at DEF CON basically the same style as this one there was a part related we had developers do two things validate certificates which was more similar to this one and then actually the first part of the year's experiment was please do issue a sales sign certificate for user ball and this was the high level task and they were given the plain command open SSL to do that no high level tools really and there we were interested also in what the key lengths are because they weren't prescribed and if they asked we would have told them well SUC fit the interesting part was that of those people that needed to google how the command actually looks like the vast majority even though they were guys that basically did the open SSL command from the top of their head everyone set the key length manually so there is a nice default and the default is 2k currently which is not bad but nobody used the default because when they found the tutorial or a manual page there was always an example showing you how to set the key length and they just copy pasted the example and adjusted if they saw fit some of them created shorter keys longer keys but nobody really used the default because all the tutorials showed you how not to use the default so this was kind of the tutorial world and the documentation world to blame I think nobody did shorter keys than 1k there were only a minority of people doing 1k keys the vast majority did 2k keys and there were a couple of people doing 4k keys manually that answers the part of the question regarding our research and key lengths regarding the usage I'm not administering that many servers or I'm not that knowledgeable to be able to say to you that there won't be disadvantages of using 4k keys it's objectively slower and I was told by people I kind of trust that it still is a significant change if we transition to 4k keys completely that things would just slow down and we don't need to do that now so if it's not security critical 2k keys is probably okay but I dare say that there are people in the room more relevant to answer this question than myself so this will have to be sufficient any other question yes it's a long term I have two calls and they think browser if I was 4k because for example we also escape from that so the browsers have unified the need of checks for 4k keys well the browser world is an all sort of other exciting and interesting place elsewhere you that would be nice on the other hand it would be very difficult in such a consortium you can see ca slash browser forum to agree these things so there is a body actually and a place to discuss these things but then if Google decides to push certificate transparency as a security enhancing thing they can do it because their product is important enough and even if some other browsers disagree they can kind of push it themselves and agree on everything so being flexible and advancing faster would be more complicated now I don't claim either of the solutions of going together or going as everyone wishes is strictly better than the other just the world is complicated there was a try at some point to at least standardize the way that warnings and security indicators are presented in browsers the lock icons the green bars and stuff like that and we see standard saying what should the interfacing browsers have regarding this but it's not that much adhered to and I don't think it's really did fulfill the aim so I'm not sure such an agreement in between browsers would work on this field either