 One, welcome everyone to the Crypto and Privacy Village. This is our seventh year and DEF CON 28 here in Safe Mode. I am Chaim Cohen, I'm one of the hosts here. Let's just get some stuff out of the way. You can join us at Discord, we are, you can find us there, we have a couple of channels at the gold bug just opened up. So if you're interested in the gold bug puzzle, we did that very well. And let's just introduce our first speaker, Hano Burke, and he's gonna go. Yeah, hi. Well, let me switch these slides. Okay, you can go full screen and you can go. Okay, do you see the slides? I see them. Yeah, okay, great. Yeah. Hello. So I'm Hano, and I'm going to talk about start TLS today. This is some research that I've been doing together with Damian Podibniak, Fabian Ezing and Sebastian Schinzel. And we are soon going to publish a paper on this. Yeah, we kind of figured out that start TLS seems to not be very well researched and it has some issues that are not very well known and we looked a bit deeper into this. So from the general idea I guess most people are roughly aware what start TLS is, but if we talk about TLS connections, there's two ways to do a TLS connection. There's the one way is that we have an extra pod for the encrypted version of protocol where we directly start a TLS connection on that port. And this is called implicit TLS because we kind of implicitly are using TLS based on the port number we already know that this is an encrypted connection. And then there's this other way which is start TLS where we start with some protocol that is not encrypted. And then we have some special command which is usually start TLS command that upgrades this connection to be encrypted. And in the web we are only using implicit TLS basically. So what we usually do is that we have HTTPS which is kind of its own protocol on its own port which is a HTTP protocol with a TLS wrapped around it. But start TLS the most common usage is in the context of emails. And in this talk I am mostly going to talk about the connection between an email client and the server. I will talk a bit about server to server connections later but also we haven't focused so much on this because this is kind of a different issue because the considerations for server to server are quite different than from client to server. So all email protocols we use, I guess you are aware of that, we have SMTP, POP3 and IMAP, we use SMTP to send emails, POP3 to fetch emails or IMAP also kind of to fetch emails but also to manage a mailbox on the server. All of these protocols support start TLS. And for the client to server communication these protocols also all support implicit TLS. So we kind of as a user we can choose which one we use. So if we can choose between the two then the obvious question is which one is better and which one is more secure. So how does this start TLS work? So these email protocols they are all kind of relatively simple text protocols so it's a line-based protocol where a client sends a command to a server and gets some kind of answer. So this is an SMTP session here, the S and C at the beginning that is a server and client. So if you connect to an SMTP server the first thing is the server sends this line which is basically a hello message which contains its host name and a special code number. And then you send this hello command that is basically from the client always the first command which is kind of introducing the client and then the server usually gets back with a list of features that the server supports. And here you can see in this list that it ends with this start 250 start TLS which means okay the server just told us that it supports start TLS. And then the client can use this start TLS this is simply a text command it gives in this command start TLS then the server answers with a 220 code which means everything's all right and the okay behind that's basically meaningless that's more or less like a comment. And then we do a handshake and for the colors here the gray thing is the plain text part of the protocol and then we have a TLS handshake and the green thing here that is then encrypted and authenticated with all the properties that we usually expect from a TLS connection so we expect that there's some sense of security here while the thing at the beginning it's plain text so an attacker can read it an attacker can manipulate it. And then basically we start over because kind of all the information we got before the TLS handshake we don't really trust it so we send this elo command again and then maybe we can do a login and send it to me and you don't need to understand the details of how the protocol works but the basic idea just that should stick here is we have a plain text protocol we have a command that starts the TLS connection then we do a handshake and then we kind of move this protocol into TLS. Now first something which is obvious that if you use a start TLS in an opportunistic way this is insecure so there's some email clients implement start TLS in a way where they will say okay if the server offers start TLS then I will use it and if the server does not offer it then I will just stay in plain text mode. So we will use start TLS if we can but if not then we will just do plain text and a network attacker can attack this we figured out that this is actually not that common anymore this used to be the case maybe 10 years ago but today most clients don't really do this anymore and it's obvious that an attacker can in such a scenario an attacker can just they can pretend to be the server and then they can say I don't support start TLS and then the client logs in and then the attacker knows the username and password so that is obviously insecure but that's not really what our research was about because like that's yeah but this turns out to be not the only security problem with start TLS so you could think okay this is a problem but if we implement a client in a way that it always forces this start TLS connection and will just refuse to connect if it cannot do that then this is secure but it turns out that this is not the case so if you look at what start TLS is then it's kind of a form of state transition where you have the protocol which can be in two states it can be in a plain text state and in an encrypted state after the TLS handshake and this raises quite a few questions so at first you can ask okay we have the state transition but which protocol features can we use before we turn the encryption on what can happen in the plain text part and also how should we treat the data that we got before the handshake for example server features or if there's any login because we cannot really trust that data as it's as it's obviously not secured and also you could ask like do the implementations do they have a clear understanding what is encrypted and not encrypted and do they clearly separate that and so just one example which is a relatively obvious problem the IMA protocol has a feature which is called alerts which is basically the server sends a message to the client and the client is supposed to show this to the user and these alerts it turns out they can be sent at any time so they can be sent in the plain text part which is interesting because like this is a screenshot from Outlook where the IMA server sends a message to the user and this message is not secure it is sent in plain text even though the mail client is configured to use a secure connection and here you can even see like you can even include a link which will be highlighted so we have here your IMA server wants to alert you to the following please download micro something something I can put in a link where he provides a malicious file so we have an alert send in plain text but for the user it looks like this is a legitimate message from his mail server so he may want to trust it so yeah this is just a design problem of the protocol how it is specified but then we came upon the issue of buffering bugs and this is something that has been known for a while and the question here is really is something part of the TLS session or not and two clients in servers have a good idea about this so there was a vulnerability in 2011 already so this is nine years ago which was named plain text command ejection in multiple implementations of start TLS this was discovered by the author of postfix and what happened here is like if we look again at this start TLS connection and this is like only the I've shortened this a bit I've cut away the stuff that currently doesn't matter like we have this client sends the start TLS command gets an answer from the server and then there's a handshake happening now what can happen if we do something like this where a client sends the start TLS command and then sends something else in plain text and then gets the answer from the server that he can now start the TLS handshake and then they do the TLS handshake so it turns out that there are quite a few implementations of servers that will then process both of these commands the start TLS and whatever other command you send together with the start TLS in a single TCP that it will put this in some kind of buffer and process every command after another and then it will answer to this command that was appended to the start TLS in the TLS session so we kind of have something here that traverses from the plain text part of the protocol to the TLS part of the protocol like we send something in plain text but the server kind of thinks that this is part of the TLS connection so that's basically what I just said so this bug was originally found in postfix but it turned out it affected multiple mail servers including servers Korea, Qmail and various proprietary mail stacks so there's an advisory by USSERC from that time which contains a long list of affected servers so this turned out to be a very widespread problem now you may wonder how you can attack this and this is kind of I think this is maybe part of the reason why this bug is not very well known because the postfix authors they have described very well what the problem is but they haven't really published any exploit or explained how you could actually attack this so it was kind of seen by some email discussions from mail server developers where people were like yeah this is kind of a weird behavior but it's not really a security problem but you can actually attack this pretty easily and this takes a bit of thinking what's going on here but so this is the first part of the attack which is we start a connection that is in plain text so we have again this the start messages that's not really the interesting part the interesting part begins where we have the start TLS and then this red stuff this is injected by an attacker so the attacker is injecting here another ELO command which is that is just always the first command that comes after start TLS that's why we have to send that and then the attacker locks in with its own credentials so our assumption here is the attacker also has an account on this mail server but it could be something like public email provider so that is a plausible assumption and then the attacker is starting to send an email like he is setting a from address and a to address and the to address particular is his own address and then it sends this data command which is basically now the email content starts and then we get an answer from the server this is now the answer to the TLS command so now we are doing a TLS handshake the client and the server but we have these injected commands and then the server will answer to these injected commands so it will send this normal greeting that always comes after the ELO command and then it will authenticate the user and then it will say okay now the email starts and now what happens is that the client doesn't know about any of this it doesn't read from the server because it doesn't expect it to send anything it sends his own ELO command it authenticates and then it starts sending a mail now what happens is that all this stuff all this blue stuff which is client sense this ends up being part of this email that the that we started sending to ourselves as an attacker so the attacker now gets an email and that contains the authentication data from the victim from the client so this is a pretty severe attack like if you're a network attacker and we have a server vulnerable to this we can force the client to send his credentials to us there's a similar attack with IMAP that is a bit more tricky because like you can in IMAP you can kind of store a mail into a mailbox and then it's the same idea one problem with that was that in order to do that you need to know the exact size of the message that's going to come that means you kind of need to guess how long the password of the user is but you may be able to see that with a side channel attack and also it turns out that many mail servers on failed login attempts they will just try to reconnect so you have kind of as an attacker you have multiple tries to do this so it's still a feasible attack but it is a bit more difficult than the SMTP attack okay so this bug was discovered in 2011 so you would hope that people have fixed this but as probably no one is surprised it's actually not fixed everywhere so we did some scans and we found out that in SMTP servers 1.5% are vulnerable in IMAP and POP3 it's more it's 2.5% I guess the reason why this is different is because this was originally described in an SMTP server so that's probably where people had it on the radar but the very same class of bug can happen in basically every protocol that supports start TLS although I have to say for POP3 we we didn't really find an exploit to attack this but it's still kind of something that should be fixed now there's another question you can ask and that is can we have a same kind of bug but the other way around that is on the client side so can we confuse the client to get data that was sent in plain text and the client thinks it's TLS data and in turns out yes we can do that so the idea is pretty much the same the client sends a start TLS command and the server sends this 220 code which is just approving that yeah I got this start TLS now we can do TLS and then it sends then we can append additional data that's coming from the server and then the client will interpret that as part of the TLS session so yes this attack also this vulnerability also exists on the client side so yeah that can be vulnerable to the same class of bug and we have called this a response injection because like the other thing was called a command injection and here we call it a response injection what this is not that severe but for example it can be used to buy an attacker to spoof mailbox content you kind of have to guess what commands the client will send but if you know which may client it is you can usually know that so for the user an IMAP inbox that is not his own and that contains other data so it's still an issue and the really surprising thing was that more than half of the mail clients that we tested for vulnerable to the spark so this is a very very widespread issue so to sum that up a bit this start TLS command injection it was discovered in 2011 and it affected multiple implementations and even though many implementations were found back then they're still a widely used more vulnerable implementations also the same class exists on the client and the majority of implementations are vulnerable or were until we have reported it to them and just very generally if you're doing an implementation of start TLS and you're not aware of this bug and you're doing a naive implementation then you would have some kind of command buffer where you read the commands line by line and then you will usually have this bug too so it's kind of one of these things where you have to actively be aware of this bug in order to prevent it and this in my opinion the key of sign that this is not something where we should say this is a flaw in the mail client but this is a systemic problem in the standard like if you have a situation where the same bug appears again and again and again and in multiple different implementations then you should wonder if this is kind of if this is something where we can just if we can do better on the level of the standard and not say like it's the flaw of the implementations yeah there's more issues with start TLS one the IMAP protocol the IMAP protocol has a feature which is called pre-auth the idea here is that an IMAP server can when you connect to it send a line which basically tells the client that you're already logged in you don't have to authenticate so the idea here is for example if you have a controlled network where you do authentication based on IP addresses maybe then you can say okay if the connection comes from this IP address then we just assume that this is already authenticated and the user is already logged in so now start TLS the standard clearly says that it must have an authenticated state so if we take these two things together like we have a feature start TLS that you must not use in an authenticated state but the server answers with this pre-auth line which says you are already authenticated now what should a client do like we have a client that connects to a server it gets this response it's configured to use start TLS then we kind of have a contradiction like this is kind of a logical inconsistency in the standard it's not really the standard doesn't really say anything about this this is just where the standard has an internal conflict and after all the only reasonable way for a client to react to this is to just terminate the connection because it cannot do start TLS in this situation it also so there's no way to secure this connection and if it's connected configured to have a secure connection it should just not accept this now not surprisingly some clients will allow pre-auth connection and then just don't do start TLS because they can't and this is also a security flaw you also obviously can do mailbox contents like you're a network attacker person in the middle and then client connects you send this pre-auth response and then you're kind of the upstream server for that client so you cannot steer credentials because the client is already authenticated but you can forge the content of the inbox and also you can sometimes steal mail for example if you're using IMAP usually a sent mail is stored in a folder so that you can steal and some clients also when they connect to an IMAP server they start synchronizing folders so there you may be able to steal mailbox content then there's an IMAP feature which is called mailbox referrals and the idea here is that the server can tell a client that it should use another server so it's something like a forward feature where a server can say like I'm forwarding you to this other server if you want to access this mailbox and now the interesting thing is if you combine these two things you have this pre-auth which means you can prevent the client from starting a TLS connection and then you can do this referral which refers the client to another server and this can be our attacker server so where we prevent the TLS connection from happening as an attacker then we forward the user to server and we control that server so this can have TLS and the client logs in and then we have the client's credentials so this is also a pretty severe attack we have to say here that this mailbox referrals feature and there's a similar feature called login referrals these are not widely supported so we only found one mail client where we could perform this attack in a mail client called Alpine so most mail clients they just don't support these features so they are not affected by this but still it shows that there are features in IMAP where it's not really thought through what that means in combination with start TLS so yeah to summarize that so start TLS has a systemic problem with buffering bugs they can appear both on the server and the client's features in IMAP that are problematic or insecure if you combine them with start TLS and also start TLS does not really provide any security advantage over a normal implicit TLS connection so our conclusion is you should not use start TLS for mail connections for email client connections like this so if you are using email which I guess most of you do and if you are using a mail client then you should change its configuration to use the implicit TLS parts and not to use start TLS and then you will not be affected by most of these attacks some can still happen but some it's enough when the server is supporting start TLS but most of these attacks will be prevented if you run a mail server you should definitely make sure that you don't support implicit TLS I mean there's an RFC that says that you should use the implicit TLS ports but there are for example the Microsoft SMTP servers they don't support implicit TLS for SMTP and if you can then disable start TLS now we are aware that this is not an easy thing particularly if you have existing customers like if you have a user base then telling them yeah sorry half of your mail configurations will break next week that's a tough thing to do so I'll say it would be ideal to just disable it on the server but this is probably not feasible for people who have existing user bases but if you if you start like a new mail server setting then you can start right away without start TLS so then I want to talk a bit about this special case which is the server to server or MTA to MTA connection in case you don't know MTA it stands for mail transfer agent which is another work for mail server so this situation there is a bit different for one there's currently no way to use implicit TLS on MTA to MTA connections that's just not part of the standards there's no RFC how to do this and it would not be easy to implement that and also traditionally the server to server connections are purely opportunistic so usually mail server doesn't validate a certificate because also usually the certificate host is not the host of the email address so even validating the cert doesn't make much sense and you usually don't have an expectation against active attackers now this is changing a bit so there are two standards that one is based on DNS second called Dain and the other is called MTA STS which is kind of a trust on trust use kind of mechanism that are trying to secure these services or connections it would be a different talk to talk about the advantages and disadvantages of those but my personal opinion is that MTA STS is the more promising mechanism but that's a controversial discussion let's just put it like this but so it would be challenging to introduce an implicit TLS mode into MTA to MTA connections like right now we're just using star TLS even if we use these advanced mechanisms we're still using star TLS and the problem here is like that we have a lot of compatibility issues here like in email it's kind of it's a very old protocol and things are very established and like probably a lot of mail servers have a firewall that they don't allow other ports than port 25 so that it would be very challenging to change the protocol to support an implicit TLS mode so I don't see that happening soon so if we want to secure these MTA to MTA connections like if we talk about something like MTA STS then we should really need to make sure that we fix these buffering bugs like these command injection and response injection that I have talked about earlier if you rely on if you think that you want to secure these server to server connections then you also need to think about this and you need to basically audit the implementation so that they don't have these bugs yeah then just quickly also start here as is used in many other protocols like XMPP FTP then manage CIF which is a mail filtering thing then held up MySQL in NNTP so and I guess this is obvious but the kind of bugs that we found they can be present in different applications of all protocols that support start TLS or something like start TLS so there's lots of potential for future research here like this is something that should be tested in more settings so yeah and also our general recommendation that if you can if you have a choice between start TLS and implicit TLS then you should prefer implicit TLS for future protocols maybe you should just only support implicit TLS yeah and then something that is kind of not on the security side but we can also ask what about performance what about speed like because like I come from Germany and Germany is known for having pretty bad mobile internet so I'm not aware of that but if you ever go to Germany that's what you will experience so if you try to send an email with a bad internet connection it's pretty annoying like you have these things that you may get a connection error or the email may be sent but it's not stored in your send folder because these are two different connections so having a better performance here is good like and also more reliability if you need fewer connection round trips then it gets more reliable so and this I also find interesting like there's been a lot of effort to reduce round trips in TLS like in TLS 1.3 by design has if you make a new connection it has one round trip less than older TLS versions and there's also a zero RTT mode which is kind of there's some security concerns about this but the TLS communities decided that they still want to do this despite these security concerns and I find this interesting that so much effort has been put into reducing TLS round trips but it seems nobody has tried to optimize email protocols the same way and if we look at start TLS then all the information that is exchanged before we do start TLS is kind of meaningless because we cannot trust it anyway so we need to retransmit everything after the TLS so for example the server capabilities you're always getting them again after the handshake so you can think about this that everything that is happening before we do this handshake it is just wasted it doesn't have any meaning so this kind of directly leads to that if you start TLS you need more round trips it depends on the protocol and also it depends on the implementation but it's usually two or three round trips more than if you directly use implicit TLS so avoiding start TLS is not just more secure it's also just faster which is I guess good and there are some other ways this is kind of now going a bit into other topics but there are other easy ways to improve email protocol round trips so email authentication and it uses something called this is kind of an authentication framework that is used in all three email protocols and it's kind of an abstraction of different authentication methods although most of the time everyone uses user name and password but they're different authentication methods and if you look at this how this looks like in iMac we can have something like the above where we have an authenticate plane and then we have some base64 and this base64 it is basically it's a zero byte then a user name then another zero byte in the password and that is our login so this is one round trip to login and we can have this thing below where we have authenticate login then we have a new line so this is kind of optional we can in this authenticate line we can already send data or we can just not know this and then the server asks for a user name this is also base64 encoded and ask for a password so this is three round trips so there's quite some difference here and it's quite varying what will happen depending on what client and server configuration you use so the first thing is that iMac has a feature called Sassel IR which basically means you can initiate an authentication right away with data so this is the thing that you have this authenticate plane and send something in the same line where or you can also not do that and of course this is faster this saves you a round trip if you can send data right away with the authentication command and also Sassel has two different modes for username password authentication they are called plane and login so to make matters even more complicated both pop3 and iMac they have built an authentication that is not part of Sassel which is kind of part of their protocol itself in iMac this has one round trip in pop3 it has two round trips so this login method this is not standardized and it is also declared obsolete by the IANA so the IANA has a registry of Sassel authentication methods and there it's declared as obsolete and what it basically does is it sends a username and the password by 64 encoded in two round trips or just two lines in other words and then there's this plane authentication method which is a standard and it sends username and password encoded in one round trip so you save another round trip here so the plane method is standardized and it's faster than login so it's clearly the better method and from the security it doesn't really matter it's a normal username password authentication there's nothing fancy about it so almost everyone supports both servers and clients but there's not really a reason to support login so if you want to have faster email then you should use implicit TLS and non-start TLS you should support the Sassel IR which is specifically in IMAP and you should disable the Sassel login method like for example I figured out my own mail server it supported both and my client if it supported both it used login because that was the first in the client it was it's in cross mail it's the first method that it supports so it's lower for no gain and I disabled it on the server and now I'm saving a round trip here so yeah yeah that was it yeah don't use start TLS thanks for listening and I guess we had some time for Q&A hold on welcome back everybody can you Hanoh can you unshare your screen maybe can I just see you I just want to put my background on okay so we have I think we have three questions for you can you hear me okay so is the implication that the clients in the SMTP case for ease just ignore responses from the server I expect them to be a state machine that says wait what I haven't started sending them yet no that's not the implication so the client is not ignoring these answers but it is like it is processing these line by line so it is sending a command to the server and then it's reading a line from the server and processing that and that is why if you do this attack you kind of have to inject all the commands that the client will send so it will get the answers that it expects is this understandable we are monitoring the discord so if they reply back I can do it so actually give me one second I think we are going to try and share the slides with you so everyone can see it give me one second share this sorry everybody let's see so they want me to I know we have dead air let's see if I can share the screen share also in the discord so that's okay I will be able to see questions later if they are I see your screen now okay I do this one the second question I have in the context of SMTP do you see MTA STS having positive negative no change to the start TLS risks it doesn't I think the question is framing it wrong the thing is MTA STS is introducing an expectation that we didn't have previously that is that these connections are secure against active attackers and these attacks can compromise this expectation so before we could say these attacks don't matter for MTA to MTA because we don't expect it to be secure anyway and now we want to introduce connection security for that so now we have to care about these attacks but it doesn't really change the risk it's just that what we try to achieve with MTA STS that is compromised if we have these vulnerabilities does that make sense I mean I mean yes I understand it let's I mean again we can be in the slack discord and maybe later if people have more questions you can join in there last one that I have in what kind of issues does the speaker see with implicit TLS so I think if you compare start TLS and implicit TLS there are no added issues by implicit TLS because it's just the simpler thing like with start TLS we start in plain text and then we do the state transition with the potential for errors and with implicit TLS we just don't have that so I don't see any risk in implicit TLS that you wouldn't have with start TLS of course it has all the risks that you can have otherwise with TLS like all the if there's a bad implementation if there's whatever bad randomness or vulnerable algorithms then you have all the same risks but you would have the same risks as well okay we're getting another question any thoughts on telnet start TLS I have no idea sorry I have seen that it exists but I'm not sure is anyone using that I would be curious if anyone's using that I feel people use SSH there usually and then our final question is just thanking you for the talk is anyone working on email pop 3 replacements with something more secure I'm not sure there's not really a problem with pop 3 I mean this is kind of not the issue of the talk but I feel pop 3 is a really simple protocol and I kind of like that I'm also like one of these people that I'm still using pop 3 like I like to have the mails on my pop 3 so I think there's yeah if you use it with TLS then it's a fine protocol well again we would like to thank you for your talk if anybody has any more questions you can hang out in the discord we'll try and relate them to you anything like that I want to thank everyone who's watching the live stream for sitting with us through the first ever DEF CON safe mode and the glitch crypto and privacy village I want to remind everyone again that we have the gold bug challenge that's still there and I would like to again thank you for your talk and I hope you enjoyed yourself I guess we wait okay I can if you're not