 Thank you very much for coming, Upset Co.L is one of my favorite conferences, I'm always waiting for it every year and actually this is my first time to give a talk here so I'm really excited and hope to go away. So this talk is about certain type of nobilities, so this talk is about certain type of nobilities, those what we can call the big type of nobilities where some chance of memory or arbitrary pieces of memory are linked to the potential attacker. So this is a nobilities that can be explored. A little bit about my background, so I started my career path as an artificial intelligence developer. So I started my career path as an artificial intelligence developer in a few startups in Israel and Canada, gaming startups, lots of gaming development. Then when I came back to Israel I joined SAP as a development architect and over there at SAP I changed my path to start and work on security, where I started as a security researcher in Germany, in France and then security architect in Israel. Then I was also in general motors as a security architect, working on connected vehicles and then today I'm in the product security lead at SAP. So I want to take you to England. The year is 2011, it is a new year's eve and we're 2012 is just a few minutes away. Stephen Hansons receives an update code for OpenSSL for Morgan Secondman. Morgan Secondman is a respected academic who's an expert in communication and encryption portables and Stephen Hanson is a co-founder of OpenSSL that was founded in 1998 and is still its lead developer till today. So it's almost midnight, new as eve, everyone is parting outside, Stephen Hanson reviews the code that just got and submits it. This is how it all began. So moving forward in time, about two and a half years later, Nymetha tweets this message. Nymetha is a Google security researcher. OpenSSL 0.3. Our old win is a previous lecture that was just before me mentioned OpenSSL. OpenSSL as most of all know is one of the main of the infrastructure of the Internet. It's very, very important. We cannot have the Internet without OpenSSL today. And this is the heartbeat. This is what we, how heartbeat became to us in 2014. And by many, many expect it is considered today the most, the worst security bug ever. So we're going to talk about a little bit about the heartbeat and to understand what it is, how it happened. So heartbeat is a heartbeat vulnerability. And heartbeat is a feature of the server which we're going to just cover just now to understand what's heartbeat in order to understand what's heartbeat. So heartbeat is basically a way for two servers or client in a server to try and see if the session is still alive. Sometimes it's also being called keep alive or heartbeat. In this case, let's go through the session, the process, and see how client checks together with the server that the session is cleared on. So the client begins with creating a message. This is just, the payload is just a random set of characters. Then the client encrypts it with the session key that it still has from the previous session. Sends it to the server. If the server still has the same session key, it can decrypt. The message takes out the payload and then it creates the response using the same exact payload that it got from the client. Decrypts the payload, creates the response with the same session key. Sends it back to the client. The client manages to decrypt the message and then the client compares the two payloads, the one they sent, the one they got back from the server. This should be the same payload. If you got the payload and they are the same, everything fine, we can continue on. This hardware is configurable, it can run every second, every few minutes, it depends. So that was heartbeat. Let's see what is heartbeat. So you can see here, the client tells, sends this message to the server, look, I'm sending you a payload of the world bird, which has four letters, and the server returns the bird in four letters. And then the client says, look, I'm sending you the payload, hot, with 500 letters, and the server returns hot, with 500 letters. So the other 497 characters that they send back are just from the memory, from the server's and run to the memory. And this is an open server, open server basically accepts connection from lots of servers, lots of clients. What it has in the memory is credentials, tickets, cookies, keys, tokens, all those, what you call secrets. This is what opens the server as in the memory. And this is how it looks like a day after the disclosure, someone checked on Yahoo server, and you can see that you can get up all the secrets from Yahoo. So it's not only Yahoo, it was about 70, sometimes people say 80% of the servers on the internet were exposed to all before two and a half years. Google, Yahoo, Facebook, banks, you name it. So what went wrong? Let's go back again to the heartbeat, the feature, and let's zoom in to where the server prepares the message, the response using the payload that you got from the client. So here we see on the left side the client, the message after it was decrypted from the client, we see that this is very high level, of course this is just cell code. We can see that we have the message payload and the payload is why we need the payload there, because the message has several different parameters or fields, and the payload is only one of them. So the server needs to know where it starts, where it ends, so it can take out the payload from that message. The server copies it to the memory and creates those two parameters. One is the PL, which is pointer to the beginning of the payload, where it starts, and the other one is the length of the payload which it called here payload. So you can already see that I'm using by the way the exact same names that are from the original code and I think you can already start to see that there's some kind of confusion here. The payload is actually not a payload, it's the payload length and PL is the payload. So then the server prepares the buffer or the place in the memory to prepare for the response. So it prepares, it creates a buffer with the same length of payload. And then what it does here, it copies a block of memory from one place to the other, so we have the payload from the message copied down or whatever to where the response is prepared. So here we can see that if we send heart, the server begins by pointing to the H and then takes three places, this is the block of the word heart, it takes that, copies it to the server, to the response. And then it will decrypt it and send it back to the client. What you can see here is that the payload, which is the payload length, is actually completely controlled by the client. The server doesn't check anything, there's no any sanity check on this power meter, on the payload length. So here we place the client with if the evil tells us that the payload length is 500 for the world heart, this is where the server accepts and sends back how we saw before, those all 500 characters from its memory. So the reality was that the blocks that the server could send the maximum length of block of memory because of the way the heartbeat feature works is 64 kilobytes of memory. That's a lot. So let's view a short video of how heart rate attack looks like. So I prepared an NGINX server, they give me a canthusia so I cannot use all the features, it's amazing too. So here I prepared an NGINX server that is vulnerable to this heartbeat open cell. I'm using the best certificate there is and I'm using TLS12 so the problem is not with certificate and it's not with the TLS12, it is with open cell for availability. This is just to show you what the issue is here. So this is the code that we use for that page here. We have the phone where we put in data. I want just to show you that the world password there, the value of password is not used anymore and just go to the server's test there in the memory. We don't do anything with it. And here I'm going to use NMAP-NMAP as a tool that can check the environment, tells me if there are any servers and if they are vulnerable to anything and we found out that this server is vulnerable to heart rate. And then we can run a Python script that exploits the heart rate and basically it tells the server to send it back that memory. So now we can see the memory is quite empty because we didn't send anything to the server and now we're going to send something to the server. We're going to fill out this form and send that data to the server. So in this case it's a victim, it's one of the users that use that open cell server to connect with it and send it that data. And then we submit, we can look at the post request. This is the cookie that was returned by the server. We see that it returns some of the data and then with the parameters that we sent, you can see that we also sent the password. So now the next move of the attacker, which is someone else here, is again to run the script that exploits heart rate and then we can see here that we actually get something in the memory and this is exactly what we sent. If we look at this, this is now in the attacker's hands, get the cookie, all the data that we sent and also the password. So that was how heart rate looks like. And what I saw here is that the server sends its secret to the client, but it gets worse than that. What can be worse than the server sending its secret to the client? So immediately after this disclosure of heart rate in the community, people started to ask, well, this is a lot of memory that can be sent back from the server to the attacker. 64 kilobyte of memory and you can send those requests again and again and again. You can really exhaust the whole memory down on the whole server's memory down. So there were discussions in the community whether it will be possible to download a private key on the server. And Cloudflare, a CDN company which we'll look at a bit later as well, decided to set up a server and GNX server again in their Cloud and it was vulnerable with this version that is vulnerable to heart rate. And that's the community, go ahead, attack that server. Let's see who gets that, if someone can get that private key. A few hours later, two guys download the private key and a few hours later, another two guys download the private key. And that was really a shocking thing because private key is basically the fingerprint of the server. So now if I had a private key, I could maybe spoof your bank server while you're sending me all the information. We don't know anything, could be Google, could be anything that was spoofed out there. And this was open for two and a half years. But it's still getting worse than that. So what can be worse than that? Remember that we had 70 or 80% of the servers on the internet exposed to that availability. What is worse is that there's no audit. We don't know if something was exposed or exploited. We really don't know. We don't know if all of my Gmail emails or my secret or my bank account was compromised or not because hard bit, since it was a side feature of open server, they decided not to have audit on it. So we actually don't know. And that was open for two and a half years. Many people now, after the latest findings, they think that maybe that could be also a backdoor that was spoofed by some company to make it happen. We don't know. So what we can learn from that? We can learn from that review is a responsibility. Review is very, very, very important. And it's very good if you have your reviewers as your senior developers, do the review for the other developers, for the more genuine developers. It's an opportunity for the senior developers to mentor. The junior developers give them guidelines, tell them about best practices, how to write code. And of course, they have to be very, very responsible by accepting the code. So if we look at the original code of hard bits, the one that was submitted, we can see many, many issues they all like about parameters that don't tell anything about those parameters. We can see hard-coded values. We can see use of macros that it's very, very difficult to understand what those macros do. And also, we also already saw those misleading names that can be very confusing. So the reviewer had to go through all those issues, you know, all these others and it can block the reviewer from actually seeing the right or the real vulnerabilities. The vulnerability, by the way, is the last line here at the bottom. So if you see some kind of code that is written like that, it's better that you reject the code, train the developer how to write a good code, because then if you have a good code that you can actually read and understand, you get a better chance to see and to find those vulnerabilities. Don't forget to audit everywhere. And it's even better to send the audit through C-Slog to all kinds of alarm systems that can give you some kind of alarms. There were also, after hardly, there were so many discussions about the security of open source, because until hardly everyone believed that open source is more secure than closed source, because let's say when I'm writing a code in a closed source company, I'm writing my code, someone reviews it and then we submit it and that's it, nobody ever sees that code again unless there's a fix to do or change. This is unlike an open source where you have the code is open there and everyone can review it as many times as they want. But we found other issues with open source that many, at that point of time, many open source projects were not really go and follow any best practices of development. They didn't have the full SDLC like you would see in big companies, not all of them, but this is what was very common in open source projects and today this is most of, I think, I didn't come, but I know this is very known issue in most of the open source projects that I know. Definitely the one I joined, I was working on the open source of the efficient changes that was also led by Microsoft in the University of Washington. We did the whole SDLC as you would do in any company, you know, statistical analysis, reviews, everything there. Moving ahead in time, we get to this year, 2017, February, and we get this tweet by Alex Dower from Cloudflare. He tells us about TicketBit that is found at F5 big IP appliances. So let's see what TicketBit is. Before I get to TicketBit, I just want to give you a bit of a background. So before we mentioned the handshake of TLS, so TLS handshake, which basically is there to create session keys that you can use to communication with those session keys. And then once we ended up, there's another way that the client can try to resume the session. So the client, we did a like a pause, a stop in the conversation, and then the client wants to reconnect to the server. You want to see whether we can use the same keys that we still have from the previous session, whether they are still valid. This is to save the generating of new keys, which is very pricey in terms of performance. So you can see here that the server holds like lots of session IDs in their keys, and the client has its own key, and it sends the session ID to the server, asking them, basically, can we continue and use the server's checks in his memory, find that session ID, and says, yes, we can use that session since I have it, I have the key already. At this point, the server changed its state. This is the artificial intelligence way of thinking, we think in states. So the server changed its state, now it's moving forward to another state, the state now it is in the session state already, sometimes we call it cypher state. And it tells the client, yes, we can use that session, the way it tells the client that he used it is very important to understand, it just sends the same session ID. If the server doesn't want to continue with that session, because it doesn't have the key anymore, because it's the time pass, it's not valid anymore, it will send a new session ID. This is the way the server tells the client whether to use that session again or not. So in this case, we want to use the same session, so the server just returns the same session ID that he got from the client, which is the session ID that the server gave it before when they just curated the keys and did their whole end check. So at this point, also the client got the session ID, it compares the two session IDs, the one that he said, the one that he got. If they are the same, it also moves to the state of session and both can continue from now on top with the same keys that they had from the pivot session. So now let's see how this protocol works with tickets. So tickets are there to save on the server side. There are servers that connect with lots and lots of clients and servers and they don't want to spend their memory and resources of managing sessions on their side. So in this case, what they do, at the end of the session, at the end of the end check, just before they move to the state of session, they already have got the session key, I'm talking about the full end check, they just generated the session key, they encrypt it with what you call stack, which is the session ticket encryption key, it's a special key for that, and they send it to the client. So now the client has the session key encrypted on their side, they also have the session key, not encrypted, but they can use, and they have also the ticket. The server doesn't keep anything, completely status. So now the server, the client wants to resume that session, it has the ticket, it wants to send that ticket to the server, this is my key, I want to use it. Now we want to use the same protocol, if you remember the same, resumed here as protocol, the protocol told us, you have to send a session ID, if the server accepts that session ID, return the same session ID, if not return your new session ID. So we don't use the same protocol, we don't want to create a new protocol, and that's, and now we send the ticket, we don't have a session ID, so in this case the client generates a session ID, this is generated by the client, sends it to the server, the server gets that session ID with the ticket, if the stack is still the same one that it can use before to encrypt that ticket, it cannot decrypt the ticket, it has the key, both sides have the keys, and they can continue the session. If not, the server will just send a new session ID telling the client, let's go ahead and go to the full TLS engine, so now we can look at what is ticket data, and starting with giving you a bit of background how it was found by Philippo from Cloudflow, so the situation was like that, they had Cloudflow as a CDN, I mentioned before, it's a quick CDN, it's content delivery network, what they do is they want to speed the delivery of content, they enable servers to get to and to reach to further server and the globe, they use that by caching the internet on their data centers, they also compress, so you can see here that on the server, on the server side there's an agent, they call it Cloudflow, they call their agent Aragon, that enables that connection between the server and the data center of Cloudflow, and the situation here is that we have also A5 big IP load balancer in the middle, so the connection of the TLS is between DH server of Cloudflow and A5 load balancer, so we're going to concentrate over there, the problem is that the customer, the Cloudflow customer complained that something with the connection doesn't work for them. And if you go look about how the Resume TLS looks like we see immediately that every time we do, we try to use the Resume TLS, we get the front layer of the communication that doesn't work, there's a confusion in the protocol, the server is in one state and the client is in another state and we're going to see how it happened, why they're both not on the same state. So we go back to the Resume TLS with Ticket, here the Ticket is on the Cloudflow side, it wants to send it to A5 load balancer to resume the session that they had before, it sends the Ticket with remember a generated client ID, the server checks that it says yes okay we can use that Ticket, we can use that key, I managed to decrypt it, I have the key now, let's use it, I'm sending you back the same session I sent it, Cloudflow checks that it says no, it sent me a different session ID, so I'm moving on my state to a full TLS engine, the server we have to remember is now already in a session state, they cannot communicate, the protocol is completely broken at this point and that's why we had this thought available. So let's see what happened there by looking carefully into the response of the session ID from the server, what did that server send back? So this is RG's welcome, this is what Cloudflow sent, the session ID they sent, you can see 16 bytes of session ID, this is the session ID that Cloudflow sent to A5 and B, and this is the session ID that A5 returned, so we're done sending 16 bytes, A5 returns 32 bytes, you can see that the first 16 bytes of A5 are exactly the same as the one they got, so they really want to go ahead and use that session, it's the same session ID but they have now another 16 bytes over there, what are these 16 bytes? And this is what made the confusion on the clients and on the Cloudflow, suddenly it got something else than they sent, this is why Cloudflow decided to move to TLSNG, so these 16 bytes are memory, so apparently A5 always returned 32 bytes of session ID, no matter what you send them, so the client, because this is a generated session ID by the client, it cannot even also send them a session ID by one byte and get back 31 bytes of memory, so A5 is not completely wrong there, if you read the description of the protocol, the protocol really suggests to use 32 bytes of session ID, but it's better of course if you get something from the client, it really better that you also check it, you have to check what you get there. So these are the appliances that were exposed, it's all the big IP family of products of A5, of course this is fixed, lessons learned, design overview maybe you could find this issue, also what I thought is maybe dynamic analysis and fuzzing, sending different lengths of session IDs maybe could bring this up this issue before we release it to the customer, so moving ahead in time about nine days later, same here, same just nine days later, now we get this tweet by Tavis Amande, who knows Tavis Amande, very very famous person, he's one of the leaders of Project Zero, Project Zero is a project that is established by Google, by leading security researchers, they want to make the internet more secure and he finds all the time issues, he really finds all the time, if you connect with him on the Twitter, it's a treasure, so we saw that tweet, could someone from Cloud for Security urgently contact me, if this is the worst thing that you can get on the Twitter, you really don't want it, it's a nightmare to get something like that, and this is what we're going to know, Cloud Bait, so Cloud Bait is with availability in Cloud for, we mentioned Cloud for before, three times already, this is the third time now, so it's really funny, I'm not working for them, so again, CDN on top of making connection much faster and performance wise, they also give a lot of security help to their clients, and so it's also a security services, and Tavis says, look, Cloud for I've been looking customer Sessions for months, from Uber, One Password, Fitbit, OkCupid, you really put it in the internet, he marked it, but there were some messages from OkCupid, people, you know, full messages, this is our page look like, usually you won't see that, but in this you can actually see the next memory on the page, usually the page doesn't have something to show it, but it's always there, and get with a response from the server, so lots of memory coming from Cloud for, leaking from Cloud for, now, this is a closed source, I don't have the source, but I managed to do some server coding after, you know, reading a lot of blogs by people from Cloud for, and by Tavis on Monday, so I managed, I think to understand what happened though, and this is what I'm going to show you now, so Cloud for, they used the comments, thanks so Cloud for used third party library that library is used for many things, here what you can see is to check whether we are at the end of the page, then it's checking out the end of buffer, but use it to check whether we are at the end of page, here what we, the P is our current character and PE is the character at the end of the file okay, so we can here we can increment P step by step, until we check whether we are at the end of file, whether we are at PE we want to know that, and then we go to some kind of end of file small time link, so anyone sees an issue here, what does it do? that is not the issue, P is like a pointer and P is, we see almost we see an entity that we know of them, the problem here is the equal sign, so this is basically we check buffer, okay the end of buffer, you don't want to use equal sign because if in the code we manage for somehow to pass PE, because we recommended PE a bit further we will never be equal to PE anymore right, so we just continue we will never be again equal to PE, and therefore this is a classic way to create a buffer overflow or buffer over one, depends if you really write, the better way to do it is this way, okay bigger or equal, so even if I pass the end of the buffer I know to stop over there so let's look quickly on this cell code here, this is a parcel that Cloudflare wrote and they use that library that we just saw so the parcel basically goes on the HP man tries to understand what tags they have and do some kind of smart things with them, so we have one tag that ends with the right end of tag sign and also at the end of file, this is a very extreme situation this is what also Cloudflare said look, this is a very extreme situation it's an end of tag, end of file and only if the end of tag is wrong, there's an error in there okay, so this is, we're going to look at how it happens, so let's run the script here let's run the code here, it goes on this tab, okay and then it checks whether at the end of they find the end of tag sign yes, I found it, let's do something very smart with this tag now and also I want to check whether I'm at the end of the file, so now, before I'm going to check whether I'm at the end of file, I'm moving that pointer one step backward why do I do that? because we saw before that the library that we use to check whether we are at the end of the file is actually incrementing it one step further, right, so we want to stay on the same place to check whether we are at the end of the file, that's why we moved it one step backward, and now it's moved one step forward, we are at the same place we are equal to P we are at the end of the page and do something else with it basically stop reading now let's look at the situation where it ends in the wrong way so we got to the wrong sign there's error handling over there look what we forgot we forgot to move the pointer one step backward and now we also have this incrementing of the pointer even further we already passed the end of file you can see that, we already passed the end of file we'll never be again at the end of file we're already after it so we're just reading and we'll continue to read the content and we'll just send it away to the client so in this case because it's Cloudflare, it's multi-tenant lots of different clients different people connected, they just send information of other people to that client and that's all for memory you can see we had two issues here we had to have each one of them you can say maybe it's low but together they created some very critical impact one of them if we could fix one of them or if we had only one of them we wouldn't have that issue that we test so it's the combination of both issues, it's really very condition to have something like that that's in the code review again all the time code review is very, very important unit testing we used to test extreme cases maybe this is an extreme case that could be found using the testing so five minutes later and we are in the summary so we wanted to make a connection between the client and the server here we see it's an open connection everything can happen there I can put many in the middle and tamper with the data read the data even stop the data whatever I want to do and therefore we want to have the CRI by confidentiality integrity and availability we want to implement that on our connection over there currently we don't have anything of that there's no insecurity in the situation and in order to have security we use the less connection we use firewall and we use even cloud security services now we have the full CRI but what happens if those are vulnerable basically we are now completely vulnerable again the conclusion is that security products like any other product they increase the attack surface and therefore they can expose to very bad things as we saw down and we always have to follow those standards and best practices like in the whole industry of design, code review, audit logs clear circuits for memory test and test and test, unit test, static, dynamic penetration test everything you know, all that should be there thank you we have two minutes for questions you can read everything was clear I'm not sorry I'm sorry, Nancy how well did they go? open access so of course they fixed that open access is okay so this was a very interesting thing because after I've been people understood the open access is really the infrastructure of the internet and they created a group of I don't remember the name, actually had it somewhere here that is sponsored by many many companies to be able to create a bug bounty on those infrastructure open source of the internet because it's open source it couldn't do anything like a bug bounty and today if you find an issue with the open access there's a consortium of companies that can give you something so this is something that's changed to understand the importance of open access and those kind of infrastructure things on the internet and we also want to open the community in order to make them better as you guys are leaving we have a short coffee and pastry break over here outside he's down in the garden and so on for 15 minutes we're shortening