 Yeah, let me verify that standby. We're going to click over here and I have to turn myself on yet we're in. Right, great. Yeah, share a shot with the new speakers and so, although I'd love to be handing you a bottle right now, I'll ask you to take your favorite drink and join me in a quick shot, guys, so cheers on your talk. Okay, so that concludes the Q&A session. It's been good, good, good questions. We do actually have some questions already queued up on the chat so the first one is other than the one UPS you guys demoed, what are the sort of devices that you successfully exploit? Yeah, so for the remote code execution. We exploited two different vulnerabilities on two different devices. They appear in our white papers with full details of the exploits. One is the UPS and then the other one is called a Digi Connect module. It's a system on module for Ethernet connections and then there's a few different devices that embed that module so they all, like the exploit would work the same on any device embedding that system on module. But that's just those two. And then for information leak and just denial of service and stuff like that, we exploited them on quite a few different devices just to make sure they're vulnerable. Great. So we have another one that you have the track identified script that you provided, but you're wondering if there was more active indicators found and if there'll be an update to the track identification script. So we were staying with the same identification script that we've reviewed today. We have seen some false positives and we're trying to investigate that and see why we get those false positives. There's something interesting going on. The reason we're getting false positives and we're not quite sure what it is. So there might be updates on that front. And we have released together with McAfee, we've released IPS IDS signatures. So I'm not sure if that's the question or regarding detection, but for detection we're using the same scripts, which anyone can just ask for and we'll send you by mail. And there's some false positives with those scripts, unfortunately. Great. So we have another question about, is there a way to safety release consumer projects based upon ESP 32 or ESP 8266? If no, what hardware development kit would you recommend to start with? Is that relevant to your talk? Okay. Moving on. So what research methodology was your use? Did you manually fuzz, review or something else? I know from your talk, you talked about static analysis and I was actually interested in hearing more about how you did that. Yeah, so we mainly reverse engineered and we were starting with the track demo they were offering on the website and now they removed it. And then we purchased this Digi Connect device and we found some headers online and we had luck because this Digi Connect device has symbols. So we can compile the custom firmware and we have debugging capabilities. So we reverse this firmware and that's basically how we found most of the vulnerabilities. And along the way with the research we purchased more devices and found some more variants of the vulnerabilities. Yeah. However, on the UPS device that's relevant regarding the methodology of installation, we saw a slightly different version of track and we also had, well, we didn't invest much in creating debugging capabilities, we didn't have a JTA connector or GDB interface or nothing of that sort. So we relied mainly on a labor static analysis and also some relatively informative crash dumps, small stack traces, stuff like that. So yeah, so researching heat corruptions using static analysis, I think maybe not traditional. So had you had better tools, would you have used them or do you think you got the results because you went the way you did? Well, if we had better tools, we would definitely use them, but we didn't see fit to invest in creating these tools. Just because the exploitation primitive is relatively strong, it's controlled overflow on the heap and the heap of these devices is very simple, rather deterministic. So we decided to go for the low hanging fruit first and paid off. For the actual vulnerability research, I think the manual static analysis was really the way to go. Just because there were so many different architectures and different versions and different types of devices. So there wasn't like one tool that could be used to emulate all of these or fuzz all of these or something of that sort. And there were a lot of special cases of different types of architectures that weren't supported by IDA and stuff like that. So yeah, I think I think this was a tasteful manual analysis. Great. So another question is that your second white paper is it independent of the first one or did it build on the first one? Completely independent. I mean, they're related because they're the same vulnerable. They're the same ripple 20, but different vulnerability, different device. Right. So for the DNS vulnerability, does the show code have to be off on Merrick? If so, is it possible to get an RCE on other architectures? Well, for on the UPS device, as we said earlier, we saw a slightly new version of Trek, you can also see this in a presentation that on some versions, also know where the characters in DNS domain names are enforced. And then that's the overflow you have to use often the characters, but on some versions, it is not enforced. So on the UPS device, it wasn't forced. And luckily, it was also the processor was also x86 space. That's solved properly. I mean, out of my church on x86 processor. So, I mean, it could probably be done in theory, in a different way, for example, you can have a location primitive that allocate non alphanumeric data in the heat, such as just, you know, the DNS packet data, for example, it's not alphanumeric, so in theory, you could do something different. We just chose the easy path. So yeah, it is exploitable on other architectures. Great. Yeah, I think it depends on the heat implementation or so. So some strike versions use a heap, which is not validated and asserted as as the UPS. So, in these circumstances, it's probably more easy. Great. So another question came in. Are there similar TCP IP stacks that might be interested to look at interesting to look at? That's a really interesting question. So with one of the vulnerabilities, we're not going to say which but we we sort of we look for reference in other TCP IP stats. And then we found that about half the TCP IP sets that we looked at have the same, like have variants of the track vulnerabilities and that's research we're going to be releasing in the future. So, like, different TCP IP stats seem to make the same types of mistakes. So that's a good way to do research into TCP IP stats. And then as far as which TCP IP stats to look at, there's, there's a bunch like just open Google, you know, depending on if you want proprietary open source just typing your search and you'll find I might know what he's hiding the There's a bunch of proprietary or open source Yeah, there seems to be a follow up question that expands out a bit beyond just the TCP IP stack. Since that was just an example of a library at the root of a huge supply chain and did you see other, you know, libraries or other, you know, Stacks that you would consider evaluating next Yeah, we did. But we're sort of still deciding what to allow you next so we're keeping those to ourselves at this point but we did see other super interesting stats without really long supply chain. I mean, I mean, that's how IOT is built just like Lego blocks right so just take any device break it apart when you get to the smallest piece. That's your research target. Fantastic. So, another question came in how do you explain how you got to quote hundreds of millions of affected devices unquote. So, there's, there's quite a lot of devices of different types and we sort of try to sum them up. The majority of the numbers actually come from Intel devices and HP devices, and then did you also adds quite a big number of devices. But we just sort of tried to assess given open information how many of each of these types of devices is sold per year to reach hundreds of millions. If you sort of count all the devices that contain the code you probably reach billions. If you count all the actually affected and turned it on, it will be between tens of millions and hundreds of millions. So it sort of depends how you count. But yeah, most of them are printers and Intel and T. Great. So, one question I had was it when I look at your website default to accompany your presentation it looks like there's a quite a big team that was involved in pulling this together. I was like, I mean, it sounds like the core of you all were at JS of J S O F, but how did you rope in the collaborators that you used on this. So the, the vast majority of the work was done at J soft with Moshe doing. The work from the beginning finding the vulnerabilities, etc. and Ariel doing the exploitation work. And then for very brief periods. We use some outside help just when we were under under timeline or something like that. Most of the people that you see there most of the names actually work for J self. You'll see a few names that don't work for J self one is the kid of one of the employees who we just mentioned as, you know, because his mother was away. And then somebody else that helped helped along the way but most of the work 99% of the work is done in us. I have a really bad question. Are you ready for it. I mean, you talked about in your presentation, why you gave the 20 but why ripple. Okay, so we'll give you the real answer about why 20 that nobody knows. And the real reason why 20 is because if we called it ripple 19 and people would associate it with the current affair current situation. So that's why we went for 20. I mean, there's other reasons, but that's that's the behind the scenes. And just between you and me and all our viewers. The ripple is because of the ripple effects, right this one vulnerability that sort of expands and the impact expands and expands and expands it's how we experienced it but it's also how over the years straight stack. I went from from device to device and got wider and wider effect and wider impact and embedded more and more devices. So we felt like that really captures the supply chain ripple effect. That's cool. I like what you're going with on that. So how big this one got obviously this got a lot of attention. There were a lot of people talking about this. How has the added attention to your work changed the way that you do your work to change your daily life if you found more people are asking you questions about more stuff. So it's been quite busy. We've seen all kinds of responses all kinds of questions all kinds of reach out. Mostly to date most of the stuff has been sort of reactionary network operators. You know, want to figure out what to do about ripple 20. And then of course some researchers interested in the work reaching out asking questions companies wanting to collaborate on helping mitigate and fix these issues. So hoping for sort of more long term more companies thinking about the next time calling it ripple 21, like how do you prevent the next time. How do vendors use security hygiene exploit mitigations, you know, how do you look into their supply chain, because I think that's like this is not the last time this is happening right. Somebody asked about good research targets so there's already people in the audience looking for the next supply chain vulnerability. I think by this time next year we'll have another one for sure. Probably not as they're going to be publishing it but somebody will. So I think that's the, the interesting talk about how do we go forward from ripple 20. Great, so we have a question. How do you evaluate the patchability of such high level supply chain vulnerabilities down the line, especially with regard to less frequent updated as frequently updated IOT. And so we sort of know it's called a common knowledge or something that we say that some devices are unpatchable and some of the companies we know, see operations. Some of the companies are still investigating whether they are vulnerable. So it's going to be difficult many of the device and I'm going to be back for years just for those reasons. Some of them it's difficult to patch. Somebody reach out and saying, we can't patch these devices for a while because they're too critical. And like there's serve a lot of customers and we can't afford any downtime. And they're sort of trying all kinds of virtual patching network level mitigations. How effective that is, I guess it could be effective if done right, but we definitely assume many of the devices are not going to be patched for years. It's my personal assumption. So I guess a follow up question that would be, you know, if you're a developer of IOT what's what's the lesson to take from this. So, so I think one lesson is look into your supply chain and understand their security, you know, ask them the right questions. And another question is, prepare for vulnerabilities using good exploit mitigations and security hygiene. Right. So, some of the vendors affected, we're actually able to release a statement saying yes, we're affected but it's not that bad. Like green hills and Intel to some extent with some of the devices, because they, they use different types of expert mitigations and security hygiene and then, even though they didn't know about the ripple 20 vulnerabilities. The impact was lesser. So, like, and then even with with the with the Schneider UPS devices, which of course being patched since the heap was was sort of stronger version of the heap. Then we've seen in other devices and therefore was slightly more difficult to exploit them. Say the digital connect device that used an older allocator with, which was less safe. So, third party risk supply chain and export mitigations are the top two. So another question from the audience, more of a general research question. When starting a project like this, how does the timeline work? You say like, let's give it a month and then you give up or you allocate three people until they figure out something or is it some some other approach. So it's always difficult with research. But we've been doing this for a while. Our timeline worked more or less like this. We took the death con and also black hat cover papers finish state. We took as much time as we thought we might need back and then we. And then, you know, we, we, we also scheduled time for the disclosure process, the coordination process, etc. And that's when we started the specific research. And then towards the end towards the disclosure date, we were like scrambling with the exploit and we actually put the point together and not three weeks from start to finish. We sort of just dedicated a specific amount of time to his death con and black hat. Oftentimes it's very, very hard to schedule this type of research. You never know what you're going to find how long it's going to take. Great. So I guess a follow up question would be, I mean, did you, did you feel confident you were going to find something when you started or was this mostly a hunting trip? There's always something. I don't think we've ever encountered a target where we haven't found anything. So yeah, there's always something. Great. So the, I think you answered this a minute ago, but one of the questions that came up out of band was did the ups devices that you demonstrated that did those have patches available. Did I hear you say yes now? They do have patches available. Of course, you have to use the to patch your device using the patches, but they are available after installing them. Right. So we've reached that point where I, you know, my, my ability to do radio completely fail me because I've run out of questions, but and so folks, please. Please keep the questions coming the last eight minutes we've got here. I will have offer you guys a chance to elaborate on anything that you wanted to add to your, your talks or any additional points you wanted to make while we're looking for more questions. So there was that there was an additional question we started with of you had several things you talked about at the very beginning of your presentation of these are a number of vulnerabilities that you showed just one. Would you talk a little more maybe about why just one and any other things we can look forward to seeing in the future. I know you talked about it a little bit, but I have a question other questions about what are the work you did. We chose this vulnerability because we think it's sort of the most interesting vulnerability. And we had a cool exploit on a demo to show that it's, it's real. We think it's cool because it will bypass that right the DNS request will exit your corporate network potentially depends how it's configured. It's actually done with triple to any because we're, you know, we're looking to work forward to our next research project in the research that there are some offshoots some other stuff that we found out that we're still working on and we're going to be releasing shortly. We do have some new research that we're going to be releasing in a few months. And then of course we do a lot of, you know, work for for customers. Most of our work is just customers. We do penetration testing and research and stuff like that. So, but from the from the research labs side, a little bit more about Ripple 20 some new stuff we found out. We sort of released only on Twitter, but we never called it documented on a white paper is, and the listeners might find interesting is that the track stack actually supports encapsulation of IP in IP, and it will forward packets to itself. The destination addresses is the right one, which is really cool because that can help you bypass firewalls. And, you know, another network equipment that might stop exploitation. And, and we found that out when we wanted to test the exploit routes over the internet. But the one for the did you connect. And, you know, just our company firewall was blocking it so we encapsulated the packets and, and, you know, and they bypassed the firewall. So that's something new. But mostly we were working towards our next research and I think in a couple of months we should be able to release. Currently 100 disclosure coordinate process. Great. Yeah, unfortunately, your future research becomes def con talks, you won't get to drink at those. So that you have to that that's only the first time so we did have one more question come in and said is jsof going to create any any tools like nmap NSF, et cetera to help find the devices that may have these issues. So we already have detection scripts. You can send us an email that is on our website and you can get the detection scripts. And also, so I'm correct me if I'm wrong, but we have the filters for what's it called. So, so we have the there's passive detection scripts for the exploit themselves. We have some passive information about how to take the track stack and some active detection scripts. You can email us them to us we also plan to release them open source which is kind of for the graduate yet. They are also available. I think tenable release them as a package and the release them as a package. Probably a few others that release them as a package, but those are the ones we know about. And yeah, until we release them open source, you can just email us, you know, anyone whose, you know, legitimate request will just send you the scripts. You know, take us a couple hours and you get the script. Great. If I remember correctly, your contact information is in the beginning of your talk, correct. I hope so. If not, it's just ripple 20 at jsoft dash that's J S. O. F. Hopefully someone was typing faster than I did when you said that. But so we do have one last question. What specifically reckon what specific recommendations would you have for third party library vendors that could minimize the possibility of vulnerabilities like this. So code review would be a big one exploit mitigations. You know, it would be another one, but for third party library suppliers like track that, you know, relying are relied on by pretty big amount of IOT and critical devices. I'd say penetration testing a really good security review and secure code review. As well as just security in the design is absolutely necessary because, you know, doing this once for track, it's just like we found one vulnerability and it affects everyone. Like one secure code review would protect everyone. So the deeper you are in the supply chain, or why I guess deeper depends on where you're counting from but the further the further at the beginning of the supply chain you are, the more you should be investing in your security and the more your clients should be asking you about your security and probably will be going forward. Yeah, I think one of the lessons from your talk was to recognize the fact that these things do have. Like you said that ripple effect. And if you're if you're a maker of libraries, you know, you know, be careful of what what where that your code may end up at if it has those vulnerabilities right. I don't think anybody building TCP IP stacks were were imagining that they could be protecting incubation technologies or other medical systems that keep people alive right. So, so I think a lot of these devices, they do realize where they end up. The thing is, they started writing the code 20 years ago and nobody was thinking about security. And then the code probably worked really well, and just nobody wants to touch it right and, and somewhere along the line, you know, all these companies sort of forgot that, you know, they probably revamped their own security processes and activities but, you know, maybe they figured if there's something in a third party is that it's like a third party problem or maybe they didn't realize, you know, they've been using the software for 20 years. So I don't think that it's that track didn't know whether the code ends up. It's just, you know, 20 years ago, nobody was talking about security. And that's when the code was written by a lot of this code is really old, not just fact, all IOT code, a lot of the code is, is not very fresh code. Oh, great guys. So we're at the top of the hour. I want to be respectful of your time. I really appreciate you guys taking the time to take these Q&As and I appreciate all the questions that have come in on the, on the discord chat. Any last words before we call this folks. All right guys. We really look forward to more good work from you guys. Great questions. All right. Have a good one. Thank you. Bye.