 Our next speaker is Sergei Gordechik, Sergei has been doing security research, products and services for the past 15 years, more than 15 years, since 2011, he is director and scriptwriter at Positive Hack Days Forum, the largest cyber security event in Eastern Europe. Sergei has for instance been working at Capersky Lab and Positive Technologies. He's also a visiting professor at Harbour Space University in Barcelona and leader of the SCADA Strangelove Industrial Cybersecurity Research Team. Today Sergei will talk about how to hack software to find networks and keeping your sanity while doing it. Let's give a warm round of applause for Sergei Gordechik. Hello, hello, good night. Let's start to refresh our memories. This is a big honor for me to speak on the 35C3 because my first talk here was on the 29C3 with SCADA Strangelove team and I think I can skip this introduction, thanks for our host, because everything is here and what I want to say about me, still I am very Russian, live in Abu Dhabi and do all this stuff because I saw this album in the airplane when I flight here. So except Bitcoin only. So let's start to talk about software defined networks. What is software defined networks in general and is divine in particular case, is a magic. So according to Gartner, it will kill MPLS, it will replace all your Cisco and Juniper devices or Huawei if you prefer Chinese, but it's bad, you know, according to last news. And it will solve all your network problems because it has AI inside and it will magically optimize network operations and do everything including security. So because it's perfectly safe to implement inquiry networks, efficiently and security. So okay, sounds good. What is actually software defined networks? It's so simple. If you're familiar with the routers and switches, like it's just accept packets and send it somewhere according to routing tables or mac tables, but software defined networks is quite different. And when we tried with our team to understand how it's work, our first impression was like this. We are hackers, we don't want to deal with this shit, but the only challenge we met before you hack something, you need to actually activate it and make it works. Which is why we start to understand how software defined networks and SD1 works. So what's the main difference between traditional LAN and SD1? In traditional LAN, you have different devices which solve very specific purpose, for instance, switches or routing or firewalling or network load balancing. In case of SD1, you have just a server which runs operation systems, in most cases it's like Linux. And on the top of this operation system, you have very specific models like CPE which do specific network functions. It can be firewalling, it can be routing, it can be switching, it can be network load balancing. So you replace specific devices with one big server which magically do everything with AI and in the cloud, sure. So in the SD1, we have several layers. So low level is data plan when actually you process packets and decide how to go it, in which way how to firewall or drop it. We have control plan which manage different routers, different devices. We have management plan which can help to apply policies and orchestration plan because it's serious things that should have something which called orchestration. But on the technical plan, again, we have hardware with operation system and a layer which called network function virtualization. What's network function virtualization? The way to apply different network functions, sorry, to the specific device. It's very useful, for instance, to the network operators who will provide you with the specific box. And if you want to activate any functions, this can be a web application firewall or the sandboxes. We just upload very specific virtual machine. It can be Docker, it can be QVM image to your hardware and you can start to use it because inside of this box, you already have all the system infrastructure which process packets and pass it from one virtual network function to the other. This helps to organize things like the service chaining so you can distribute different network functions on the branch level or in the cloud level or on the HQ level. For instance, things like content filtering which can be very heavy from performance perspective can be distributed. As example, on the branch level, you can use simple things like the antivirus to process content. On the HQ level, you can use more heavy things like a sandboxing. And if antivirus or specific rules see that this content is suspicious, it's forwarded to HQ through the MPDS or other VPN and next processes in the HQ. Or you can also analyze it in the cloud through the simple things like the cloud threat intelligence where actually your SD1 box will send MD5 hash to the cloud and check is it good, not bad or send all files to the cloud to double check it. Not bad and I think that is why SD1 become more popular and you can see that the military guys in US decided to switch to SD1 because of security, cost saving and all these benefits. Okay, security sounds very familiar for us and we decide to obviously hack it. I think most of you have experience in hacking of different network appliances and you know that sometimes you need to have complex things like an R&S soldering kit or debugger, etc. But not in this case. Because SD1 actually is not appliance, it's virtual appliance. And to start hacking, all you need is to go to the AWS or Azure and just activate this virtual appliance for, I don't know, 10 bucks per month and next step is get root on it. It's a very good talk presented on different conferences including Zero Nights, how to hack virtual appliance and we use this like a checklist for our research. Because if you hack in virtual appliance, you already have access to this system. You can mount file system to the other virtual appliance. You can grab ETC shadow. You can find a lot of different backdoors just through the static analysis. But all the good things starts from the Google. For instance, to find admin password for one of the SD1 appliance, we just Google for GitHub and found what most of the scripts which use it to automate with appliance use username administrator and password 1 to 3 and actually we found that this username and password is hard coded there because there is no way to change it. Next step to root it is just to Google for old vulnerabilities. For instance, in Silver Peak, we found that guys had reported vulnerability in September 2015 and it's still working in March 2018. So Google works because Google Foo is strong with this one. Next thing is grab it's always work as a string, you know, and using grab and codeword password, you can find a lot of interesting things like hard coded password in different application, in the configuration files, in the database connection string, in the system logs, because again, it's virtual appliance and someone had deployed before you start to use it. So in the logs, there are a lot of life-interesting information. In the shadow file, like in one of Cisco appliance ETC shadow file, which used this encryption in 2018, you can do some forensic because again, if you get virtual appliance, someone had deployed and sometimes we're trying to hide with this and you can see that cutting the best history, you can find that someone RAM scrub a WS shell script, which actually set up different password, et cetera. So if you somehow can recover a script, you can find a lot of interesting information with this kind of password of admin users and you can see that from this password, you can find the hash and next try to brute force, it just was my guess that maybe because Versa have password, Versa123, maybe other network appliance like Silverpick have similar password, Silverpick123 and this guess was successful and it's good because you cannot stop the progress. If you have experience with red teaming in the enterprise network, you know that Cisco Cisco or Huawei with the administrator and Huawei123, it's quite common password, in this case, it's more complex thing, sometimes very elite like this tattery network stuff, you know, but still, if you did not get the route with these simple steps, again, it's virtual appliance and you can always patch it. So you can change, for instance, hashes in the ETC shadow, you can change boot script, you can change remote management configuration, the password and next booting with configuration and get route password to do next step, security assessment. So security assessment, in the beginning, we did in very, you know, not scientific way, we just hack all the things, but after all, we did some, let's say, scientific research and we have an article, I will give you a link like SD1 threat landscape with the step by step assessment, what you should hack to get maximum results. But let's like mix this thing with funny hacks and the scientific approach. So from the system engineer point of view, SD1 have hardware, which of the shell hardware operation system, in most cases, again, of the shelf Linux and different virtual services. Let's start from the operation system because again, again, everything you saw in the recent talk about BMC and remote management interfaces related to hardware, it still works here unless it's disabled by the vendor, but it's highly unliked. On operation system, we did a very simple research, we just check the patch level of the all components installed on this box and you can see that patch level ridiculously old. For instance, all these things we found, it was an open SSL library, which was released in May 2006. It's for network devices with security functions, but our guess was that they choose this library because this library too old to be when we're able to hard bleed a child. SD1 wins because the oldest open SSL library we found in commercial product wasn't the same before, wasn't the semi-semantic VCC, which was released in 2007. So SD1 is like old school really. Next thing related to operation system configuration is sudo and it's actually everywhere and actually everywhere for management interfaces, including web services, shell, etc. And it's implemented in terrible way, as you can see, Tri-Pil-W data have all ability to execute all command and some scripts just execute any command through the sudo. That is why if you have any small vulnerabilities in the web interfaces, like in this case it's command injection, you can execute command with sudo. So it's again 1990s. Next point is software, not system, but software design point of view. It's from software design where a lot of open source components which implement IPsec, routing, but from management perspective they use mostly HTTP and things around it. So let's analyze HTTP and web management interfaces. So in this case it's not so old school like a system site, everything based on Node.js and JavaScript, it's like very cool, but under the hood you can find hard-core mix from the Perl, Java, PHP, whatever, which like, I don't know, looks like guys develop it the last 10 years. With all this modern Node.js stuff, developers confuse the client and the server because you know JavaScript on both sides and it's hard to understand where is server with client side. I will show you examples. And there are a lot of simple things like a slow HTTP dose attacks, which should be fixed for a long time ago, but still you can stop web interface with few HTTP requests. So few examples about the client side. JSON-CSRF is everywhere, so almost no web interface implement protection from cross-site request forgery in proper way. XSS is everywhere and this is not a problem. So as a response from the product manager of one vendor, they told me that XSS from cross-site application is not the issue because Chrome blocks it. It's just an example of using XSS on such appliances. In this appliance, we can use combination of the XSS and cross-site request forgery to download and upload certificate which use it to authenticate with the server which like control plant with management server. It's just one HTTP request. And obviously, there is no response from the vendor. We just silently fix it, so we decide to publish it through the full disclosure. One example of the perfect authentication. So in this case, you can see that it's client-side JavaScript which just send request to the server to the login status function. And if user is gorget, it's go to the request page. In our case, go to the username and password page. So this 100% client-side. Now we have a lot of checks on the server-side just if you can change it. Second example is just perfect. So I think he tried to port the authentication from the server-side to client-side and understand what is JavaScript which still JavaScript, but it doesn't work in browser. And he just commented and said if username is this and password is this, then go home. So authentication is passed. So this thing. Highlighted box. So this is all authentication on this box. Next funny things which related to SD1 is about different privilege escalation. If you're already able to get access to any of virtual appliance inside, you can try to establish connection from these appliance to other appliance through the local host function. So why this can be interesting? Because there are a lot of open source components for the remote management. For instance, like challenge box which provide you shell through the web interface or Munin or Solar which system management boxes which just bind it to local host. So you cannot establish connection from outside because it's listening to the local host. But if you're already on this box, and you can connect from this box to local host, then this establish, this connection works. And this works because on each virtual, on each appliance, you have a lot of virtual appliance which still listen to one IP addresses. If you have experience with the Docker, for instance. So all docker's containers have all IP addresses, but from the all computer, it's actually connection from local host. So if you own any of the virtual appliance, you can own next all virtual machines installed here. This gives us different interesting ways to escalate privilege inside the box. For instance, if you're able to, let me switch to the laser pointer. You can see it, okay. If you can get access to traffic processor, for instance, for the some TCP stack vulnerabilities, next you can get access to management application. And this management application, in most cases, have no any traffic filtering and trust to management application of all virtual network functions which run on these appliance. So you can do horizontal privilege escalation or next jump down to operation system level. And next go to the management plan, upstairs to the management appliances. But it's really boring to find the application vulnerabilities in such big amount of code. That is why we just download this code from the different network applies and drop it into the interactive code analysis system. In this case, we use positive technologies application inspector. And this help us to find a lot of vulnerabilities including such funny things like, for instance, poorly patched vulnerability in the Citrix SD1, which was patched in 2017. But it still works if you use not get HTTP request, but post HTTP request. So we patched it once again. In this case, better. Also, it's utilization style vulnerability. So it's obviously patch traversal, but it's just reminded that attachments lead to the jealousy and the shadow of grid that is. So if you send attachment shadow, you can get shadow file. So this is a full list of vulnerabilities we found during just source code analysis without, you know, brain interaction. Next step is crypto because the security appliance, it shouldn't implement cryptography. And there are two things. It's SSL TLS and IPsec. In most cases, SSL TLS is used to protect management interfaces between the different appliances. And because it's automatic, it uses different kinds of automatic setup, we found that there are a lot of things related to the unsecure configuration. For instance, we can use SSL TLS without forward secrecy. So if you have access to the certificate, you can sniff off traffic even without full man in the middle. Different things related to the old side pursuits like a triple desk or RC4. For IPsec, we found that in most cases, they used a very strange way to select a certificate or their pre-shared keys, which in most cases just hard coded. So one example from real life, our example we will publish soon from again Citrix Netscaler. These appliances use master control node protocol to communicate between the orchestration plan, control plan and the data plan. It runs on the TCP 2156 and uses TLS without forward-perfect secrecy. And what's interesting, a certificate located in the home tolerate user certificate and account WWW data have full access to this certificate. For some reason, I don't know. Okay, it should maybe read it, but why to write. And what's interesting, all SG-1 appliances we're able to find during our security assessment, we use the same key pair, which is located in appliance. So all SG-1 appliances in the world used to protect communication between management engine, same key pair. So if you know this key pair, and it's obviously you can just cut it from the file system, you can passively or actively sniff traffic, do manage the middle, spoof management appliance. And if this device has any web application vulnerability, you can overwrite it. I don't know why, but maybe if in next turn we will change the certificate, you can download old and do manage the middle again. Interesting stuff we found, we run some tests from the DOS attacks and found that Surikata, which uses it in SG-1 appliances are the ideas, is vulnerable to regux DOS. So it's old story, some regular expression can spend a lot of CPU time if you send specific queries. It's also fixed in the default Surikata kit, but still working some modern SG-1 solution. And for sure if you do some fuzzing, it's always work and give you some fun. Unfortunately, we cannot present reverse engineering part because most of such SG-1 solution, we have restriction in the licensing agreement to the reverse engineering. But just for fun, I think that some of engineers also love Star Wars and have Marvel. That is why installation functions cause Marvel sucks. So just an overview of detected vulnerabilities. So green is good or bad, so good for when they're bad for us, we are unable to detect it. But you can see that most of classes of vulnerabilities, like hard codes, broken access control, old products or Linux components or third party components were in most of such system you can find in most of such system. So just select any SG-1 and make a shot. Interested thinking, SG-1 is zero touch deployment. So that it's a very cool feature. For instance, let's imagine that you have a branch office, you need to deploy a branch office. With SG-1, it's absolutely not necessary to go where or establish Telnet or SSH connection and try to upload configuration. All you need is just to shift this device, note with unique ID, set up it through the cloud console, and shift this device to a remote location. This device will automatically connect to any internet it see around, connect to the cloud server, the node configuration and start operation. So, for example, how it works in the Citrix system. We have visa plans which ship it to the office. It's first try to establish connection with the surround data plans. If no, it's try to go to the zero touch deployment service, present own ID and next visa service will provide all configuration which you upload through the SG-1 center. So, from security perspective, this scheme looks terrible. Why? Because this SG-1 and cloud deployment server should be friendly. But any attacker, not. If you know ID, if you can brute force with ID, you can pretend to be this device. If you have any weaknesses in implementation of this management service, you can own all devices which deployed for the service. And as you can see, even Cisco, which is like the best device from security perspective, we found on this product line, let's say, have enough vulnerabilities. And this, you can see that zero touch provision in command injection vulnerability. So, it's cloud server which to which all network device should connect sometimes. But also, we found very funny things related to the distribution of this device. Because as I told you in the beginning, most of such device can be activated as a cloud appliance through AWS or other cloud services. And we found that most of default images use old version with non-vulnerabilities. So, you go to AWS, you trust the vendor, you activate this system, and you receive attack because there is no vulnerability. It reminds me, in those stories, I'm really old man, sorry. You know, Code Red, Nimda, this kind of key door, this kind of warms. And this real disaster when you just install fresh Windows 2000, which have internet information server by default, and just connect it into internet to download patches, you receive new infection and need to reinstall it from the beginning. So, these things look similar for me, but it's much worse because in this case is a security network device installed on the perimeter of your network. This overview of up-to-date statistics, you can see that very few vendors, actually no one of vendors have up-to-date version in the deployed on AWS or internet. So, I think it's abuse of the force. But it's also a very interesting part of story. As a security researchers, we always work in the responsible disclosure way. And as you can see, according to this article, some vendors, we also understand that responsible disclosure is very important to communicate with community, to fix issues, and we even have product security incident response team in place. Great. But when we try to submit vulnerability report to this vendor, we are unable to find the email of this product security incident response team. So, there is no pool. We tried to Google for it a different way, no luck, but we found that guys who did similar research before, they found a great way, sent email to CEO of this company. Unfortunately, my Google Foo is not good enough, but I am unable to find the email, but I found this guy on the LinkedIn. And he answered it in few minutes, actually, and put me in contact. So, if you try to deal with SD1 vulnerability reporting just do this way. So, we prepared this table about different vendors, how we communicate with researchers, and you can see that actually Cisco, Citrix, and the account, which actually they were not bad, but all the rest is just the beginning. This is my favorite mail from the one vendor. When we send notification, we start to ask them, why we send this email from the Gmail? Do we have official ID? What they mean? I need to present my passport or whatever to submit vulnerabilities. But the funnest thing here, this vendor wrote that their device is not generic web service, which have full access, full Internet exposure. So, after reading this email, I go to sleep and during the night, someone told me. So, to understand the threatless escape of SD1, we built a bunch of script, which works on the top of the different search engines like census, showdown, Google, and use also Nmap law scripting platform to fingerprint different SD1 solution. We have an article published as SD1 Internet census and also some tools, which can generate beautiful maps, if you want to present on the House Communication Congress, because in other cases it's useless. What we found, there are not so many SD1 devices yet in the Internet, about 3,000 management interface, which contains no vulnerabilities, and you can know it in a few minutes. Also, we built some kind of vulnerability assessment tool, which helps you to find no vulnerabilities in these SD1 devices. This example for open SSH patch level, as you can see with some CVs from 2010, 2014, etc. This open source, you can find it on the GitHub. We have two versions, one SD1 harvester, which use Google showdown and census to collect information on all the all-verse Internet. And also SD1 infiltrator, which is a bunch of the network Nmap script engine scripts, and you can use it during penetration testing. So, it's not necessary to be connected to the Internet, you can just use it inside the network. When we did this research, we also found interesting article from Silence about dark web market, when the bad guys sell user names in password to different network appliances, let's say enterprise-level network appliances, and we found that there are some IP addresses, which we found during our assessment, Internet harvesting in this list. And in our experience, there is no such thing as a luck. We tried to find why such appliances can be so easily hacked. And obviously, the default password, which hardcoded, sometimes never changed, was used on these appliances. We tried to reach vendors and say, guys, maybe it's a bad idea to use, for instance, hardcoded SNMP, not hardcoded by default community, like public and again public for read-write, but they told us that SNMP is off by default. But still simple showdown search show that more than 200 users of this SD1, they enable this SNMP service and still use default password. So, we have a lot of tools which published in our GitHub, and please contribute. There are a lot of things to do with the new fingerprints for SD1 harvester and infiltrator, with SD1 threat landscape description, with new vulnerabilities, and also it's a special CCC release. We start to publish metasploit models for the SD1. So, we have public vulnerability description. You can create own models for it. And conclusions. So, from my perspective, how SD1 development like cycle works, so someone come with brilliant idea, okay, let's build SD1, because Garner told us that it's like brilliant, they have AI in the cloud. So, what we have to do, we can download a bunch of open sources, put off together, set up default routing things, and after all, use it as SD1. So, SD1 is a bunch of open source, which is not bad, but still, you need care about it, and you need to install patches, configure in proper way, and maintain. These complex products have problems with patch management, have a lot of management interface, like machine to machine, and also human to machine interfaces, have a lot of weak defaults like password, hard coded certificate, PSK keys for the IPsec, and many vendors, unfortunately, have issues with the patching responsible disclosure. And this in the cloud. So, if you decide to switch your network to the SD1, hack it before buy, or you will fail. So, thank you for your attention. I want to ask you to give big hand to the SD1 New Hope team, to Dennis, Maxim, Nikolai, Nikita, Alek, and Antoni, who did most of the things here, just like a frontman of this group. Thank you so much, guys. We now have about like 15 minutes for questions. And you know the rules? Please move to the microphone. Any questions? Aside from kind of generic things that any random Linux server can have on web interface on SSH, have you looked at any specific SD1 security problems, like with the encapsulation of tunnels or some stuff like that? So, SD1, you know, there is no like technology like SD1. So, for SDN, there is kind of protocol which more or less reuse it in different vendor, in different solution. And SD1, every vendor implement things in own way. As an example, it's like a Citric management protocol, etc. One thing we did here, but we didn't publish it, is for the virtualization. Because, again, this VNF story, very interesting, because if you have vulnerability in any virtual function, next you can get access to our virtual machine. But again, problem here, but there is no standard and different vendors call VNF, so implement VNF in different way. It can be QVM, it can be just a script which we upload to their appliance. So, you're saying because everyone's writing their own code, there are a lot of bugs to find for people who put the work in? Yeah. Okay. And you can just like, it's not necessary to try to buy things through the eBuy to hack it, you can just go to AWS and activate it for free. Okay, Mike, one please. Mike will be simple, there was a lot of vendors that you were looking, what about Juniper SD1? We're not all yet. Looking forward for controlled investigation. Thank you. Good time. And yes, please. Okay, say thanks for the talk. You mentioned a lot of these virtual machines, like running in hard Linux or something, how do you, what are they, what are they usually on, these kernels, always 2.6 or something like that? Not always 2.6, it was like the worst example. So, some of us like, you know, newer. So, it's not necessary. Again, for network function virtualization, some vendors, they call VNF, just a bunch of script which they download and change configuration of default sensor. And in this case, we can use very old kernel. Other way, more, use more recent version. Yeah, that's okay. I mean, the hardening, how is it telling you, the hardening of the Linux kernel, is it like, good stuff? In most cases, no way, so no hardware or things like this, no JR security or Linux, nothing. All right, your question. Have you looked into Cisco Moraki? No, we, for Cisco, sorry, it's night. Let me find the list. For Cisco, we did our exercise with the VipTeller. All right, last question over there, please. Just advice, maybe you drop a slide in with the 90s code, they want the hard coding password back. Sorry? Because there were so many hard coding passwords everywhere, maybe you should just drop a slide in the 90s code, they want the hard coding password or lock it back. I don't know. So it's public things. So was that a question? It's just recommendation. All right, we have a few minutes left, but if there are no questions left, then I would call it a day. So another warm round of applause for Senator Gamble.