 guys. Hello and welcome. Super glad to be here. This is my first time speaking at DEF CON. Very humble to see so many people. Thank you guys for coming. So I heard there might be some problems with the projector so I do apologize about that in advance. So without further ado let's jump into some practical Cisco switch exploitation. So I'm a penetration tester at Kaspersky lab. Also skydiver. So if any of you guys skydive raise your hand. No one? Yeah, buddy. So long story short I'll be talking about critical vulnerability published by Cisco systems on March 26 and it announced that around 300 different models of Cisco switches were vulnerable to an unauthenticated code execution vulnerability. And the advisor also said that there are no signs of exploitation in the wild. There's no public exploit available. So there was one quiet evening just writing one of those pentas reports when I decided to take a break. Go to my Twitter feed and all of a sudden I stumbled upon a tweet about the critical vulnerability about the Cisco advisory. So here's what I saw. Red, critical. So we got a CVS score of 9.8 and in a nutshell this was about a remote code execution via telnet service in Cisco switches and that happened because of two major problems. There was a failure about clustering. Cisco switches might be joining clusters and surprisingly enough they are using telnet for communication between cluster members. And that's not just to play in telnet. They're using some of their specific options, cluster management protocol options. And also these options are not correctly parsed on the server side. So this according to the advisor, this leads to remote code execution. So the vendor advice is disable telnet. Which is a solid advice actually because you know that telnet is an old legacy protocol. It's been superseded by SSH. Okay. Do you hear me now? Okay. That's better. So telnet is a legacy protocol which has been superseded by SSH like 20 years ago. And now according to the advisory you can now get access to a Cisco switch through it. Is it really serious? No public exploit. No knowledge of exploitation. So guys, challenge accepted. That's why security research is like here. So I'll walk you through the process of creating a working exploit for these switches. And to start off, the first thing I did is research. The information is available in the public about this vulnerability. And basically there are two things up for grabs. One is the Cisco advisory I was just talking about. And the other one is the original source of information, the vault seven leak that happened in March of 2017. And that is actually the source for Cisco systems themselves. That's where they got information about the vulnerability. So this leak affected many vendors like Microsoft, Apple and Cisco. I decided to sweep through the pages of this leak myself to find something. I honestly was hoping to find some kind of exploit there or at least a step by step explanation of what's going on. But no, it was even a challenge to find the pages themselves. Because everything is codenamed. And the codename for this exploit is ROSEM. I don't know what it means. And all I got were the testing notes for it. So I got a bit of information that I got from the vault seven is that it works in three ways. It has three ways of operating. It's set, unset and interactive mode. Which brings us to the set mode. So they run a script that basically turns off the authentication on Telnet connection. And after that, all the subsequent connections to Telnet are credential less. So you just pop out level 15 Cisco iOS shell. The unset is completely opposite. You just get the authentication page back and all subsequent connections require passwords. And the most interesting one is interactive mode. So you run the exploit, you immediately get a shell. No subsequent connections are affected. So this is my guess from the vault seven pages. I don't know how it exactly worked, but I assume they work this way. So I decided why not do the same? I had two Cisco switches available. I just needed to cluster them up. Dump some traffic in cluster, configuration and I hope to get a working exploit. So this is how I felt. At the beginning, but you know that the life is harsh and I look like this actually. So guys, a few words about clustering those switches. So this is a way to actually a centralized way to administer several Cisco switches. You have a master switch and you have several slave switches. And you just log on to a master switch and you are able to pop up a shell from any of the slave switch from the shell of the master switch. So this is kind of what it looks like. If you log into the master switch with a level 15 privilege, you will get a level 15 privilege shell on a slave switch. And the switches are doing this by discovering themselves on the L2 network. They're broadcasting CDP packets and they're building up tables of possible candidates for clustering. So you add a master and slave switch and then you are able to run the R command command which pops the shell. So I was running Wireshark during running this command and testing all this. And the advisory, you remember, said that there's a telnet going between this master and slave switch. So I look up the Wireshark. What do I see? Where's telnet? No telnet. Some fancy unknown L2 protocol. I wasn't expecting that. Telnet? No telnet. So I was really unhappy and almost closed the Wireshark window. But then I decided to tie some more commands into the shell and I actually observed that there is telnet in the traffic. And what's funny is this L2 protocol actually encapsulates IP protocol where the source and destination are not IP addresses but chopped MAC addresses of the Cisco switches. And of course in IP packets there are TCP packets and port 21 and we got telnet inside. Okay. Okay then I said. Next thing, I was looking at the traffic for some anomalous traffic of telnet but everything was the same as with the generic telnet connection except for the initial handshake. You know, telnet before you are presented with a credential prompt and the shell, the server and client are negotiating the options. And what's interesting there is that I found an interesting string flowing between the server and client and it says Cisco kits. So we have to remember this string as this proved to be an important string. Back to the Vault 7. One of the important bits of information that I got there, there was notes of some engineering testing the exploit. Of course, no exploit was available at Vault 7. But this was kind of an error log for telnet debug from Cisco and it says that this is some kind of anomalous output and we see the same string that I observed in the Wireshark dump, Cisco kits. Since this is track 101, here's a telnet command in the options 101. This Cisco kit string was transferred as a part of a sub negotiation option of telnet. It can be used to transfer custom parameters before the telnet session. Well, I decided to go the easy way and I was sure I will get a shell easily and I just replayed the traffic into a generic telnet connection. But no, it doesn't work. I'm still presenting with a credential check and if you look thoroughly through the Cisco advisory page, you can see that they actually have an IPS rule for this vulnerability and it's called Cisco iOS buffer overflow. What does this sound like guys? It's some reverse engineering. All right. So if you want a reverse engineering, you might want to get the firmware out of the switch. Easy. You just log into the switch, you list the contents of a flash directory. There's always a firmware there. Cisco switch has a copy command which is able to copy the firmware to an arbitrary FTP server so getting the firmware is pretty easy. What does the researcher do with the firmware? Of course, Beanwalk. No problems here. Uncompressed into a 30 megabyte binary. Well, things are going good, I thought. But no. Every reverse engineer knows that when you open up IDEPRO and it shows this tiny thing column, you're doing something wrong. I did not recognize it. Of course it didn't. But jokes aside, this firmware is PowerPC 30 bit. It's easily, you can look it up in Google that Cisco switches are buying PowerPC. It's no big secret. And the entry point, the actual address of the firmware to be loaded is 300 in hex. And you can see that during the switch startup. Since I didn't see any code whatsoever when I fired up my IDEPRO, there's actually a very nice script by a guy called Federico. And he created a script for IDEPRO that automatically recognizes all the function. And it's actually cross-platform. So if you have one or two functions already recognized manually, it just sweeps through all the code and recognizes the other one with the same prolog. So it looks something like that. You just specify the most common prologs for a function. We'll just loop and scour all the function. So this helped me to discover like 90 or 97% of all the function which was enough. Well, of course, firmware, there are no symbols whatsoever. And Cisco iOS operating system is basically a single binary. And even when you're looking through the code, it's kind of difficult. Kind of difficult to follow because there are lots of places where the functions are called indirectly. The call tables are filled at runtime and it doesn't make it easier to follow the control flow in static. So of course, since we have problems with static analysis, we have to set up a debug environment. And there are some problems with that. First of all, of course, there's no development kit for Cisco. Although the switch itself has a GDB kernel command which is a actual GDB server with a slightly customized protocol. So the problem is that new GDB client versions do not support this. There is a manual on the internet how to build an old version of GDB for it to work with a Cisco router or switch. The other option is to use a debugger created by NCC group. It's called IID. It's pretty good. And there's some people who actually built an IID adapter for the debug IOS but I didn't find it in public. So I started up with GDB. I opened the manual and patched it. So what did I get? Well, almost nothing. I was able to read and write memory on the switch. But that's all. I wasn't able to run the firmware. I wasn't able to put a break point anywhere. So, guys, I resorted to the last possible option. I did. It was kind of a smooth experience, you know. This debugger, I think it was built for some very narrow models of routers. It had some user line code that requested information from the router like process list and router configuration and the switches output different format. So the debugger was crushing all the time and I had to really debug it and cut out lots of user line functionality to get it working. But nonetheless, it did work for debugging the kernel of IOS. Not too long ago, not too long ago, like a couple of weeks ago, I stumbled upon this peak comparing Windows debugger and AliDB debugger. Well, this how I felt when I was debugging using IID. But, you know, head soft to NCC group, this is still the best tool available to debug IOS. So now to the actual meat of presentation, the actual vulnerability. That's Cisco systems encounter and I started with looking for strings in the firmware. They were all recognized. It was easy to get them. And remember that string in the dump and also in the failed output in the volt 7 leak? This is it. I started with it. So I got some cross references to the string and it led me to one particular interesting function which is I called it called Cisco kids. And it was actually doing what it was doing. It was parsing this custom magic string and there was a vulnerability in it. And because the telnet code is rather symmetrical, the code for parsing, for sending the string and the code for parsing the string was really near and helped a lot. So this might be a little bit difficult to process but this is the client portion that sends this magic telnet option and we can see that it is managed by a format string which is some byte, then a string, then a byte, then an integer. Then we get another string between the two columns and we get an integer. Matches perfectly the string that I was looking at in the wire shark. So let's look at the code that actually receives this magic option and parses it. I stumbled upon this code. So it basically reads the first byte, reads the integer, reads the Cisco kid string and then it reads the string that is between the columns. And funny enough there are no boundary check, no length check whatsoever and it's just read until the next column. Guys, what does this look like? To me it sounded like a classic overflow. So, yeah, I had an overflow. It's an absolutely classic one. Instead of sending an empty string between the two columns, I wrote some capital A's because capital A's mean business. And we can see at the bottom of the slide that the power counter, program counter is actually like an EIP for input platforms. It's instruction parameters. We can see the capital A's in there. So we now are in control of program counter. This is how our flow looked in the debugger. We see that stack is overflowing with the capital A's. And we see that the current instruction is BLR, which is a power PC for branch to link register. And what it does, it checks the link register, which as you can see is 41, 41, 41, 41. And branches to it. So it jumps right to the data we had sent to the switch. The stack. So the power PC stack frame is kind of the same as Intel in a way. We have local variables up above the return address. So if one of the local variables is a buffer and boundaries check, we of course have a problem. Execution will be controlled by user supply and input. So now what do we do after we send some capital A's? Of course we generate some cyclic pattern. Cyclic pattern is a way to determine the exact location where we actually overflow the program counter. It works in a way that every four bytes in a buffer are unique. So when you overflow with cyclic pattern, you just read the contents of program counter. You fit it up to a special method that recognizes this tiny four byte portions of buffer. And this method gets you the offset of the buffer that overflows program counter. I use spawn tools for this. It's a great framework for pawning stuff and also has the ability to generate cyclic pattern. So it works kind of like this. You just generate a string for about 200 symbols. You create a payload with this magic tunnel option you send it. And again when the switch crashed, I observed the value of d8 ad in ASCII which is you see how it is in hex. You just fit this up to the cyclic find method. And we get that the actual program counter is overflowing at offset 115. All right. And guys, this really looked too easy because actually R9, so the PowerPC has 31 general purpose registers and the R9 registers at the time of overflow was pointing at our buffer. So it looked too easy. So we just have to put our shell code at address pointing by R9. And no batches were in place whatsoever. Okay then. Here's a screenshot from IID. So you see the R9 register being pointing to our buffer. So we have to find a gadget in memory that just jumps to R9. And it is here in the screen. So it's kind of easy. You just load the R9 register to a special CTR register. The second instruction is garbage. We don't need it. And then we branch to CTR registers. So this way we just jump to R9. What could possibly go wrong? Jumping the shell code is always fun. But no. Access exception again. So I failed again. The device actually rebooted. I don't know why. I didn't know why. So it seemed to me that heap and the stack were not executable. Why did that happen? Is this actually the data execution prevention? Honestly guys, I still don't know. But there's been talks on this matter before. And one of them was really good at Black Hat, not Solongo by Felix FX. And he actually suggested that this might happen because of instruction and data caching in PowerPC. Here's a slide from his presentation. PowerPC has a cache for data and instructions. So what happens when we write some data to stack or heap actually and then transfer execution control to this data? When we transfer the execution control data to this data, there's actually no data there because the data we just wrote is in the cache. So this might be the reason why it's not working. So in the end, we are not able to execute code. So what do we do? Our last resort is return oriented programming of course. What is it? It's a generic technique used in exploitation to bypass data execution prevention. In our case, it's a way to bypass instruction and data caching. And it works this way. We don't write our own code to stack or heap. We are reusing the code. And in the binary or in my case, in firmware, we use the stack as a source of data for those instructions. And we chain this little snippets of code in such a way so we can perform our needed actions. And those might be either let's say arbitrary reads, arbitrary writes. And then when we're done, we are transferring the execution control flow back to the original function. So the each gadget, each candidate gadget has at least two conditions. So it must execute some payload, either read or write. And also it has to have some code that is transferring execution flow to the next gadget. And there's some limitation to this approach. Only, of course, you are as good as the number of gadgets that are in firmware. There's a limited set of them. And when you're actually chaining them up, every gadget is modifying stack in a way. Because mostly we use function, epilogue as gadgets, and they all work with stack. So when they move stack around and this might corrupt stack frame that is below your stack frame, and this is tricky and creates problems. We can accomplish basically two things with return oriented. Either arbitrary writes and this might lead to arbitrary code execution. So my idea was to create an arbitrary memory write primitive. The idea is quite simple. We have a find, we find a code snippet in the firmware that takes value from the, two values from the stack. One value is the memory address we want to write to. And the other value is the actual value we're writing. So for example, we load the memory address at register R3 and the value to register R31. And then we chain to the next gadget, which actually performs the write. So, I've been talking about it already. So typical gadget looks like this. It's a function epilogue. It does stuff with data on stack and then it jumps. So the LWZ op codes are for loading data and the last op code, BLR, is actually to jump out of the function. And the last op code, BLR, is the same. So I'm going to be talking about the write primitives that I used to actually get this exploit working. This is the actual gadget I'm using in the original proof of concept I released not too long ago. So first and foremost, with the gadget we have to set up our, the address of the next gadget. So we do this by loading data from stack to register R0 and then loading the R0 register into the link register. And you remember I told you that link register is used for branching. Next thing we do, we load the address we want to write to from the stack. And stack is pointed by register R1 and power PC. We load this into R30. We load the actual value to R31. Side effect from a gadget like this, it moves the stack a bit like 10 and hacks and then we branch to the value we put in R0. That's it. We have two registers with an address and a value to write. And we chain to the second gadget which uses op code store value. And it writes the value of R31 to R30. And this is our basic write primitive. So it consists of two gadgets, the loading and the writing. And the result is we just wrote arbitrary data to arbitrary address. It is easier to do this with automatic tools. This wrapper tool is really good. It kind of semi-automates the process of building gadgets. You can look up, you can load a binary of firmware in it and look for gadgets with search and wildcards. Okay. So we have basically have the ability to write arbitrary data into the firmware. How do we get code execution from it? The plan is to find some place, some critical place in the firmware which we can patch and that will result in code execution. So you can patch the control flow that actually checks the credentials. You can patch some telnet inner structs that are used for authentication or you might look for some function pointers that return critical data related to authenticating you on the switch. Well, I thought why not patch the execution flow? So looking further down the code where the switch is authenticating you, I just decided to patch this if branch. Well, instead of if and all of those conditions, I just put it like if true. And guys, it looked like it worked. But, you know, again, this only worked under debugger. The exception was triggered during the live setup and, you know, I was kind of desperate at that moment. I kept on looking and a couple of, I would say days, you know, I was looking on this code and something, you know, catched. And a lot of code on this slide. And what's interesting is the first line which is indirect call for function that I named is cluster mode. And it is indirect which means that it's not called directly, it's being referenced by memory location and this memory location has the actual address of the function. So the second interesting thing is that there was another function down the code that was called indirectly and it's called getPrivilegeLevel. And also, it was referenced by memory location. And so long story short, I just wanted to tell you both of these functions are referenced indirectly and we can apply write primitives to them to change this memory location. But the question is why are these functions important? And they are important because if the function is cluster mode, returns something that is not zero, then we go down the branch which checks your privilege level and that is the only check you need to get a privilege 15 shell in iOS. So if we patch is cluster mode with nonzero and then we patch the pointer to getPrivilegeLevel with the function that always returns 15, we can see here as a power PC snippet that if our privilege level is not minus one, we're just presented with a shell. Okay. So, easy enough. We will actually take this pointer, pop up IDEPRO, look for a function that returns something that is not zero, one, 15, doesn't matter. We will take the getPrivilegeLevel pointer overwritten with the function that always returns 15. The firmware actually had those functions, so it was pretty easy to find. And all we have to do now is find suitable gadgets to make the necessary writes. So the first gadget, I was already talking about it, it's just loading up information, I mean data from stack. We take the pointer to his cluster mode function. We take the value we want to all write it with and that is the address of a function that always returns one. We add to the stack pointer and we make a branch to the next gadget. The address for this gadget we already set up during the first two upcodes as you see. So, loading the R0 register and loading R0 to link register. Second gadget does the right. So, the first upcode gets to registers, gets R30 and gets R31 and writes to the first upcodes to location reference by R30. Okay, so now we have overwritten the is cluster mode. It always returns a non-zero value, so we go down the branch that checks the privilege. It's kind of more tricky with the second write because it actually needs to do references. But guys, this did work. And finally, finally, I was presented with the level 15 iOS shell on the switch. You know that it's demo time. I hope the, I'll check if the internet works. If not, I'll just show you the video. That's actually SSH connection to my lab, so it's not too fast and I can really see what I'm doing by a try. So, yeah, we have a Cisco switch on this IP address and we can actually see that there's a password problem, right? We type in some basic stuff like Cisco, Cisco, it doesn't work. So, to the exploit itself, you can see what I'm typing. It's good. So, the options are pretty easy. So, you just specify the host and you set. Not sure if it worked because there are some problems with timings, but I will check. All right. Is there a shell? Yes, there it is. So, you can actually see the version of firmware. You can also see that we have a show proof, right? Yeah, we have a privilege 15 shell. So, what's good about this exploit is that we can easily unset this. So, instead of set, we just go with the two dashes, I don't see. Okay. We go with unset and yes, it goes back to its original state. Right, back to there. Well, it's good. Demo gods were happy with me. So, yeah, that's the video. We'll need it. So, side notes. I do pentest a lot. So, I've encountered many of those switches on pentest and successfully exploited several of them. And what helps with this, this exploit is actually very firmware specific because offsets are different from firmware to firmware. And what helps to determine the version is two protocols, SNMP and CDP. SNMP, if you get a public or a private string, you're able to dump the firmware version and that helps you to develop to customize the exploit. Or if you're in the same physical broadcast network as the switch, you can listen for CDP request. They also give information about this version of the switch. So, this helps a lot. So, in a matter of like one or two hours, you can find, you can customize your exploit and get a shell on the switch. So, about further research, of course, I managed to do only arbitrary writes on this switch. It would be good to be able to run an arbitrary shell code instead of just modifying memory. And also it would be really good to actually automate the process of building ROP gadgets because it's kind of a tedious process. And, you know, first time it's fun, but not so much when you do it like a tense time. And again, I just thought about not like a week ago, what if we know that switches are using clustering via the CDP protocol? And I'm almost sure, but not 100% sure that there's no authentication in place when they're sending the still net clustering between them. And we also know that the master switch can easily get the privilege 15 shell on the slave switch. But what if we are in the same broadcast segment and we just build a CDP packet that tells the other switch that we are a switch, and then we craft an layer two telnet connection requesting a privilege 15 shell with argument. What will happen? Well, this actually reminds to be, remains to be seen. I'm working on it. It's an ongoing research and, you know, stuff to think for you guys. Thank you so much for your time. You can check the proof of concept code on GitHub. Contributions, of course, are welcome. Thank you so much.