 So our next talk is called Hacking ISPs with .2 PON protocol over Ethernet, PPPOE, we have Gal. It's the second time at DevCon. I guess we didn't scare him the first time. But please give him a big cheer. And let's start. All right. Thank you. All right. So hi, everybody. And welcome to my talk on hacking ISP with PPP or E, or as the title says, .2 PON protocol over Ethernet. And first I'd like to thank DevCon for having me here again this year and this time in person. So I'm excited. And before I begin, let me share a quick story with you. So on this research, I've been working on and off for the last two years. And as of course, it was during the pandemic. Like everybody else, I had to work from home. However, this research involved this device on my kitchen table for almost a year. And let me give you just a quick example how it sounds like. I'm also going to pray the audio gods that it's going to work. No audio. Wait. So I got a fallback for that. And I'm just going to play it for my phone. So this is how the device sounds like. Yeah. So yeah. So before I begin, I would especially like to thank my wife or Tal that sits here with us today for sharing a single bedroom apartment with me and that device. Yeah. Thank you, honey. Yeah. Yeah. Tough year, the pandemic. All right. Okay. So enough about that. So, okay. So let's begin. My name is Gal. And I've been causing rockers on embedded device for quite some time. And as I said, this is two years long research. And I actually started working at it when I used to work at Aleph Research. And I ended it in Cyber Arc Labs where I work today as a research manager. And today we're going to talk about three subjects. First, I'll introduce the idea and motivation behind hacking Internet service providers, ISPs. After that, we'll do a crash course in L2 communication protocols. And I present some cool vulnerabilities I found. And lastly, I'll share how I was able to research this kind of equipment. And hopefully, I might convince you that at the end of the day, big ISP network equipment, it's not that different from small home router research. Okay. So let's begin with pawning Internet service providers. So we all know how a classic remote attack is executed. Usually, attacker looks for a victim on the Internet, maps the attack surface by discovering what protocol the target uses, and then try to use Oday or Nday and hopefully get RC on the target. Ironically, many of these classic attacks are nowadays taking place on home routers. So the idea is that ISP network equipment, it's not that different. Instead of the Internet, we got our ISP network accessible from our local modem. And instead of the Internet server, we got the ISP equipment that provides us services. So all we have to do is to map the attack surface, find an Oday, and hopefully get an RC on the ISP network equipment. But why attacking ISP? Well, if I was the proud pwner of Evilcom LTD, I might be able to do some of the following stuff. For start, the obvious denial of service and other ransom activities like shutting down the network and asking for money to stop doing it. But I can also execute DNS hijack for the entire network, aka become man in the middle for all subscribers by redirect their DNS to my own evil DNS server. And another interesting idea is to target a specific subscriber by their actual identity, a.k.a. the actual name and other information the ISP stores on them. I also might be able to, might be connected to the Internet backbone for IP allocation, so I might be able to execute this crazy scale distributed denial of service. And lastly, I can execute all sorts of attacks by abusing common protocol and technologies used by the ISP. Okay. So to understand what we are about to attack, let's understand how basic ISP network operates. Keep in mind that this is a simplified example for DSL-based network, but other tunneling protocols such as PPPOE, L2TP, G-Pone, they all follow more or less the same concept. So when you get your router from your ISP, it has two roles. One is to provide network services such as Ethernet and Wi-Fi to your home premise, and the other role is DSL modem functionality. The DSL modem basically uses the good old telephone line to connect you to your ISP and then to the Internet. Your modem is hooked up to a phone line, and the telephone company connects the other end of the line to something called DSLOM. DSLOM is basically a multiplexer that extracts digital information from analog signals by sends from multiple modems over the telephone lines. Okay. The DSLOM is then connected to a broadband remote access server or BRUS, which is basically a big router that routes traffic to and from the DSLOM to the ISP network. So we can abstractly think that our modem is connected with a layer 2 cable directly into the BRUS itself. And we can also bridge the modem traffic to our own evil machine. So now we can think abstractly that our malicious machine is connected with a very long Ethernet cable directly to the BRUS. So if the BRUS is the ISP's front-line router and we are connected to it with a network connectivity, then hacking this BRUS is the equivalent of hacking the ISP's home router. All right. And this is where home router takes the power back. They can attack the BRUS in the same way hackers can attack them from the Internet. Once we control the BRUS, attackers can execute some of the attacks I mentioned before or attack other ISP network equipment. All right. So for this research, I decided to target Redbeck network smart edge equipment. Back in the 2000s, Redbeck were a big player in ISP equipment. They were actually involved in defining some of the protocols I'll show you today. In 2007, they were required by Ericsson that continues to manufacture new products under the Redbeck brand. Some of these devices are pretty big and can support up to half a million subscribers. For example, the one that you see at the bottom right. And they all use custom NetBSD operation system called S E O S, smart edge O S. And they also use a PowerPitch C architecture. Okay. So now that we have a target and we see it's worth hacking to it, let's understand what kind of protocol can be used to attack ISP's. All right. So let's start with point-to-point protocol over Ethernet or PPPOE, which is a layer two encapsulation protocol. This is a common protocol to connect to ISPs. And in our case, the PPPOE server is the ISP's brass. And the PPP client is our DSL modem or malicious machine. And remember that we just saw that abstractly the client is connected with an Ethernet cable to the brass. So as the name of the protocol suggests, by using PPPOE, we can encapsulate PPP sessions over Ethernet frame. Okay, great. But what is PPP? All right. So PPP stands for point-to-point protocol. It is also a layer two protocol, which is mainly used to tunnel ISP packets to and from the modem and by doing so enabling the Internet connectivity. So PPP negotiation is where the client authenticate to the server and get its configuration parameters. And if all goes smoothly, a PPP tunneling interface is created on the client's side. At this point, the client receives DNS configuration and usually an Internet-facing IP address. So at the end of negotiation, our router or machine should end up with an interface similar to this one. All right. Also at any time, both the client or the server can terminate the session. Yeah, and that's it. Now that we understand the general flow of the protocol, I'd like to focus on PPP and especially on the session negotiation part of PPP. So PPP is a layered protocol that has three components. First, there's an encapsulation component that is used to transmit datagrams over a specific physical layer. I'll soon go over on the specific format and how it looks like. The encapsulation is used to transmit something called link control protocol, LCP. This protocol establish, configure and test link as well as negotiate settings, options, and use of features. And lastly, after the connection is established, different protocol can be used to negotiate and facilitate a layer three network layer. In our case, to tunnel IPv4. And we use for that the IP control protocol or IPCP. Since PPP is a relatively big and have many protocol layers, I decided to focus on LCP and this is why I did that. Well, this is the first protocol used in PPP. It used to set and receive different configurations. And no authentication is needed. Okay. So to really understand PPP, let's understand the encapsulation format. So first we got an Ethernet frame with a PPPOE payload. The PPP encapsulate a PPP payload. And in this example, the encapsulated PPP packet is LCP type of packet. Now let's understand how LCP packet looks like. LCP packet contains an option payload to pass different parameters. For example, authentication protocol, magic number, maximum receive units, et cetera. And each option has a code number, length, and data. For example, here we see that LCP packet that contains parameter for maximum receive units, authentication protocol, and a magic number. And yeah, this is a different option values for all these fields. All right. And there are all sorts of other parameters defined in the protocol. But the one I found most interesting was actually the last one, code number 19, endpoint discrimination option. I don't have enough time to get in what this option does. And actually soon you'll see it doesn't really matter. But yeah. But I discovered that although the RFC defines that the parameter length should not exceed 20, in the smart edge devices, it actually handles packet with bigger length. And it also has no validation on the data field. Also, I noticed that when the PPP log is enabled in the smart edge device, I get long entries similar to these. Here we see the different LCP parameters written to the log. For example, this is the authentication protocol used. And this is the magic number. And this is the AA as part of the endpoint discrimination field. So I'm writing a log entry with any data, I wish, and with a bigger length than accepted. Yeah, I think you all know what's coming next. Yeah. Vanilla stack overflow in the smart edge log entry. And this is a great time for demo time. Okay. So for this demo, I will be using four terminals. The bottom blue terminal is only a monitor terminal connected to the smart edge device. Yeah, this is the monitor. And the red and the green terminals on the left will act as the STD in and STD out listeners for the reverse shell. And on the black terminal, I will execute my attack connected as a DSL subscriber. Okay, so now I'm running NETCAT to listen to STD in and STD out on two ports. Yeah, here I'm using the first one. And the second. All right. And now I will execute my attack by piping being a sage to two telnet clients. So here I'm piping the STD in to port 1337 and then pipe being a sage and STD out to the other ports. Okay. Fire away. So now I'm sending the first two PPPOE session, creating a PPPOE session by sending two packets. And this is the LCP payload, malicious payload. And you can see on the monitor screen that I got a core dump, meaning that I managed to smash the stack. And now I'm using my STD in NETCAT to send commands. Here I'm using LS. And of course, I'm going to echo my user and see that I am root. And yeah, let's do it again just to thank you. Thank you. I can also grab some other configuration like the DNS and I can write and read the DNS configuration and the admin for the device itself. But yeah, once once I got root, I pretty much it's really much ends. Okay, so not now that we've seen a full working RCE, let's talk about how to research this kind of equipment. So the most interesting conclusion I got from this research is that ISP network equipment is not that different from home routers when it comes to vulnerability research. Let's talk about the differences. So home routers are of course cheaper, usually around 300 bucks, while brass entry level is a bit higher. Both of them usually never gets updates from the ISP, which is great for hackers, but really bad for everybody else. And as you already heard, brass are way noisier. So if you plan to store them at on your kitchen table, you should expect to annoy people around you. And probably the hardest part is that set up and configuration part since ISP equipment usually takes a specific expertise to install. So if you got enough, so if you got some extra dollars and you're stubborn enough, you could also pawn an ISP network equipment. So based on my experience with this research, I'm thrilled to present my seven easy steps to research and pawn ISP equipment. First, firmware emulation, then setting up debug and development environment, jailbreak if needed, get or buy an actual device, search and hopefully find vulnerabilities, write an exploit, and finally celebrate with your favorite beverage. Okay, so first step for every embedded device research is usually getting the firmware. I was lucky enough to find one online. And after a quick beanwalk, I realized Smart Edge uses a net BSD OS and a power PC architecture. Luckily, up until 2006, Apple were using power PC processes with their BSD based kernel. And this nice fella called Kearny posted a wonderful Reddit blog on how to emulate a very similar system to the one Smart Edge are using. And that way I was able to emulate the user space of the Smart Edge firmware. Right now for debug and development. I realized that cross compilation to a different OS and a different architecture is pretty much a nightmare. So I decided to use my QMU machine to just compile staticly tools. I also use this SSH TCP dump trick to sniff packets from my emulated device into my wire shark host. Well, that was super useful to understand the different protocols I'm presenting. But I also had an issue with debugging. My GDB multiarch on my Ubuntu machine refuses to connect to a power PC net BSD system. So instead of spending time solving this issue, I just decided to run the GDB client from my emulated environment. And yeah, this actually was very useful later on when I remotely debug an actual device. So it just saved me some time. Okay, so next step was to understand if a jailbreak is needed for debugging an actual device. So as you can expect, the Smart Edge are using this exec underscore CLI binary to handle all console commands. So looking from telnet SSH serial ports, they all end up with this jail terminal. But exec CLI must run some code of some kind of other binaries. For example, when you use the telnet command in that shell, it executes this SE underscore telnet binary. By the way, this is the same telnet client I was using for my exploitation earlier at the POC. Luckily, I found that this telnet client has an internal telnet command for just popping up a jail free sub shell. So yeah, so another thing that please note that this is a telnet client. So to run the telnet internal command to pop the jail free shell, I use this trick where I do telnet to my local host. And here you can see the invoke sub shell command. So all I have to do is just use the exclamation point command and pop myself a jail free CLI. Now that I know that if I'll get my hands on an actual device, I can execute any code I desire. For example, this gdb server that Redbeck was kind enough to live in their firmware. Right. So next, I decided to buy an actual device. Theoretically, I could continue my research on emulated device and maybe even find vulnerabilities. But I was missing the actual device configuration. I also had to do some serious LD pre-do Voodoo magic to get the emulation working, which made it limited to specific binaries. And it was pretty unstable in general. And also, if I have an actual device, I could develop a full working exploit like the one I demonstrated. Right. So I went to eBay and bought the cheapest device I could find, which is the smart edge 100. It cost around two grand. And a lot of research was kind enough to fund this purchase. By the way, at that point, I left a lot of research and moved to cyber labs. Yeah. So I finally got an actual device. But I soon realized that I got little knowledge in setting up ISP equipment. And I had to acquire some kind of ISP technician skills. So I started by reading this 100 page of basic how to work guide to understand how to physically connect stuff. I then had to read this 360 page of basic configuration to understand what configuration I need to do in order to create an ISP alike setup. And lastly, to work with the CLI, I had to use this 900 page manual. And this is how I pretty much felt from all this useless information that I'm going to use only once. Yeah. Okay. So to apply all this information, let's first go over the device Sashi. Most of the port on the device are either Ethernet or optical ports. To these ports, the ISP connect other network equipment, such as switches, routers and the Islam. And most importantly, the subscribers are connected through these ports. So my attack should be executed from one of these ports. The other ports are used for managing the device. So we got through Ethernet management ports, and a single serial port. So there's two ways to configure and manage the device. The most convenient is of course, the Ethernet ports by using telnet or SSH. But these ports can also be used for other monitoring such as log monitoring, packet monitoring, and even remote debugging with GDB. But the serial port is still very useful since it's foolproof. Meaning that even if I completely messed up the device configuration, I could always reverse it with the serial port. And believe me, I have messed the device configuration like a million time. All right. So finally, I was able to configure the device with around 200 configuration commands. And I was ready for a full ISP-alike installation. My setup included an Ubuntu machine, three network cards, and a single port, a single serial port. So I started by connecting a serial port to the foolproof console. I then used one network device to connect to the management device port. And then connect another network interface to simulate an actual subscriber connected to the bus. This is where I executed my vulnerability through this port. And lastly, I installed a device in our server room and connected my Ubuntu machine to the internet so I won't have to work for my kitchen table no more. Okay. So now that I own a smart edge device, and I understand how to configure it, and I got myself a server rack, I finally became the proud owner of Evil Communication LTD. So you guys are more than welcome to become my victim's ad to become my clients, of course, free of charge. Right. But beside becoming an ISP, I finally reached the research phase. So now I had three useful tools to help me with the research. The smart edge log, a GDB server, and a wire shark monitoring. And now let's talk about the demon itself that handles PPPD. Smart Edge is running a demon called PPPD that handles PPP sessions. With some reversing, I realized that incoming PPP packets are being handled by a function called packet receive. Packet receive calls a function called Demux input packet. For every packet it receives. And this function is a log stream multiplexer for input function. Basically, it means it uses function pointers. For different log depends on what sub PPP protocol is being used. Here we see three logs protocol functions. For example, the LCP, the IPCP and LCP. And as you guys already know, I was interested in the LCP log. And this is where I discovered that the endpoint discriminator log uses this unsafe mem copy to the stack. So if we go back to the LCP options payload example, this is where my mem copy copies the data I control with invalid length to 35 bytes of stack array. Right. And finally, it's exploit time. So exploit development was very straightforward. There were zero stack mitigation. And I chose to use a single rope gadget to exec VE. But when I was using the reverse IP shell in my demo, I was making a naive assumption. In the demo, I assumed that the brass has some kind of IP connectivity and was able to create a reverse shell to the remote server. When the attacker can control the brass by and then the attacker can control the brass by connecting to that remote server. But what if the brass has no IP connectivity, or it blocks by a firewall? How can I get a shell on the device using only layer two? So I decided to try and develop a layer two only shell, meaning the entire shell will be executed over Ethernet frame without the need for IP protocol, meaning the subscriber can control the device directly. So the first problem was that my exploit was limited to command shells only, like we saw on the POC. So first I had to execute my exploit to deploy a bridge head. Bridge head is a term for running a small exploit to update a bigger exploit. So if I was looking for a way, sorry, I was looking for a way to execute a shell code that will allow me to upload files using only layer two communication. Luckily, I had TCP dump installed on the device. And I found this very cool trick to upload a file from a specific MAC address. Here we see that the TCP writes the content of 106 frames to a file called L2 shell. And I also used a filter to only pick up frames with a specific magic MAC source address. So now I am no longer limited to a shell command. I can upload any binary I want and execute it by running my exploit again. But since I'm dealing with layer two connection, I needed some sort of raw socket functionality. Unfortunately, Smart Edge implemented their own undocumented raw socket, so I couldn't just use it easily. But fortunately, Slash Dev Slash BPF was installed. And ironically, the only binary using Slash Dev Slash BPF was the TCP dump I just abused. So I had to configure BBF with these two flags. The first is telling BPF which interface to listen. And the other flag just received the packets immediately. And now BPF listened to alert to traffic. And if it detects a frame with a specific magic, it extract and run its codes, the specific commands. And now I truly done with exploiting the device. But wait, there's more. I would like to share another interesting vulnerability discovered by Omar Sanfati, one of our researchers at CyberArch Labs. For this, let me explain how PPPoE session works. So every PPPoE packet has a code field. This field sets what type of message is being used in the PPPoE negotiation part. So the first phrase is not mandatory, and it's mainly needed for the client to discover the PPPoE server MAC address. It does that by sending an initiation packet and receive an offer from the server. And the next step is mandatory. And in this phase, the client requests the session number to start a session. So the client sends PADAR, aka request, and receive PADAS, aka session from the server. Also, it's very important to notice that both sides can terminate the session at any time by using pati, aka termination. So Omar discovered that when the client sends PADAR request with a broadcast as a source address, the smart edge PPPoE server sends a session packet with a broadcast as destination address. Since no client expects an FFFF as a destination address, it sends it again and again. And if the server is configured with session timeout, it sends a termination packet. But this termination packet has a broadcast as destination MAC address, meaning the server asks all the clients to terminate their session. Thank you. There is a bot here. There is a bot here that this attack, it still depends on the switch policy and whether the ADSL endpoints terminate the session and if they receive such kind of a request, I mean a broadcast termination with no session number. But anyway, it's a bug for sure and a really cool idea for an attack. Now that we managed to pawn our smart edge devices, all it left is to drink rum with the fellows here at DEF CON. By the way, this is my noop shot from 2020. And as for fingerprinting, so I managed to detect around 500 devices in 55 different organizations from 20 different countries that were using the smart edge devices. But remember, folks, these devices are not publicly, they're not supposed to be publicly facing. So I'm sure there are many more out there. And I reached out to Ericsson with my finding and we worked together to reproduce and understand the issues and their severity. Ericsson suggested to handle the CV assignment and they provided a CV number for the stack overflow with a critical CVSS score and another CV for the denial of service bug, but I haven't updated the slides yet, with a medium CVSS score. Ericsson also announced that smart edge devices has reached end of life, meaning these vulnerabilities are infinity day. And we also brainstormed together on possible mitigation and they communicated the issues to their customers. And as for mitigations, well, I strongly recommend to disable PPP logs on the smart edge devices. As for the denial of service attack, I believe it can be blocked with switch configuration to block this kind of messages. And also smart edge, well, they are really all devices, they are from the back 2000s and it's highly recommend just get rid of them and replace them. All right, so to conclude, I hope I convince you that ISP network devices are not that different from home routers when it comes to vulnerability research. That old school is still cool and a 50 years old network device can still cause a big ruckus. And as always, ISP usually don't pay attention to updates when it comes to both their endpoint and their backend equipment. Right, looking forward, a blockpost will be published soon and I might also look into other attack surfaces I discovered while researching the smart edge devices. And lastly, I might have a look at other vendors and see if some of those technique works, I mean, the denial of service works. All right, thank you for your time. Feel free to reach out. Thanks. Feel free to reach out to me on Twitter and go see the latest season of solo opposites. It's awesome. And if you got any questions, feel free to ask.