 Oh, there we go. All right, thank you, everybody. Welcome to I am Become Load Balancer, owner of your network. I'll do a little introduction about myself. Yeah, I'm Nate. I have been hacking networks since I was about 12 years old. I spent about 17 years building and troubleshooting networks. I worked for Microsoft for seven years, four of those years, or two of those years I was in the network engineering group. I spent a bunch of years shipping out Windows patches for Patch Tuesday. So I'm sorry, but that's what I did. And I did a stint in Defender for Endpoints, doing some endpoint research. And I'm about to start a new job on Monday that I'm not going to give away the secret just yet. I worked at F5 Networks for 10 years. Seven of those years, I was Microsoft's dedicated engineering design escalation support contact, which is where I get fun pictures like an F5 device encased in the block of ice, which we will get to the story behind that a little bit later. I like to speak at conferences. This is my first time at NorthSec. I've done primarily blue team defensive focus talks. So this is the first talk that's more on offensive evil shenanigans thing, which I'm excited for, getting a little bit of feedback. I was featured in Wired Magazine in 2020. I helped start a group called CTI League. We built a 1,500-person volunteer InfoSec group that was giving out free threat intelligence and perimeter assessments to hospitals to try to prevent them from getting ransomware during the pandemic. And then my side stuff, I like to spend drum and bass music because I'm an old school techno kind of guy. Enough about me. So the TLDR for those are maybe uninitiated. What are load balancers? These are very large, expensive pieces of networking hardware. They're usually deployed in what they call a failover pair. So if you think of like HSRP for routers, it's the same concept, right? You've got a device that's passing traffic. If something happens to it, you can failover and can ideally not lose any of your connectivity. They do layer four to layer seven load balancing. Some of them can be web application firewalls. You can do VPN, you can do DNS load balancing. They're sort of like a Swiss army knife of networking for things that are not just your basic switching and routing. They also do SSL and TLS offloading. So people will use these things when they're actually F5's biggest selling point first was you could have a very powerful SSL accelerator chip, slap the certificate on it, load balance it across your pool of dozens or hundreds of servers and save money on SSL and also be more performant than going to specific servers themselves. The nice thing about these things is because they're such a sort of a core routing and traffic shaping device, you get almost unfettered access to the internal network once you hack onto one of these devices. They're generally on multiple VLANs. Most of them have internet connectivity. Some of them are talking to Active Directory. There may be VPN sessions. So they're a very juicy kind of fun thing once you get on it to see where you can laterally move and pivot and the other sort of maliciousness that you can perform. And they're mission critical, right? These things are the think of like your core routers. These are very important devices which are generally not updated unless someone absolutely has to. F5 has had some code quality issues over the years and people who've managed them kind of get to know that if you don't need to upgrade the thing just leave it running. So a lot of these things are running two, three, four, five, 10 year outdated versions of code. And the nice thing about these from the attack perspective is because they're proprietary you don't have much in the way of EDR endpoints like solutions that can monitor what's going on with these devices. So even when we were at Microsoft there was we had remote logging and things like that but there wasn't really any way you could detect if someone got on the device at least not in any timely fashion. Another fun thing about it which is we're gonna get into here with these vulnerabilities is that these devices always have a web GUI enabled. The F5s in particular are Apache with Tomcat. I wish I could tell you it was updated, it's not. So there's also other vendors like Citrix, A10, a few other companies. I'm gonna be specifically talking F5 today but the design concepts are very much the same, right? Citrix runs BSD as their management operating system F5 runs CentOS. You can pretty much most of what they do is identical. It's just slightly different like commands and things to get around. And then they also have shells. So Bash is the standard one. They have their proprietary TMSH one. I will point out that as I'm going through these slides you're gonna see TMSH commands. The reason I'm sticking them in there verbatim is because you can then plug them into the POC that came out last week and run these commands. So the idea is you can copy, paste and play with this stuff in a lab which at the end we'll talk about how to build one. So the deployment methodology. When I was building this talk I was initially going to use the CVE from 2020 this 5902 past traversal vulnerability that F5 had in June of 2020. That was what I was gonna use to show all the demonstration stuff for you. However, they blessed me a week ago with an even more easy to exploit and an even easier to POC vulnerability that's also a CVSS 10 which essentially without getting too deep in the weeds you take a connection header you basically stuff a authentication connection header inside the connection like the connection colon header feed it to the device and it believes you now have root access to the machine zero authentication required you can run any command as root it's gorgeous. And all devices have management so they sell these things it's generally a big networking switch they also sell VM versions of it we're not gonna go too much into the VM stuff except at the end because that's how I built the lab to show you this stuff and I hope that's kind of readable I mean it's green on black if you're not using green on black are you really hacking things? So they have an OOB management interface the hardware devices will have a switch a bank of switch ports and they'll have an actual like gigabit interface that people can plug into the management side. They also have these network interfaces you generally plug them in to a trunk port a lot of people then tag B lands on top of them and then because it's in a HSRP kind of design each device will have its own non it's a static IP that's attached to the device and then there's an IP they call the floating address this is bound to the device that's passing traffic this is where your server's default gateway will be if the device fails over that IP address will float over to the other device as it takes over the thing that people don't realize and we saw this last week when people were doing incident response is that not only is the management GUI and SSH enabled on the management interface every single self IP address on the device by default will be listening on management traffic so people may be thinking oh I'll just turn the management interface off and I'm safe every other self IP the device has is also listening on those management ports so it makes locking these things down complex behind these things are generally pools this is the taxonomy that F5 uses pools of servers which is just your server resources this is all stuff you can look at in the config you can kind of figure out what's back there most people name these pools fairly explicitly so it's like web server pool or my active directory pool or whatever their load balancing so there's all sorts of juicy stuff we may not be able to get too deep in the weeds on the virtual servers are the actual traffic interfaces this is the thing that's going to be facing the internet facing the clients this is what everybody talks to and then it sort of just disperses traffic across the back they use these concept of profiles this isn't super relevant but I'm trying to give you an idea super isn't super relevant to attacking them but it's useful to know how these things operate so they'll make a virtual server what these things will have is like a layer four profile if it's going to serve let's say it's serving TLS it'll have a layer four profile for TCP then it will have a profile for HTTP because it's going to be doing HTTP traffic then it will have another profile for SSL which is where the certificate and all these other details are stored so these things have a very it's a very kind of convoluted way of configuring them but once you sort of dig in around you can start to understand okay how is what is this thing doing and it's yeah it just takes time one of the other things to notice about these is that when they fail over because they need to shift the layer two address that all the servers are talking to for their default gateway and at the very least they need to update the switch with where this layer two address is like they have a concept that they call Mac masquerading where you can have a Mac address that's on the floating address so if it fails over the Mac address isn't going to change for the back end servers but what it is going to have to do is the switch the cam tables in your switch are going to have to know this Mac address change from this port to that port so if you fail these things over by accident you're going to get that's going to get noticed they're going to all the scenes you fail over event they may notice that traffic gets interrupted for a second the idea here is I'm trying to teach you how to not get caught if you hack onto one of these things and only a red teaming environment of course so a little bit about how these things work at the low level this is a slightly outdated slide or a slightly outdated picture they split the things into sort of two planes this AOM part of the graphic that you're looking at is for one of their older platforms I don't believe the new platforms have these anymore that was the always-on management it was literally a separate CPU that ran its own separate instance of Linux that you could mess around with if you got onto it but the really important part is there's what they call the traffic management microkernel this is the code that F5 writes that excuse me processes all the traffic it's what handles all the load balancing it's all the sort of brains of this device everything production happens there you'll see this HMS part this is the host management subsystem so this is where your Linux stuff works this is where this is the part we're going to be attacking the other important thing if you break TMM if you do something that causes it to lock up the devices will fail over and you will probably get caught if people are paying attention this is usually noticeable now the management side is CentOS it's not a very updated version of CentOS I think their most current up-to-date code is running like CentOS 7.3 and I believe CentOS 8.0 is current some of their older platforms go all the way down to CentOS like 6.5 so you can pretty much do whatever you want here and the interesting thing about how the architecture works is when the devices boot up at least the hardware ones the first thing that takes over all of the resources of CPU and memory is their proprietary TMM code then it yields back a certain percentage back to the Linux side so you're kind of protected from really going breaking anything too badly because there's a check there it says okay if the Linux side uses too much CPU resources it just stops giving it resources so the idea is they want to always pass traffic and never let you infinitely loop something in Linux that then takes down the whole device now your traffic plans on these things can be tens possibly hundreds of gigabits a second at this point this is a little bit out of order but should you get on one of these and should you decide to start sniffing around or maybe you just trying to capture sensitive information or whatever your sort of mission and this engagement might be don't put TCP dump on any of these interfaces that have TMM attached to them if you've ever been a Cisco router or a switch person that's like doing a set debug all you will basically be dumping 40 to 100 gigabits of traffic into TCP dump on the Linux side that will cause TMM to lock up the device will stop working and you will probably get caught one of the ways to figure out what platform you're on is this is our first of the TMSH commands if you ask it to show Sys hardware and I apologize that I don't have an example of what it looks like it will actually tell you what hardware platform it is is it a virtual instance how much processing power it gives you ideas and then you just go look it up on their website and say okay how big of a box am I on right I was like am I on the $2 million chassis or am I on the like $40,000 chassis so once you get on one of these things say we use one of our fun exploits to get root there's some things you need to know to not once again not get caught these devices because they're in failover pairs you use the concept of a shared configuration right so this is the stuff where you add your virtual servers, your server pools your SSL things and this is of course synchronized between both devices so that if it fails over it has the same config if you change things here there's a fairly good chance you'll get noticed the devices are smart enough to notice when a change is made to one side and it'll say oh I'm no longer in synchronization and if somebody's paying attention a big if they may say why did the devices just go out of sync in their config let's go take a look as I said changes that impact the traffic plane will definitely be noticed if you start doing things like changing these you start messing around with the load balancing configuration or trying to fiddle with the actual stuff that's serving traffic a lot of times it'll happen when you when you apply the change it sort of resets that configuration so you may drop active sessions you may cause a traffic interruption like some blip that will get caught people say what happened to my F5 why did it why did it's like have a blip now if you just change stuff on a single device there is a section of the config as I mentioned like the self IPs are just for the single device that doesn't get synchronized nobody really notices that so keep that in mind and if you don't know how these things work if you're not familiar with the underlying tech don't touch the traffic plane they have this cool technology that they call iRoles which is basically a TCLTK modified TCLTK language which allows you to do deep packet inspection on traffic flowing through the device and then you can manipulate and steer traffic based on like binary payloads I think I know one person in the world that's good with these things yeah I mean if you want to mess around there you can but I would highly advise that you either steal their config and then set up a VM lab and try to do it but it's there be dragons there so once you get on all of this is sentos right so the logging is all in var log they do have like I mentioned remote logging so this is the type of thing you would you can check using this exploit you can actually say you know before you maybe jump on the box and start doing things you could say look TMSH lists this syslog it'll tell you if the device is doing a remote syslog if you dig further into their configuration documentation you could in theory disable the remote logging before you hack onto the device and sort of pre-cover your tracks they have a proprietary somewhat proprietary authentication system it's based loosely on Linux PAM the only account you will ever be able to add to the Linux side is or you won't be able to the only account you can log in that's actually a Linux account is root so trying to echo something into Etsy password or create a user through the add user thing it's not gonna work they just you can create the user but it won't let you log in with that user history files you know clearing your tracks home user is where all your history files are created there's two of them there's your normal bash history and then you've got the TMSH history you need to clear out both of those if you're trying to keep people from seeing what you're doing and let's see here so user accounts so if we're gonna hack on to one of these devices right we've got this cool exploit from last week the idea because it gives you root access you're gonna need to log into this thing somehow now more advanced red teamers you may be able to spawn like a net cat reverse shell I'm not a red teamer full disclosure so you probably have better tricks than I do and if you wanna talk about it later please come see me but you can create user accounts and it will be noticed right if you created a user account you'll notice that it goes from online and in sync to online and changes pending so this is the kind of visual cue that if someone's watching it they'll notice well something just went differently however because users are not a traffic serving part of the configuration you could then just synchronize the configuration it won't interrupt anything and the all of a sudden the devices will go back to being in sync and you've created your own user and now you can log in and do whatever you want the advanced shell and I have an example of this coming up when you create a user when these user accounts are created they give a whole bunch of different permissions but there is an option of which shell you give them if you say shell none then they can only log in via the GUI if they say shell TMSH then they can log in only with TMSH which restricts command line access you can generally feed bash commands through it it has a bash option you say okay just bash minus C give it the bash command it'll run it but what it won't let you do is just drop to a full bash shell and start running around in the file system and doing all the fun evil things we like to do the other interesting thing about these devices is you can actually disable the root account at Microsoft we did this all the time we would basically set them up we'd lock them down we restricted SSH access we turned off the root account the fun thing about this I was testing this in my lab as I disabled it and then I used the exploit to check whether it was disabled and I was like oh the root account is disabled so then I used the exploit to re-enable the root account and now I could log back in now this seems like pretty cool but keep in mind I knew the password for the device so if I don't know the password for the root account my options are either change it which now if they go to log in and it doesn't work that's definitely a sign that something's wrong or just add a user and give them bash access and then continue on with your day but it's an interesting way that these management the management stuff that they've done is sort of convoluted and there's so many attack paths the vulnerability that came out in 2020 like I said past reversal against the Java servlet page in Tomcat that allowed you to run TMSH commands as root this one was the connection header which hit a different it hit the REST API that allowed you to log run commands as root it seems like we're in Groundhog Day sometimes the other interesting thing about it is they don't have a firewall per se like these things have the concept of what they call a net it's a self allow list let me see I have a picture of that so what this is is essentially just a list of ports and protocols that you will allow to be accessed on the self IP address now some of them if it's a SSH there is a daemon running there if it is TLS there's a daemon running there if you turn on some random other port it just means that you can talk to it on that port the fun part about this is because they're not shared and again we can go if you get one of these devices you see okay well there's a root account or I've created myself a new account and then you try to SSH in and you get a connection refused well you throw the exploit payload and you look at what the actual self allow list is and you're like okay well SSH is disabled well then you just throw the exploit payload and enable the SSH port and now you can get back in and this isn't shared nobody's going to notice unless they're paying extremely close attention they also allow outbound connections by default the thing though to keep in mind is and the way we would deploy these in sort of like a really good environment is the actual device itself had no default gateway there would be a default gateway for the management network but what we would do is we would put them in a network we would plug them in trunk them put a bunch of VLANs and then the device doesn't need a gateway to talk to to pass traffic back and forth they use this funky thing that they call auto last hop which essentially the device will record the layer to address that it received traffic from and then when it goes to respond or it's passing the load balance traffic back it'll say well I don't have a default gateway in this network I'm just going to send it to the layer to address that sent it to me so it's an interesting way of routing without actually needing a router if you will and then to be consistent they have three different names for their self IP ACLs if you get on the device itself it's called a self allow list in the configuration or it's a self allow list when you're changing it when you look in the config file it's called an allow service in the net self configuration and then if you go to the GUI it's called port lockdown confused yet back towards the web shell so the thing about these devices like I said they run sentos they only have Python 2 I was going to do a whole bunch of cool demos with like impact it and a bunch of other things and I realized most of the fun hacking tools are written in Python 3 now the other thing about these devices is they've stripped out all of the stuff that would be useful for you so there's no compilers they are sentos but they don't have the rpm command so you couldn't even like grab a Python 3 rpm so if you're going to take post exploitation tools with you you're going to need to build them and test them in a lab make sure that they work and then you can just drag them over to the device when you actually get on to it they're mostly sentos on x8664 most of these devices run intel chips so it's not really that difficult to do this stuff it takes some preparation and planning what they do have because of the way these devices are set up and because of some of the things that they need to be able to do is they have a full suite of LDAP tools I believe it's all the standard sentos stuff so you've got LDAP tools you've got SMB Client you've got Netcat, ironically it's Cron, you've got RC scripts you've got an SSH daemon I think they might even have FTP and Telnet on them still not as a service just in the daemon itself or the client itself so this brings me to a kind of fun part which is as I was building this talk I was first I was like okay this is cool and then I don't know if anybody is going to grok this Mandiant comes out with a report two weeks ago about this threat actor called Unc3524 which they believe is a Russian associated state actor whose way of getting into networks was hacking into load balancers sand devices, conference room cameras phone systems all of the devices that run for proprietary code where you cannot run EDR I was like so honored so if you want more information I highly recommend this article it's super fun so I notice as I'm reading this they're talking about their quiet exit backdoor which is this modified drop bare SSH daemon and this really funny part was somebody sent me this while I was building the slides and I had just finished pocking out a Cron entry to build a reverse SSH tunnel on reboot now again not a red teamer it's not an efficient way to do it because I had to generate an SSH shared key put the private key on my C2 server and do all this other tomfoolery but this worked I was like okay I just plopped this in there I rebooted the device and then it boots back up and makes an SSH tunnel back to my C2 once again I have to log in with an actual username and password on the device but again you red teamers you can do all of this stuff and then the other thing to note is if you were going to drop a web shell I believe the web path is like user share something they mount the user file system read only so I've seen a lot of attempts when we were doing IR where there's these commands trying to sort of echo a PHP file into the web directory that's not going to work the advanced actors have figured this out and we saw this in 2020 the first thing they would send was a remount command so it doesn't look any it doesn't look like anything happened and that K number is the knowledge-based article on F5's website that you can go and find out more about this there's also a bunch of links at the end that may be useful for you there's some fun things you can do with network device discovery on these things so one of the features that their customers from F5 really love is this concept of cookie persistence I believe they actually have a patent on this thing but what it does is you say okay you get a cookie that persists them to this specific server and that way the traffic will work and the concept too is if it fails over because the configuration is the same you'll still be re-homed to the very same server because you have the cookie the device says oh I know what server to send you to so it's actually one of their cooler features the better part about this feature is you can decode these cookies and you can figure out what the backend IP addressing scheme looks like because and it might be a little hard to read the IP server that's what the cookie is named by default and then you can actually look in the details of it and it tells you the pool name inside the device and then the numeric strings of the IP address and then the port that it's running on and I think there's some other piece of detail that it's been 10 years since I worked there so that's kind of a fun thing if you don't even have access to it yet you figure okay what is the backend server network look like the other fun thing like I said these things are SSL TLS concentrators so a lot of places will just not run SSL on the backend so they'll do it securely on the front and then everything in the back is clear text now most banks don't do this they do have the ability to do re-encryption on the backend so you see like financial institutions and people that are super security aware will re-encrypt but a lot of people that are like I have an online commerce website I'm just going to SSL it on the front and then clear text it on the back yolo so like I said be careful if you TC feed up figure out what server you're going for just craft a TC feed up string to only pull traffic for that specific system and then it's far less chance of you getting in trouble they do remote authentication so they can authenticate against LDAP Active Directory, Radius, TACACS this is all stuff you can see in their config files so again TMSH list auth will show you whatever their remote authentication stuff is set up as if you see an auth source of just an empty parent it means that it's only using an account on the device if you see anything else it'll say type is Active Directory it'll have the SAM like the DN search path it'll have the SAM account name I couldn't figure out how to crack the passwords I think they are using salted encryption when they do these things but if you're a password cracker like go to town on it and then TMSH show off will give you users failed logins and accounts that have been locked out on their local devices they just kind of leave it as the default and the default does not lock you out after a number of failed attempts and then one of the fun things about these is because they deploy them in pairs they actually have this concept of clustering so you can deploy 3, 4, 5 of these things in a cluster, I don't know why you would but you can use the exploit and you could actually like discover other devices on the network so let's say somebody forgets and leaves just one interface in a place where you can reach it and it just firewalled off well you can use this, you know, list cm device and it'll show you all of the other devices in the cluster and then you can go and theoretically laterally move I'm going to redevelop this talk later in the year and probably try to basically have the device exploit the next device by using the exploit say okay send this command use the payload to get to the next device pull a shell back to me so it's just how evil are you then the GUI runs on 8443 for VMs so if you're scanning 443 in a network scan you might find some virtualized F5s and then I have showed enquiries that I like to throw out there I like to discover things on the internet when the vulnerability comes out you can get some of those there valuable config items and this is where the beer thing comes in so this was actually one of the Xbox alive of F5 devices when they decommissioned it they were so happy to have it out of their network they threw a party and this turned into the world's most expensive beer tap and then in case in a block of ice not even joking and beer was pretty good so configuration items are all in slash config the big IP base conf is your base device and networking this big IP conf is your shared configuration the user config is here and big IP user you'll notice that this is actually the old 2020 payload pulling out the user information which is kind of ugly and then this was the new one from 2022 which is cool because it's JSON so it actually comes out formatted and it looks beautiful and then TMSH list off user will give you hashes the hashes are not in the config config file store this is where all the configuration SSL certs, keys, all the juicy stuff that these devices can have is stored in there config GTM we don't have time to go into the DNS side of these things but if they're doing DNS load balancing and you can get root on one of these you can imagine all the evil things you can do it's all in config GTM you can also do what I would prefer to tell you to do is do a TMSH, save the UCS which is the config backup it's just a tarGZ file that they renamed .UCS, save it, download it rename to TGZ unpack it, you have every valuable file that the device has and you can go through it without as much risk of getting caught so how to build a test lab this is what I did so they give away virtual edition VMs for free for all your major hypervisors so we got KVM, Hyper-V VMware they also give away vulnerable versions so the version that I was running was vulnerable to the 2020 CV it turns out it was also vulnerable of course to the 2022 CV but it's cool if you figure out what version it's on just download that version and now you can start messing around with it I thought you would delete vulnerable versions of code from your repositories but they don't like I said this is how you test your toys and if you want to do load balancing actually license it you can use a throw away email to get one of these they don't check to see if it's a valid like a legitimate company just go to throw away mail or any of those sites you set up an account ask for registration keys, they give you a set of three they're good for 30 days you can do this as many times as you want I think I generated 30 of them in the process of testing this lab I use Proxmox this screenshot is actually the config from my Proxmox server it's very complicated and finicky to get this working if you have problems see me afterwards you can also run these in clouds azure, aws, virtualize instances same process for a demo license except you also have to pay for cloud computing time up to you and then you can download isos too the same thing with a throw away account so if you'd like to pull apart isos and look at all the stuff that's installed and see if there's old packages that you can exploit you can do that too don't buy them off ebay, if you buy it off ebay you will not be able to use it until you have a valid support contract from f5 which costs thousands and thousands and thousands of dollars a year and if you're interested in doing any of this stuff I'm happy to help you with research find me later, hit me up on twitter and with that your reference material is here I will be posting these slides to my github later on today like I said, come find me and thank you very much and like I said, I mentioned I was a DJ this was me 20 years ago before the beard I promise I wasn't born with this thing I have a soundcloud because it's what you do and thank you very much NorthSik