 In nearly a decade of working as a network engineer, there have been a number of different things that have really gotten to me when it comes to the intersection of networking and security. Typically, when I push back on these things, it tends to make people squirm and sometimes even grab their pitchforks and torches. These are three of those things, things which really need to be thought more about, things that don't always improve your security posture within your network. And these are uncomfortable networking topics. Hi, I'm Charles Rompford, and I'm here to tell you about some uncomfortable networking topics, but first, a little bit about myself. For the last 10 years, I have worked as a network engineer in both higher education and in the corporate world, and before that, I was a systems administrator. Probably my most formative years in aiming in these topics was working as a wireless network engineer in higher education, where there's a strong convergence of usability, security, and networking, and flexibility, along with manageability, to make sure that the system is working. I've also done work as a network architect, automation engineer, Linux system man, I've also been an adjunct professor, and then outside of the workspace, I do a lot of photography, knitting, and if you really want to go look it up, I am a campanologist. We'll be curious to see if anyone asks me any questions about that. Before we get deep into the topics, I really want to put some constraints around the area in which I'm talking about. I'm in this talk, I'm not talking about client networks. I am talking about networks that are providing services to clients. Typically, this would be a data center of some sort, spaces where you might have web servers, you might have database servers, DNS, things of that nature. That is the primary focus of this talk, in the primary areas in which I am talking about. A couple of these don't really apply to client networks, but can be used in those spaces. I think everyone is familiar with the project management triangle. Choose two. You can either have a project be good quality and low cost, but it's going to take a lot of time, or it's going to take a little bit of time and be good quality, but it's going to cost you a lot of money, or it's going to be low cost and low time, but it's not going to be great quality. This is something that we think about a lot when it comes to projects. We don't think about it, but what do we think about when we think about design? I've kind of changed the triangle, and typically when we deploy something in the networking perspective, we choose two. Either it's manageable and secure, but it's not really usable, or it's usable and secure, but it becomes a management nightmare, or it's usable and manageable, but it's not always the most secure option. We think, I've run into this numerous times while working in the networking space, especially around security. We're deploying security architectures. What if I were to say there are a couple of things that we can think about that we can do inside of our networks that increase our manageability, and it also increases our usability while either holding security at a current stance, or possibly even improving it, probably saying that crazy, but just hear me out. So let's dive in, and for the number of years, there's one thing that has always gotten to me, and that is when I can't ping something. ICMP is your friend. Now, before you go and get your pitchforks and torches and come after me, hear me out. ICMP is an important, functional part of the internet. It is how the internet passes messages from one space to another. And they're not just messages of, hey, are you there? Yes, yes I am. But there are also messages of, hey, that's prohibited. Hey, you can't do that. Hey, that thing don't exist. And we've even used ICMP for trace routes in the past. All of these are ICMP messages, and they're all good things. But over the years, we've come to block ICMP because everyone thinks that ICMP is bad. That ICMP is going to make their system more vulnerable. People are going to be able to find it. That really isn't true, right? So I have some questions for you, right? What is the first thing you do when you think that you are having some network connectivity issues? I'll give you a moment. Now, if you thought to yourself you ping something, that's probably what everyone comes up with, right? I do it all the time. The first thing I do is ping the gateway, ping Google, ping Cloudflare, ping something, right? Can I get a ping through, right? We also use the pings, or ICMP messages, for reachability, latency, and jitter, right? All things that are easy to figure out using a simple ping, right? And in my mind, as a network engineer, if something is not pingable, then it's down, and it's not connected to the network. As far as I'm concerned, as a network engineer, is if I can't ping it, then its low-level networking isn't working, right? So let me ask you another question, right? Oh, there we go. Let me ask you another question. What if Google, Cloudflare, and Quad9 one day decided to block all ICMP inbound traffic to their services, right? Or better yet, as a network operator, I decide to block all ICMP to your network gateway. I basically, I say, no more peeing the gateway. That is not allowed in this network, right? Take a moment to ponder this. That default test that we all use for determining whether there's networking issues or not goes away. We no longer have that very standard test that we use, right? It's become a test that we take for granted, and yet at the same time, we cripple it on a regular basis. So why do we stop ICMP traffic to servers to services, right? Typically, it's because if it ain't pingable, then it can't be found. Well, I'm here to tell you that that is false. That is quite false, right? There are many, many ways to find a service on the internet without using ICMP, right? Security through obscurity has never been the answer. It never has been, and it never will be. And all you're doing is adding complexity to your situation, right? And so as a friend, I'm going to tell you something, right? An open port to the internet is an open port to the internet, right? If you have a resource on the network and you have an open SSH port or an open HTTP port or anything, someone knows about it. I can tell you that right now. At one of the universities that I worked at, one of the researchers on a regular basis scanned the entire internet in under 24 hours on a regular basis. Every port on the open internet enabled to map out what's going on, and she wasn't using ICMP, right? If a port is open on the internet, then it is open. So I say allow all the ICMP, and let's just make our lives much easier. It's easier to figure out what's up, what's down. It makes troubleshooting. The first lines of troubleshooting, so much easier, right? And plus, how many times have you run an in-map command with the dash pn flag to just bypass ICMP? We know the port's open. Why are we hiding something that's already available, right? So some things to consider when turning on ICMP, right? The first thing from a network engineering perspective is can you at least turn it on for the local network, right? If you at least allow it in the local network, then troubleshooting inside the network becomes easier. Yes, I would prefer to ICMP be open everywhere, but if you just start in the local network, awesome, right? But some other things to consider, you can rate limit ICMP, socially at your firewall or at the host level. Remember, host level firewalling is super awesome and super handy. Here's an example command from a Linux IP tables to rate limit ICMP down to five frames a second, right? We can also limit the types of ICMP messages. There are a bunch of different types of ICMP messages. We can limit which ones are allowed. It's one of the things that we do at the networking gear level is we limit which ICMP messages we accept and handle, right? This is something you can also do at the firewall and at the host level, right? But the other thing to consider is that ICMP messages tend to help improve the usability of the network, right? And the user experience of the network. Did you know that a lot of browsers when you try to navigate to a website, which isn't allowed, that when the ICMP reject message comes, that it will present the user with a message that says, hey, you're not allowed to do that, right? That's independent of the web application itself, right? Also, like I said, there are a number of times where I've pinged something, thought it was down and went running. One time I thought the whole set of DNS servers were down, went running down the hallways to see yelling the DNS servers down, the DNS servers down, to only have the Windows administrator pop his head out and be like, I'm logged into them, they're just fine. Turned out to be another issue, but I couldn't ping them. So as far as I'm concerned, they were down, right? So just a reminder that security through obscurity is yuck. It is not worth it. An open port to the internet is an open port to the internet. So changing port numbers, not that great, right? Once somebody finds it, it will be found and it will be pounded, right? The other thing I've also heard people doing is not putting things in DNS. This is not helpful, right? When you're unable to find something in DNS, unable to type something on a command line, it becomes harder to troubleshoot and understand what's going on, right? So I say put everything in, allow all the ICMP, put everything in DNS and keep SSH on port 22 forever and ever. The next topic is one that I have a lot of feelings about. It's one where a lot of people deploy it in the hopes of building a little security blanket around them, but in reality, they're just decreasing the usability of the network and dramatically making it harder to manage their network. And what I'm speaking of is NAT. NAT is used all over the place and I'm here to tell you that NAT is not a security barrier. Really, it's not. There's a lot of NAT happening on firewalls and things of that nature but I'm talking about NATting itself. For those of you who don't know, NAT stands for network address translation and there are a couple of different flavors of it but it generally allows us to translate one IP address and port set into another. The most common use is in your house, you're probably using NAT as we speak to watch this video because it allows multiple devices inside of your home network to talk to the internet via your ISP, right? But let's focus on the enterprise space, right? And NAT's typically used to handle two problems. One being not having enough IP addresses and two security. The first one, not having enough IP addresses is solved by technologies like IPv6. If we could get IPv6 deployed in more places and get rid of NAT, that'd be great but that's a whole different talk. I wanna focus on the security aspects for now. One of the big things that I find in the conversations I have is a lot of people love private IP addressing. They think that it will prevent people from contacting, being able to get to a device or a resource but in reality, private IP addressing isn't a magic security blanket, right? A lot of people put stuff behind NATs and magically think that everything is hunky dory, right? Just because it has a private IP address, it's all of a sudden magically protected and typically I start to see people getting lax about their security posture. I've seen a couple of times where things have been compromised after being moved into a NAT and having private IP addresses than when they had public IP addresses, right? Like I said, I've seen a number of administrators start to get lax about their security posture because everything has a private IP address, right? So let's take a look at some network diagrams real quick, right? So here I have a fairly standardish corporate network, right? We have the internet, we have some clients and then we need this NAT box, I'm gonna say is the edge of our data, like a data center type thing, right? We have some web services, we have some network services and we have some database services, right? But the first thing that NAT does is it creates this inside-outside barrier, right? Now you're probably thinking to yourself, oh, will firewalls do the same thing? Yes, but firewalls do not separate out IP addressing. We have one IP addressing scheme inside the firewall or inside the NAT box and one IP addressing scheme outside the NAT box. And this leads to all sorts of problems because everything now has two IP addresses, right? You've got the internal private IP address and you have the external public IP address. So that www.tallwireless.com, if you're outside the NAT, then it might have a one IP address, but when you're inside, you have to hit it on a different IP address where it might not work, right? Along with that, we have to open up ports via destination NAT to allow outside clients and devices to talk to things inside, right? That open, again, an open IP, an open port on the internet is an open port on the internet, right? We now have a lot of NAT to track. We have to track all those translations. We now have to have ways of querying it. We have to have ways of monitoring it to make sure we don't run out of resources. And if there's a security incident, it's just one more thing that has to be investigated to understand what's going on, right? Now, you can get along with doing what's called hairpin NATing or hairpin routing, which allows a service to access an internal service to access an external services IP address and then have it NATed back inside. So for instance, the database server could talk to the web server on its public IP address. But this is just more configuration that has to be tracked, managed, taking care of. And well, I don't like that. Simplification, all about simplification, right? Now, a lot of people do their NATing on a firewall, right? So if you have a firewall and you're doing that, you now have to open up the ports to allow the firewall, you have to set the firewall rules to allow that traffic in, right? So we've created this complex space that we now have to manage and track and reason about, which isn't great fun. All while having two separate names, IP addresses for everything. Now, I don't like this and I really don't like the idea of having a device have to contact something based on where it is in the network. I should be able to just access the web service and be done with it. So let's review real quick. NAT bad, right? There's more to keep track of. This could be all of the destination NATs you need to keep track of, the firewall rules, translation records, and there could be even more, there's now more to clean up during decommissioning, there's more to set up during commissioning, right? And there's something that could totally be left behind. It also increases our auditing space, right? There's so much more that needs to be verified, checks to ensure that everything is the way it should be. And my personal thing that I hate about NAT is that there's a two of everything, right? A classic example that I've used in the past is at a previous organization I worked at, you have a DNS server, right? You have one set of IP addresses to access the DNS server when you're inside the network and you have another set when you're outside. But they all go to the same server, but it means they now have to track in the configuration for all of my switches, routers, servers, where I am in the network to decide which IP address that we're going to, right? And like I said, network position means something. Am I inside the NAT? Am I outside the NAT? Am I having to go through multiple NATs because that happens a lot, right? And the other big thing is every time I've had to solve some major problem regarding NAT, I've had to deploy more NAT, more things to track, more things to keep aware of, right? So overall, we have a huge decrease in manageability and a huge decrease in usability without actually increasing anything in our security space, right? And then like the ICMP, we're just complicating everything without actually having any security gains because it's all just security theater. And we don't like security, at least I don't like security theater. So let's take a look at another way of looking at this, right? So in this diagram, in this diagram, we have the similar network, but let's make the assumption that everything here is public IP addressing, right? So the first thing is, is all the web services, all the network services, all the database services, all have public IP addresses, right? And I'll touch about the amount of IP addressing in just a second, but roll with me here for a second, Now, like the NAT, you still have to open up ports for the firewall rules. Easy peasy, that's simple, right? On top of that, to get that actual feeling of not being able to get in, we can set up a default deny on the inbound. Everything coming in will be dropped unless you explicitly allow it, right? This gives you the same comfort feeling, right? With actual security posture of having a private IP address. Having a public IP address with a firewall in front of you that says default deny is better than having a NAT in front of you with no firewall and no default deny, right? But in this whole thing, we've simplified the architecture. We're doing native routing between our data center services and our corporate network. So now everything is, there's no inside, outside. And so we can just access everything directly by the same IP address, the same name, and it's much easier to manage across the board, right? But it also simplifies our auditing space. It's easier to reason about inside of our heads when we're trying to figure out problems and do troubleshooting. And when it comes to maintenance, it's so much easier to figure out what needs to be done and be able to build an infrastructure that is more tolerant and resilient to handling maintenance, right? Plus we get an increase in usability and an increase in manageability, right? Now you might be thinking in your head, wow, that could be a lot of IP addresses, a lot of public IPv4 addresses. And well, I will agree that it could be a lot, right? I also know that there are a number of organizations that just can't afford that and don't have the staffing to manage it or the network connectivity to manage it, right? I will admit that I've come from a number of different organizations that have a large number of public IPv4 addresses available to them and this has not been a problem. Although they've kept using Natting to solve the problems instead of the giant pool of public IP addresses, right? But I'm very aware that public IP addressing is a rarity in today's world, right? I think we've almost run out in terms of non-allocated IP space. But because of this, there are ways to approach this in a hybrid manner that allow for reachability and security and getting the best of both worlds while still keeping the manageability and the complexity down. Now I will say, you can go native full IPv6, there are so many addresses in there, but again, another talk. So let's look at that diagram one more time. So in this model again, right? Let's just say, for instance, our database services, we give them all private IP addressing, right? So now the web services can talk to the database services, great, clients in the corporate network don't need to talk to the database services, they only need to talk to web and network services, so we'll give those public IP addresses and we're gonna natively route all of that stuff internal to the data center out into the corporate network. So if there is a reason to contact the database services, they can, plus we can put a source NAT on the firewall that will allow the database services to access the internet to do things like updates and things of that nature, right? This is a model that I've seen deployed and I actually deployed myself because I am also not a rich person and can't afford the amount of public IP addresses that I need for my own personal infrastructure, right? So this is a great model, right? So really, public IP addressing, it's awesome, right? You should, I'm not saying go run and rip out all of your private IP addressing and go buy hundreds of public IP addresses at really expensive prices, but I'm challenging you to think about how you think about private IP addressing and public IP addressing in relation to the usability of your network and the manageability of your network and what does that add to your network overhead in running it? One of the final areas that I wanna talk about is network layer encryption and how it should not be relied upon. A lot of systems people and application developers rely on VPN and site to site VPNs for their encryption properties, but it ends up binding the network architecture into a very specific path. Now I'm not talking about client VPNs at this point, I'm talking more about site to site VPNs that are used as shortcut paths to connect to say a data center, two data centers together or to connect the data center to the cloud, right? But one of the things that happens with these is that people start becoming very dependent on those shortcut paths and especially the network, the encryption aspects of that VPN connection. And it should be noted that VPNs don't necessarily have to be encrypted. There are a number of different VPN technologies out there that don't encrypt traffic at all, right? One of the more common situations that I've seen is when people start getting into the cloud, they wanna build VPN tunnels between their data center and the cloud. And these are IP second encrypted VPN tunnels and there's nothing wrong with that. But one of the problems that happens is that people start to treat that as part of their data center and then they start to rely on the fact that that pathway is in fact encrypted. And what this does is it binds the architecture to that pathway so that once you need more throughput or you have a lower latency physical connection, you're now so bound to that encrypted nature that you can't move off of it. And people get scared about passing that traffic over other links through other networks without it being encrypted, but at the same time they don't want to actually encrypt that traffic at the application layer. So I'm here to say, stop relying on network layer encryption and start moving that up into the application itself. There are a number of different technologies, TLS even endpoint to endpoint IPsec are great technologies to use to bring that encryption up into the application or up into higher levels of the network stack so that we as network engineers are able to just move packets and worry just about moving packets and not have to worry about traffic taking a very specific path because that path happens to be encrypted. That should all be handled inside of the application layer traffic, right? So we've come to the end. There are a couple of takeaways that I hope that everyone gets away from this talk. One, ICMP is good. Two, security through obscurity is bad. Three, not bad, public IP good. And four, bring that encryption up out of the network and into the application layer. Well, there's another thing that I hope that everyone walks away from this talk with, which is I hope that everyone starts to take a little more time to think about not just the security aspects but also the manageability and usability of the systems that they develop. It's really easy to get caught up in we need to make this secure, we need to make this secure and not think about how are users gonna use this? What are the workflows look like? How are we gonna do maintenance on this? How are we going to troubleshoot this? How are we going to do all of those things, right? Complexity can be managed, but it can also be shifted and it can sometimes even be eliminated if you just sit down and think about it a little, right? Remember to take a look at the whole system. Take a look at what your actions today will have on future plans and longer-term strategies within the organization. I'm aware you should also have some longer-term strategies and bring in some networking subjectmatic experts, right? To help with the network design and the network aspects of the thing you're designing, right? We might have suggestions on how to achieve what you're trying to accomplish while not crippling the network or not binding the system into a specific design or setup that locks us into something forever and ever, right? And there's a lot of network people who really love building secure networks and systems who also think about usability and maintainability. So, reach out to them, right? I wanna thank you all for your time and thank you for listening. I'm on the Blue Team Discord server, answering questions if you wanna reach out. I am always happy to talk about the intersection of networking, security management and usability, so please reach out. I'm not that intimidating, even though I am very tall. And you can find me on Twitter under tall wireless, but I'm gonna warn you that I don't just post about networking and security. You're gonna get a whole lot of random. So, I hope everyone has a great DEF CON and I will talk to you guys later.