 all right. So we're ready to go. It is six o'clock. First off, I want to thank everyone for coming out so late. If you can come in for my talk, I guess there's a lot of competition here. I unfortunately am not as funny as some of the other talks right now, so. All right. So this is staying persistent in software-defined networks. My name is Gregory Pickett with health fire security. I am part of our cyber security operations group. Overview of today's talk. White box internet. What is it? It's not common knowledge right now. Start emerging really. After that is stupid is as stupid does. Next is exploiting it. I'll point out a number of weaknesses and what we can do to take advantage of them. Next is moving forward. All right. How do we remediate those weaknesses and the vulnerabilities that they involve? And then finally wrapping up. Where is all this taking us and what we're trying to accomplish, what I'm trying to accomplish with talks like this. All right. Let's start out with white box internet. Well, it's standard hardware. Blank slate. Running merchant silicon. Trident and chipsets. Intel AMD power PC processors. Using an open operating system that is often Linux. But I don't think I've seen anything that wasn't Linux. The idea is to remove the hardware from the equation. To make it a commodity by using off the shelf components. So from that point forward it would be the software that made the difference. It's critical for software to find networking. But it can be used without it. Of course the question is why do it? Why are we moving in this direction? Well it's the same reason why they're implementing software to find networking. Reducing the cost. Increasing the flexibility. Looking of course to gain more control. You can gain the control through traditional means. You can actually remote to white box either a switch SSH. You can issue commands to define the switch. To configure the network. Just like you would. Regular switch. Traditional switch. You can also script for management purposes. Or you can load an agent on it. Like puppet. Chef. To make it part. You know dev ops. This network automation orchestration. Or you can gain that control with software to find networking. With software control plane. This makes the network flexible. And of course responsive. Everything is in software. With white box either net your hardware or data plane. I don't know. Right out of the camera shot there. The data plane becomes just as flexible. You can use whatever hardware you want. You're not tied to vendors. Your technology. Any sort of commands. You learn all the Cisco certifications that require those sorts of things to point out one particular vendor. You get hooked up with them and you're kind of stuck with for life. Now. To make all this possible right. To make white box actually plug and play. They adapt to ONI. It's firmware for bare metal network switches. There's a boot order underneath. You can grab or you boot the system. And then ONI is there to boot the network. You grab, install, network operating system. And in the event you like a new one. You have to change the software. It will then go out and grab another network operating system and install a different one. You can change whenever you like. ONI comes pre-installed. On the network switch it's part of the firmware. And it automates that switch deployment. So white box is in the net. The blank slate. And ONI is in the network boot loader. What could go wrong? There are a number of weaknesses in the operating system. ONI itself actually is Linux based, right? The privilege account. The root has no password. And ONI does not force you to change the password. So it's likely to remain stale. Management services. I'll use a telnet, right? Insecure by design. Multimally weak. SSH is weak too. Installation mode. The key is only 18 bits. An entropy there. I'll cover mode 26 bits of entropy. Now looking good so far. The installer has weaknesses as well. It uses a predictable set of URLs. I'm just defining the standard. Goes to, you know, it's a one URL first. Another URL after that basically goes through a series of URLs that are defined by the standard. The process it does or carries out to install. And where it goes is very defined in the standard. First off, you get an exact URL from DHCP. After that, it will build a URL from the DHCP response. After that, it will look to its IPv6 neighbors for an installation. And then, following that, all that failed, it will go ahead and go through a DHCP waterfall. And so, it's very predictable where Oni is going to be looking. And when it arrives at a particular location, it goes ahead and looks for particular files. It goes through a series of file names that are defined again by the standard. So as an attacker, you can pretty much, you know ahead of time where Oni is going to be looking and you know ahead of time what Oni is asking for. So you can actually hand Oni a compromised installation and Oni will install it. And that's possible because there's no encryption or authentication. Now, should you hand Oni a compromised installation or any sort of installation, once that nerve confidence system is running, Oni's partition is exposed. It's exposed, it can be modified. It can itself be compromised. And with most of Kibbut, there's really nothing to stop that compromised installation, that compromised Oni from continuing to operate over and over again and doing whatever an insurer is told to do. What does this mean, of course, with all these weaknesses? There's lots of opportunities to blow Oni up. It is not very well protected. So you can blow it up. And the first thought, of course, is compromise it directly. With a rip passer that starts up blank and is likely to manage stale, you could of course log in, right? We could sniff the traffic or perform in the middle, modify traffic flows, your telnet, or easily cracking SSH, but that's not likely. Oni is up, just throwing the number out, 0.001% of time. It's job is to install an error-coppering system. All 45 seconds, once it's done with that, it sets a boot variable, which basically says from that point forward, boot an error-coppering system, reboot to switch, and then an error-coppering system runs for the rest of that time at the 99.99% of the time. Next thought is then can be compromised installation, right? Be a rogue TTP server, a friendly IPv6 neighbor, or maybe spoof a TTP server. It's also very difficult, right? It's like right place, right time. If you happen to be there at the right time, you get it, you get a compromised installation to it. So you're there for a while, right? You're there for however long the out-print system lasts, but you don't gain that nirvana of, you know, of hecherdom, right, which is persistence. And I like that you're going to be in the right place at the right time again. So is there a better way? Compromise it indirectly. An error-coppering system is going to be up for 99.99% of the time. See what you have to work with. Get past an error-coppering system, modify Oni, it's an exposed partition, right? Sit in there, take advantage of that. Compromise Oni. No secure boot to stop you from doing this and take advantage of Oni there. So over and over again, this will retain your persistence, right? Oni will keep doing whatever you tell it to do. So if you compromise the nirvana system, you are able to compromise Oni. Then any time a new nirvana system is installed, you are basically back again, right? Because now you're in the firmware. They're essentially there forever. And that's what we're looking for, right? As attackers. Or as white hat penetration has, we're looking for that persistence. So, network-coppering systems. Installed by Oni. Operate the switch to actually do all the packet forwarding and provide all the different features that you like to see in a switch. Only compatible distributions, right? There are a lot, actually, of network-coppering systems. But only handful right now that have Oni-compatible distributions. The number growing and they just added peak eight, I think, a couple of months ago. But when I started this, there's only about four that were really prevalent. First is open network Linux. After that is Switch Lite. Genius Linux and Melon Ops OS. Open network Linux. Next distribution for bare metal switches, as they all are, of course. Based on Debian Linux, this is very popular. It's bare bones with no features. It will run the switch, but you're going to have a real hard time defining the switch, configuring the network, because really it's just a reference. It's a starter that the open compute project came up with. They want you to use this to develop your own network-coppering system. They're obviously there to promote Oni. So, they're giving you the starter, so yes, it can work with it, and you can actually make something that will run with Oni. And that's what Switch Lite did. I looked at version 2.6. It's a packaged open network Linux. They took that starter and they added SLRest, which operates a lot like OBSDB to define the switch. Added an indigo open flow agent for loading on the flow table to configure the network. This is not a standalone though. It really is part of a total solution called Big Cloud Fabric. The idea is to plug the switches into the network, and then Big Cloud Fabric just takes over. You're actually discouraged from managing the switches. You are for all intents and purposes at least the way I look at it to ban on that switch to Big Cloud Fabric. We will see how well that turns out. And, of course, the whole, the total solution is maintained by Big Switch Networks. QMS Linux looked at 2.5.3. It's important to know that you're, that I'm looking at the latest versions, right? So, I want you to know what those versions were. It is based on Debian Linux as well. You would install Puppet, Chef, Ansible. It's because it's meant for DevOps environment. Managed to it the rest of your infrastructure. And it's maintained by QMS Networks. And finally we have Mellanox OS. This is version 3.3.4. It's based on Enterprise Linux 5. There's actually a newer version 3.4. But it turns out all the problems that 3.3 has, they're still, they're still there on 3.4. So, just so you know that. You would install Puppet, Chef, Ansible to make it part of a DevOps environment or e-switch to make it part of an SDN environment. So it's very flexible. Maintained by Mellanox technologies. So we, of course, this is what I do here. Introduce you to these and then talk about really the weaknesses. No encryption or authentication on switch lights in to go. Nor Mellanox OS e-switch. Within to go it's just a matter of spoofing the controller. e-switch, you just talk to it. It'll listen to whatever you tell it and just do whatever you tell it to do. Well, outdated OpenSSL. This was done because honestly I needed to fill out the fly a little bit. So, yes. I didn't want a lot of blank spaces. So switchlight, they are running a bit behind OpenSSL. And when I saw it, of course, I'm picking Heartbleed. And I did check that and Heartbleed is not a problem. But still, as someone who looks at these things, who looks at, you know, how hard these environments are, I don't like to see old software. I don't like any of us do. So, with no encryption or authentication, then these, of course, are normal to topology flow and message modification through unauthorized access. Add access, remove access, hide traffic, change traffic. It's been mentioned, I think, in the press a bit about, you know, eavesjopping. And that's, of course, something that is likely to happen on a switchlight network or Mellanox OS, right, running, Mellanox running the e-switch. But there are bigger problems. And this is what's going to end up leading to the persistence. All right. We start out with something rather simple. You know, default picks accounts. Switchlight has admin. Cumulus Linux has cumulus and Mellanox OS has admin, which are their low-privileged or, you know, safe accounts. This is a big deal for two reasons. The first is that you have a limited ability to add other users. So you are stuck with these guys. You're stuck using these. All right. The exception is Cumulus Linux. But I'm going to show you some command injection to get around any sort of limitations that they might try to put on you. All right. So, show that in a bit here. And then the second reason this is a big deal is because these accounts are the only obstacle. The only obstacle to getting on the switches. All right. Which is, doesn't sound good, right? There's a little tired key logging in, the switch is yours. And then the network. And this is where all that begins, right? Easy escape to shell. Switchlight uses a wrapper. Type enable. Deepabash, you are at the shell. All right. Cumulus Linux, you actually connect directly to the operating system. So you don't need to do any escape. You are already at the shell. Now, not so less. Well, it has a very, well, very well done shell. However, Puppet's there. And Puppet can do dirty work for you. So Puppet will actually open up a back door. Which, of course, is not in the documentation. But you will find it if you unpack the firmware. Okay. Not always helpful. And once you have that shell, all right, you get into elevation. You immediately become red. And switchlight. Turns out admin is UID zero. So when you have that shell, you immediately are red. So, Cumulus, you basically have unrestricted pseudo access. So you are a road equivalent. Also not good. And now on OXOS, you start the back door with admin. So it's running under admin's privileges. And when you X back door and take a look at your UID, it turns out you are also UID zero. So it's also red privileges. All right. That one password, get that out of the way. And we know that compromising a workstation is trivial. Key logging is trivial. All right. Leads to full control of your network. First the switch. And then the network for unauthorized access. Add access, move access to high traffic, change traffic. All three operating systems. And then of course, compromise of the framework on unauthorized access because it gets you access to the flash. It gets you access to only, you can do whatever you want with it. You modify it and you have your firmware. Compromise. So seeing your network is one key logger away. Going to show you this here. Last year, you know, I did open source, right? Easy to get access to. And people always ask about vendor products, sorry, vendor products. Because, you know, you pay money, good money for vendor products and you want to see if the money that you paid it, you know, was worth it. So this year, I decided to make a point of using vendor products and actually run tests on vendor products and you pay forward and expect to be better secured, right? That's what they claim, you know, and in all cases they are. But we always want to make sure that they are what they say they are. So I started looking at these vendor products and these are all vendor products running on equipment, right? Network equipment. I started with big cloud fabric controller. And I logged in as admin, the low privilege user, looked through the commands and I found you a bash and I like the word bash that sounded good to me. And I went ahead and got the shell there. And if you're paying attention on the account slide, right, you saw what? That was hidden and disabled. I said they wanted you to stay away. They just actively discouraged you. Switchlight they want big cloud fabric to really take care of everything. So with that in mind, we have that hidden and disabled account, right? They don't want you to touch the switch. They don't want you to touch that. Well, there's no password on that. So how likely are you to change the password? All it does is keep you from logging in. Once you have shell, you can go ahead and switch over with no password. Then you are, of course. All right. So I'm thinking to myself, I wonder if I can do that on switchlight. So I logged in as admin. I had to do a little extra work there. I typed in enable first. And then I came up as wet and I was a bit surprised. So I checked up my UID and it turns out that I'm UID zero. So I like that. Then we have, yes. Switchlight, yeah. So we have our access, right? We have our access there as wet. So we can start looking around for flash, right? For flash devices. And we can look for one that's nicely named the course, right? Oni. And we see that with the privilege we need to write to it, right? With the wet. So that's good. We got that. Cumulus, all right. I want to show that you can basically, yeah, I call it pseudo it up, right? You can go ahead and add to shadow, add to shadow. You can change the password or whatever you want, right? Sudo, everything. And then milanox. I open the back door. I will tell you what that is soon. At the end there. Save it for last. Netcat. Very useful. And I want to connect it to the back door. Admin, obviously. And then I can see password. And there I am again. Admin at UID zero. And then an extra account. Maybe I was good to know other ways to get in. Extra account there. UID zero. Excellent. And I mentioned that with cumulus links there were ways to restrict your permissions. And since you are dealing directly with Linux, you can add new users. There are some manage action problems though. In the tools that cumulus links gives you for low privilege or less privileged users, they might escape any sort of solid limitations. So sorry, cumulus linux. They do know about this. Now, I don't necessarily, I don't come out with a lot of zero days or any sort of, you know, brand new, it's, you know, vulnerabilities in products. I like to celebrate this a little bit. I like to show off up there. Yes, big across the top there. All right, cumulus linux has several command line tools that they have set aside for less privileged users. I won't name them because they're tongue ties. They're meant to be used by low-perge admin accounts. It's just taking care of the switch. You go ahead and you enter your arguments, which end up being sub commands and parameters. Those sub commands and parameters get passed to CLCMD server. I can say that pretty well. And that goes in comparison against the Rosetta. Rosetta is basically, this is what's acceptable. And if it's acceptable, it lets it through. The problem is there's command injection. So it's filtering basically. It's command injection that allows you to bypass the filtering. So you basically get to have CLCMD server do whatever you want, whether it's a Rosetta or not, and it runs as read. So any sort of implementation to put on you, you can go ahead and sidestep those. I'm running on a switch again, right? So here we go. That is a switch with a license and hopefully I'm not getting any licensing agreements that might have signed. ULIS. So there it is running on a switch there. CLCMD server. I'm going to go and demonstrate this. I did bring VMs, right? I'm not as resourceful as the guys that apparently brought a whole safe out here. Yeah, I didn't want to try to ship or carry a top or X switch onto a plane. So we're going to go ahead and make sure I get my address for this. I think I already have it. I'm my low privilege user. I got lots of VMs running here. It'll take just a second here. Okay. Is it clipped again? Oh, it's off the screen again. All right. Here we go. I have a touch pad up here. Never good at it. All right. So I'm going to go ahead. I'm greedy. And of course I have my solicitations. I'm that low privilege user. And let's say I figure out the command injection. They have a patch for this already. So I don't feel so bad about revealness in complete detail. So I have that one tool that I have pseudo privileges for. And there is my arguments. What I've done is I've injected so that my second command, my injected command looks like it's part of the label. You have to do that. Otherwise it looks at that label, that first piece as command tries to process it. And of course it's not the reset and it fails. So I need to shove it all together so it sees it all as a label and it ignores it. It is a VM. There's no guts back there. So it does say unknown command but it's inject nonetheless. Now you'll notice there's no spaces there. And to get around that you go ahead and just make a script. Put a script in your own low privilege home directory and you put any kind of command you want in there. And what this one did, someone asked that before, it just adds another user with no pseudo limitations. You can just do anything. So we can go ahead and, yes, it's a little bit of a cliche. There you go. Then I have everything there. No collapse. We work all year for collapse. Come on. Thank you. Now I feel satisfied. I feel fulfilled. So you can bypass any sort of pseudo limitations that you would change or password change, you know, and then use your account. But obviously you don't want to do that. You don't want anyone to know that, you know, you're around. All right. I'm going to go ahead exit and then pause my VM so I can release resources. All right. My back on the screen all the way. Okay, good. So once you, of course, have the Republicans, this is once again a live switch, you can go ahead and you can look at the M2D devices and find Oni and then dump from the block device. And you have your privilege you need right there to modify it and put it back. Now, a big part of this is implications, right? There's been talk about this of course, especially on the vendor side, that this is a common problem, right? We have Grubb and other other issues with different types of devices. It's important to know that personal implications are greater. All right? If you have a firmware compromise on a single server, you have important data, but a single server. If you have a firmware compromise, a firmware compromise is now possible on your switch, then you have your network. All right? You have the network on. One server, whole network. One server, whole network. There's bigger implications. And it's important to talk about how this is done because they assume first off that it's kind of firewall, it's safe. So we want to talk about scenarios that really are very possible, that makes it not as safe as they think it is. So I'm going to play the goateed, you know, network administrator, and this can happen in a number of different ways, right? You can browse the internet, you can drive by download, you can open a bad attach. In fact, that's what I'm going to go ahead and do. And I'm going to be infected by a piece of malware, the PLC I put together called Big Brother. Now, Big Brother is going to do, it's going to affect, it's a Windows binary, it's going to infect the Windows system. All right? It's going to go ahead and it's going to key log off those fixed accounts, the ones that you're stuck with, right? For simplicity's sake. Very easily could have key logged off connections to the switch. Once Big Brother sees one of these accounts in use, key logs the password. As a network administrator, you're going to touch the switch at some point in time. You're going to log in and it just waits around for that. As soon as it sees you do this and as soon as you're done, well, he logs in. And he writes a Linux compatible binary to the switch's file system. Nothing's downloaded from the internet, carrying the secondary payload himself, right? He's carrying Little Brother, writes it out. Stars Little Brother's a back door. He unpacks the firmware, shoves Little Brother in there. I mean, what are Big Brothers for, right? Reps, you know, Reps, only back up, puts only back. Now, before he does this, he modifies only so that anytime only installs an error-coppering system only also puts Little Brother back. And that's persistence over and over again and how sometimes you install an error-coppering system, Little Brother keeps coming back. But it doesn't stop there. He is a big brother. He helps out. He pivots. He connects Little Brother as a back door and then he also connects out to a C2 server. The C2 server can be anywhere. This helps get past things like VLANs, ACLs, firewalls. What he does is actually he browsers. It's a reverse HTTP shell. He uses headers and he's capable of using a proxy. So it likes to blend in. And I relay commands between Little Brother and the C2 out in the great wide world. All right. So we're going to go ahead and demonstrate this. I've got an attachment here. I'm not a malware writer. It's a little bit stealthy, but not really. He does hide a little bit, but it's still not hard to find. So make sure he's running there. Then as a network administrator, you know, I've got an attachment and I move on with the rest of my day. At some point in time, I'm going to go ahead and touch that switch. It just works. This has a wrapper just like Switch Lite. All 10 commands to get to the shell. Actually what Big Brother will be escaping himself. So now, I'm going to double check this because you know, the dental gods and all. All right. So now, while he's done this thing, we're going to go over to C2. A little bit of a delay there. So it's going to fire up. It's how it works a little bit like a web server, of course. It's something browsing. It's going to go ahead and listen. And as soon as the connection is made, we'll see a prompt. It's a little bit slow in starting up. Hopefully, it'll call up soon. There we go. Can we get that on the screen? Okay. All right. Now, here we go. So Little Brother has been started. The connection has been made. And Big Brother has reached out to the C2. It's going to relay anything. All right. So there I am with my access onto Little Brother behind whatever firewall that is there. All right. I'm going to move around a bit. We're going to go ahead and just look at the switch real quick. There's some timing and weight. It's very patient. There you go. So we are in the red file system there. There you go. But things aren't forever. Not entirely. At some point in time, it's going to be noticed. What's this mysterious connection? You know, as an administrator, you see these sorts of things. Reinstall the operating system. Well, of course, right? Because we don't really fix things so much. Especially with devices like this, the desktop. So we just re-image, right? So we'll go ahead and re-image. It's an infection. That's what we do. So I'm going to go ahead and bring up Oni and then reinstall another file system. Another operating system. This one happens to be a demo operating system. I'm just going to do it sitting there. It comes up and we'll go ahead. I'm going to stop this because it's obnoxious. And then I'm going to get my command to go ahead and get that operating system. I did this because I didn't have access to the operating system at the time that I did this. But I made sure that it operated just like switchlight. I want it to be realistic. So, I'm just going to do a thing where I move on to fixing this sort of stuff. And then we'll come back and see Little Brother resurrected. So available solutions. Right? I've been characterizing this as poor choices. And that's what we have seen with the vendors. And this is about fixing that. So these solutions are addressing primarily vendors. And vendors, what they're going to have to do. They have hardware, install environment, their operating systems, agents. That's your remediation stuff, right? The vendor stuff is remediation. Or NERC administrators themselves, architects, right? Changes enterprise architecture from mitigation. Things that you're going to have to do in the meantime. Okay? Hardware, obviously. Trusted platform module. I'm sure we're going to add big switch networks, have these put in for most XA6 based switches. You know, there aren't any new power PCs designs. I have not heard of any coming up. So that's maybe difficult to do to get them added to power PC switches. But if we can, that'd be great because we want to start using TPM. We want platform security to make sure that any sort of modification is only modified that the hood fails. So we know something's wrong. They are also working on, so we'll get them on the hardware. The next step, of course, is to get them in use. And they are also working on getting this in the standard. So it's a bit of a slow process, but they are working on it. That's a cumulus network, so the ones that developed only and got it adopted. So they're working on it. Install environment. Remove tell net. It's insecure. We can use SSH, it's okay. And then with SSH increase key entropy and force a password change, for God's sake. I think we are all capable of remembering a password or using a password safe password. So that's one of the issues not to. You know, factor reset. Who here has done a factor reset? If your hands are not up, they're probably lying. We forget these things, we screw up. Factor reset. You can do that. Remove IPv6 and TFT waterfall. Nothing wrong with having DHCP provide the installation URL. We can protect the DHCP server. We can make sure it's the only one on it. They have ways of doing this. No reason why we can't use the DHCP server. And of course, a good one would be sign installations. The best one would be, but there's always problems with keys, right, and signing and that sort of thing. I understand there's problems there, but ultimately that would be the best solution. Operating systems. Before we hit that, let's check out Little Brother. Operating system has been installed again. Let's check out C2. C2 is capable of issuing a reconnect command. If you enter commands and you don't get any back, you know that there's been a disconnect. Luckily, the operating system, the recovery system has been installed again. I did that myself, so I know there's a disruption there. I'm going to go ahead and tell it to reconnect. Big Brother is going to close his end of the socket. He's going to reestablish his connection to Little Brother. Alright, so there we are. Thank you. We're persistent, even after an installation. Operating systems. Now, we obviously can't do everything. These are ways to harden the environment. I'm an ops guy, so we think hardening. As many as you can, make this platform more resilient because there's a lot riding on this firmware, a lot more than we've seen, we've got bigger implications for this firmware being compromised. First one, changeable names. UID accounts. Change the UID zero account names. It gives the ability to do that. Reduce privilege accounts. It would also be great if we could add users, which actually cannot add users. No, actually can add a user, but you have two types. Admin and monitor. What do you think you're going to use to configure and take care of the switch? Admin has the ability to change passwords. Change the password of the original admin. Get your UID zero back. Okay? Force password change. Don't allow those to hit stale. Don't think I mentioned this at Black Hat, but one thing I've seen in our cops is the password that they use, the shared passwords. They usually have a shared password, and they have a lot of equipment. They have a big team. I've seen those passwords go for years and not be changed. So force a password change. And then, of course, remove UID zero from the admins. Remove UID zero from the admins. And then tighten shell access. Switch Lite. I would like to see a one-time password. They have self-service portals for support. Why can't they self-service themselves to give it and make it a little more difficult, a little more resilient. Human Slanks, how about a wrapper? Also with a one-time password. Now, Malonox actually does a pretty good job of protecting that wrapper. There is a way to get shell access from the wrapper, but it was taking way too long for me to reverse engineer their code. So I took a shortcut. I said I was an ops guy, right? If you unpack the firmware sockets there and you can plug that into Bash and you're in. If you were thinking about maybe modifying or having a public modified password or a shadow add user, they thought of that. But they missed sockets. Agents. This is common. In SDN platforms. They're not using TLS. So they need to use TLS. Not only just use it for encryption, they also need to use it for ops, especially for mutual ops. And then of course there's a concern with certificates and key distribution, but you've already got DevOps there and SDN to do the heavy lifting. Have them also lift the certificates and the keys. Enterprise architecture. I say the management plan. We need more than the LAN. We need to get as close as we can. I understand it's difficult to physical separation. Try to get there, right? Get as close as you can. What's wrong with jump boxes? If you have been in network ops, they have 20 monitors on that damn wall. Network admins having two monitors on the desktop. Can't you remote in a jump box, maximize that in one of the monitors, use it just like you would on any of the other workstations. They're taking a little bit of news too, but it's going to up your security significantly. And then we have audit switches. We have audit. It was a painful process, but audit has good uses. They're your friends. Make sure their path changes. And then it wouldn't hurt to also hash only partition to make sure that it hasn't changed. I have a couple minutes left and I want to get to these last slides in a run out of time for any extra questions, but we can always talk afterwards. You're seeing something here and you're seeing vendors race ahead. And we want to talk once again about impact on security and keeping pressure on the developers, which is why we're here. We're here to freak them out, sometimes piss them off and talk about the difference that we're trying to make. Getting products features to market is important. We get it. We all get it. They're there to make money and they have to be out there first. They have something called best practices. I don't think you're reading them. I think you should. I think you should develop your own best practices. Start using them. Because you're not, we start this merry-go-round again. Every year at these conferences, we hack it and then you fix it. So now it's your turn, of course, to do cleanup. I'm here to evangelize. A lot of the companies doing this are either startups or small business and they know you have a limited number of people. One guy for security. One guy for security that's there all the way. How about some assessments along the way? Some are midway through the development life cycle. Maybe an assessment at the end before you release. The key is before you release. We would honestly settle just for that one at the end. All right? And when you go, try to sell it. Security can be a feature, too. All right? We want to make a difference here. We want you to learn from desktop and server operating systems. We want you to begin to harness these new platforms, dev ops and SDM. For the most part, what they're taking over, what they're taking responsibility for is defining the switch and configuring the network. They need to take over the entirety of the platform. They need to respond to the whole thing. Whether it's checking permissions and being responsible for permissions or they're checking audit advance or they're following those up and making part of the rest of the entire platform like a switch like to do that, right? Funnel up to big cloud fabric. Maybe a separate platform to consolidate all that. How have all that looked at? Or how about logging? You're checking logs on the individual platforms or you're already consolidating the logging so you have visibility there. So you are now taking responsibility for the entire platform, not just part of it. Logic probes. That's kind of my general term here. I have some electronics back on so that's why that comes up. You can check things. You can check state. You can make sure that the platform is in the state you expect it to be. You can, a good one, hash the only partition. It's a nice thing to do, right? Make sure everything is intact. So our final thoughts here. I think we are good on our time. It's a wrap up. The security of the network application is critical. We've seen has been neglected. And it's because they think the switches are safe. They're assuming that you're following all the best practices. They're assuming that it's an ideal situation. The solutions that you purchase and the different layers, right, are operating perfectly. It's a perfect world and they're catching everything. And that's just not the case, right? A single piece of malware could easily make the crossover from Windows to Linux. I did it right here. I'm not the first obviously. That pivot. To make a great pivot, you then are able to compromise switches and get that long-term assistance and this is immeasurable. Because you have an outside of the network and we are here today trying to avoid that. So hopefully we work together, we can't avoid that. Links. All the different products they are cool products. And that is the end. Thank you for coming.