 Hi everybody. Welcome to my EuroBSDCon 2021 talk on SNMP, the Simple Network Management Protocol. I want to leave some time for questions at the end, so let's jump right in. What is SNMP? Well, it was created in the 1980s as a replacement for the Simple Gateway Management Protocol, and it wasn't intended as an intermediate protocol until something more complex was created. And they use a client server architecture, but they call the manager, the management workstation, the client, and the agent is something that runs on your server or router. The whole idea was basically that system managers should be able to sit at a central console and command their entire fleet of servers, start that process, close that interface, kick out that user off the system, acknowledge that error, dispatch this intern for coffee, black hole that spammer, whatever. Such a protocol should be both straightforward and highly structured. Everyone came to an agreement on the requirements because this was the 1980s, and there were very few people in everyone, and thus was spawned SNMP. So why call it Simple Network Management Protocol? Let's look at the word Simple first. There are seven commands, and you put them together like tinker toys and plug them in however you want. This gives us a protocol that's incredibly powerful and flexible, but it assumes a level of knowledge of the system that many people don't have today, and it bears all the scars of its history. SNMP dates from a time before a byte was eight bits. So it lets you invoke some of these ancient standards from the void. It grants you incredible system changing power, and it can destroy everything you've worked for. SNMP exposes the secrets of your systems, and if you're thoughtless, reconfigures them into unspeakable nightmares. It's like something out of an HP Lovecraft tale. Without the rampant xenophobia, but with all the alien system topologies. The topologies were in your computer all along. Your shallow human mind is blissfully incapable of perceiving them, and the more I worked with SNMP, the more I realized how disturbingly apropos that analogy is. So network, what's the network part of SNMP? SNMP is usable across the network, but networks have changed dramatically over the years. What was a network in the 80s is not like the 90s, is not like the 90s, and so on and so on. The primordial internet populated by university and nuclear missiles is nothing like today when you have countries involved in a shooting war with each other, sharing the same network with one another. Today, we usually think of SNMP as running on port 161 UDP, but it can run over TCP, TLS, SSH, whatever. So what's the management part? SNMP lets agents and managers exchange management information. What is management information? Anything I say it is, and it's structured anyway, I think it should be. I'm sure those of you with experience can see how this has already gone horribly wrong. And the information is designed in a management information-based, which is structured management data described in a MIB file, and the implementation should always match the description. Some data is defined by a standard, some by the vendor, and all of it's interpreted and implemented the way the vendor engineer thinks it should be. So protocol. SNMP comes in several different versions. Today, we have SNMP v3, which is extensible, it's encrypted, authenticated, efficient, flexible, etc. And that's been a standard pretty much for 20 years. SNMP version 2 came in several varieties, but version 2c won the standards battle. It's extensible, efficient, unencrypted, and unauthenticated, but pretty flexible. SNMP version 1 is a zombie protocol. It's extensible, it is not efficient, it's unencrypted, unauthenticated. Sadly, all three of these are still being deployed in new products today. There are brand new internet devices, especially IoT devices, that only support SNMP version 1. With all of this, why do we use it? Well, SNMP is kind of network management duct tape. It produces data you can feed to anything. You can extract gobs of information easily. You can send selected commands in a single packet, and you can extend it to include your own commands. And really, the reason SNMP gets a bad reputation is because of us. We misinterpret, we ignore things, we program badly, we're indifferent. You can structure your data any way you want, but you don't know what you want. And by the time you know it's too late, you have to live with it. So, let's talk about the management information base. It's a hierarchical tree structure. Objects have an address in the tree, and the address is identified by an object identifier, an OID. And each set of numbers here has a corresponding name that is defined in a MIB file. And you can point what they call an SNMP browser, which is a management program that lets you query an agent, and it will let you walk the agent of a host. Here, I'm poking at one of my little test network routers, and you can see all these numbers. I did not have a MIB file on this because I wanted to demonstrate. This is what the management information base and the OID tree looks like raw, but some things you can guess. So, another place where SNMP goes wrong is in performance. SNMP exposes everything. Sysadmins love information. Once you store a network administrator what you can get from SNMP, they love it too. One query, one UDP packet, easy. Well, yes, except each interface has 22 characteristics you can track out of the box. How many interfaces does your router or switch have? You have one server, great. You could dig a lot of information. You try to scale that up to a thousand servers, and that won't work so well. It is easy to dig for more information. And you have to watch just how much you're asking of the network and how much you really need. Another thing that people think about SNMP is it has no security. Well, in the early days when SNMP was first written, they had this idea of using a community as an authentication token unencrypted across the network. Again, the 1980s internet was not today's internet. There are lots of technical reasons why a community is not a password. Realistically, it's a password. There are times when it's okay to use that internally on your protected private network. Maybe you want to do it the easy way and just use SNMP version 2 communities. The real solution to SNMP security is SNMP version 3. And if we were together in meat space, I would look around the room and say, raise your hand if you remember when OpenBSD used Blowfish. There is a constant algorithm slide. MD5 to SHA to SHA 512, DES to AES, AES 128 and so on. We need exchangeable encryption algorithms in our protocols, much like SSH does. Many vendors offer SNMP with a variety of algorithms. So we have to be able to adapt as time passes. And SNMP version 3 is fairly good at that. To start with, it comes with 3 degrees of privacy. No auth, no priv, which is no encryption, no privacy, much like a community. Auth no priv, it has hash login and a packet and plain text data. And auth priv encrypts everything. And some tutorials would start you off with SNMP using the lowest possible denominator, saying you'll come back later and improve your practices. When everybody knows that once you've implemented something, slip shot, but sort of functional mostly, you never have time to go back and do it correctly. Especially when deficiencies like a lack of authentication and lack of encryption are invisible to management. So we're going to talk about doing this right the first time by using SNMP v3. This requires setting up SNMP v3 users, configuring the agent to notice them and setting those in the manager. There's special files for all of them. Most sysadmins experimenting with SNMP v3 go wrong because they leap straight to the agent and defer the issue of user management. This is exactly backwards. You must start your SNMP v3 with users before even touching agent access control. And really, in traditional systems administration, users are just an access control token. In SNMP v3, they are a unique combination of access control, authentication and privacy algorithms. And this means you have to look carefully at what everything supports. A manager may speak MD5 and SHA and SHA 512 for authentication and DES3DES and AES128 for privacy. But this one brand of router from Shakath Corp, it only speaks MD5 and DES. And this other brand of router Wellish speaks SHA and AES128. So you need a unique user for each of those devices. So think about it that way first. Which users can speak what protocols and which algorithms? And users are stored on the agent. The management client gets a copy of them. So when you're setting up a new user, you have to figure out what the username is, what privacy you want, what authentication algorithm and passphrase you'll use, and what privacy algorithm and passphrase. Make a note of all of that. Set it up on the agent and the manager. We're not going to walk through because SNMP v3 will take a little bit of time to do with net SNMP. And every device is different. Of course they are. But once you have it set up, the very first thing you do is test that the username works. If the user does not work, nothing else matters. And then you can look at access control. Here I'm looking at an SNMP.com where I've set, I've defined a community for SNMP v1 and SNMP v2c, which is the username is RW user. And then below that I've defined an SNMP v3 user. So once you have users and some sort of access control, you can make queries. Here I'm using the SNMP get command on the host gateway. And I'm querying a MIB that I want to know how long has this host been up for. And I get an answer back. Single UDP packet. Very fast. I can also group objects. An SNMP group is just a named lump of objects that are defined in the MIB file. Here is what they call the system group. We looked at this before without the names just numbers. And this is a bunch of basic information about the device. Where is it? How long has it been up? What is the host name? There are also tables. Here is an interface table where we run through the columns and then we have each row of the columns. So this host has 11 interfaces. There's going to be 11 rows in the table. The standard interface table is meant for a computer display. So I'm not going to show it because it's 310 characters wide for my little test router. But here is one of the short tables that says the network address to the MAC address. And we have a few entries here. This can be very useful to poke your network device or your server and see what do they have in their ARP table. So if you run SNMP walk on a host, you start off with that system group. There may be stuff in the MIB table below that. And to really rip everything out of the system, start at the object ID dot 1. There's also this concept of an SNMP walk. Now if you've played with SNMP at all, you may know that SNMP walk goes through the whole tree and gives you everything. But the SNMP walk is an SNMP version 1 tool. It is slow and inefficient. A walk basically sends a packet that says, give me the next object and its value. And then when it gets that, it says, great, give me the next object and its value. And you spend a lot of packets going back and forth. A bulk walk starts off by default with, give me the next 10 objects and their values. It only works in SNMP version 2 and above. You can modify this so that you can get more than those 10 objects. If you are trying to say, pull the routing table from an ISP border router, and it has millions of routes, a plain SNMP 1 walk may take over a day. And it will, you can tune your bulk walk so it takes less than an hour. So we've said what the MIB files are. What are, what's really in them? This is a full definition. And this is useful because the name is not always fully detailed. In the, when you do an SNMP walk, what you see and what that implies may not be what is really there. If there is a number, is this, does this number indicate that this is in an error state? Or how many times has this error happened? Unfortunately, these are written in academies. This MIB here, syscontact, is the email address of the person responsible. But no, we can't just say that. Now, if you're working with any kind of proprietary equipment or specialized network gear, there's the enterprise MIB. And this is an area where vendors and organizations can put their own data. Now, it's very easy to get one. I got one with a stupidly trivial form that I emailed, and they did not ask why. And I was fully prepared to say this is the most stupid useless MIB you will ever issue. But go ahead, get your own. These enterprise MIBs are all under one, three, six, one, four, one, and then you get a number under it. So, if I want to know what enterprise MIB something supports, here I'm poking at my little MicroTik gateway. And we see some things that they support that I really didn't expect. Yes, at the bottom, there's the MicroTik MIB, they have their own MIB, very nice. But at the top, they've implemented a Cisco MIB. They've implemented the UCD SNMP, that's a net SNMP MIB. They have implemented squid MIBs. I did not know that my little bottom of a barrel router supported squid, and I'm not about to turn it on, but that's kind of interesting. So, this is the good stuff. This is also the bad stuff. This is where vendors do things like hit this MIB to reboot the router, or to do a factory reset. Read those definitions carefully. Now, our BSD boxes generally all support SNMPD, the net SNMP agent. Free BSD and open BSD have their own agents, and they can operate and interact with SNMP, but this has become the industry standard for all of our sins. So, here we are, we'll talk about it. This is, you can configure the basic setup, they have a script for that, and this lets you set various alarm states for system conditions. Now, it doesn't say it's going to monitor those, it's just going to set an alarm if anyone asks. So, poke around net SNMP a bit. You can also issue commands over the network with set. This is great for your network management system, if you're very thoughtful and very careful. You can do things like turn IP forwarding on and off, if your agent has the permission to do that. The effectiveness depends on the agent, the host it's running on, and the MIB, so you can only set read write and read create objects. IP forwarding is a thing that exists in the system, you could turn it on and off. Once you've booted your UNIX box, even if you use host name, the host name doesn't really change. This also would require running net SNMP as root. You'd have to think very carefully before doing that. And different people implement net SNMP and SNMP in general to varying degrees of compliance with the standards. Here is the definition for the sysname object. And it says in the description that, by convention, this is the node's fully qualified domain name. Simple enough. I'm going to look at some machines here. I'm looking at my free BSD test, my Sentos test, my Debian test, and my little Microtech beat-it-up router. And the only one that really complies is Microtech, I'm embarrassed to say. So you need to look out for what does the standard say versus what do we really do. And to use this SNMP set command, you give the object and you'd say what kind of value you set it to, for example, the s here means a string, and then the host name. And the command output comes from the agent's response to confirm the change. Personally, I don't trust it, so I'm going to run my own SNMP get on that object to make sure that's really what it is. And again, what you can change depends on what platform the agent is running on. There's also a thing called an SNMP proxy. If you don't have direct access to a host, you can proxy through another host that's configured to do that for you. This is a useful way to interoperate, say, free BSDs, BSNMPD with NetSNMP. Agent X. You may have heard of Agent X at some point. This lets programs register themselves as handling parts of the SNMP tree. Many programs have Agent X agents. If you are running a database server like MariahDB or Postgres, or you have Apache or nginx, these all can fire up little SNMP daemons and feed into your main SNMP agent. And you can get management information for all of this. So that's a fast overview of some basic SNMP stuff. But let's touch base on something special to SNMPD and NetSNMP. You can extend it. The extend keyword in SNMPD.conf is a generic, flexible MIB for simple commands. If you hit this MIB, a command runs. You write the command. You have to be very careful doing this. But here I have a simple extend command called shogoth that runs echo runaway runaway. And if you tickle that particular object on this server, it gives you the command. It gives you what it put out. It gives you the exit condition, all of that. Now, that's trivial. Nobody cares. But you could say, list the contents of a directory. I've used this in production to monitor a directory. I couldn't get information out of any other way. I'm not proud, but it worked and it kept the client happy. And this let me look for things like core files and dumps. And did the temporary processing directory have too many files and we needed to slow the process down. Now, NetSNMP also provides other information about these commands say, you don't care what the files are, just how many? Well, this directory has 105 files. So that's an easy extension. But you can get very complex. Extend doesn't have a shell, but you can call a script. And scripts can do all sorts of things. You can now, if you edit the script and the script doesn't run, your SNMP walk stops right here saying, oops, we can't go any further because this object is fouled up. There's also something called pass through scripts that let you design your own fully featured Mib. The script has to handle set get and get next. There's examples and net SNMP.org. And I have written the most useless enterprise Mib of all time. This was an Easter egg in the SNMP book if you read it, but it's been a couple years since that came out. And so here it is. You can grab the Mib file off of my CDN website and query the table and get a list of every book that has my name on it. Now, there's other things you can do with SNMP such as this SNMP netstat command. You can query servers and routers in your network and get their netstat info, get their open stockets, their routes, statistics, whatever. And there are a bunch of commands like this SNMPDF to check disk usage, tracking changes, all kinds of this stuff. So a couple other bits of SNMP I'm going to touch on quick. Access control, there's this thing called vacuum view based access control. It's just a normal access control list. It has a weird language because SNMP. There are traps which are kind of like logs except they're very specific and very highly configurable. And when you're working with SNMP, I'd encourage you to remember that whatever you choose to monitor will not be the right choice. So good, we have enough time for questions and discussion. I'm just going to go through, yes, this is based on my book SNMP Mastery. And if you want the really special occult version, it's available as the Network Namakan with extra art. And I'm now hopefully big, big blue button has let me connect and I'm available for questions, answers and snark. If not, thank you.