 So, over the last couple of years, I've become a bit of a network nut. That's my homelab. It is one rack in a mirror, but hey, I happen to have a mirror wall in my office. The amount of glass we have in our house is insane. We assume the previous owner has owned a glassworks. It's, or something, it's crazy. But there's everything there from a tiny Cisco to a fairly decent multi-gigabit Juniper at the top there. No, but I have a 32-put terminal server which is getting a bit full. There is 10-gigabit Ethernet though. Anyone. So, how many people here have switches, routers, firewalls, anything? How many people have a backup copy of the OS image? That's pretty good. A copy of the config when you deployed it? A copy of the current config? That's really good. Do you have a copy of the actual current config, not just what you think it is? And do you get notifications when the config changes? Right, printing to the converted, that's awesome. And do you have automated documentation verification of your configuration and deployments? Not very many, good. But neither do I for at least that last one. So step one, sort of heading down this path, before you even deal with the config, deal with the OS. But pretty much all these devices, the OS install is one file or at least available in table form. For most Juniper and Cisco devices these days, it's a table, but you only need to keep the table. So credit file repository, my Junos repository is about 10 gig, my iOS repository is about 120 gigabytes at this point. iOS may be smaller, there's a lot more versions of it. So have a file server, it may be worth replicating this, or at least copying useful elements to remote sites if you ever need to recover devices firmware when you may have just loaded a version that has say no routing. This has happened. Thank you Juniper and your IPv6 upgrades. Because you really will be cut off. In the same vein, it might be useful having printed copies of how to do upgrades from backup media. I pretty much have the Juniper version memorized at this point. Most stuff actually, fortunately, doesn't require you to break into a Unix shell and mount random devices. So now you've got the OS, next, config. Well you can start out by every now and again, you copy config to a file repository or a wiki or something, you're going to miss changes that way. You can store an archive of the config every change. That's not a bad thing to do, because at least you might know the state of things and if you make a state before change, after change, if you do that every time you can then sort of see things happen when you didn't expect them. But you're going to miss unmanaged changes. You're going to miss, it was 3 a.m. and the change had to be made then, sort of thing. You can do push-based config changes on a configuration change, then the new configuration repository. Problem is this has silent failures. If it knocks the device offline, if an admin, say, turns off the automatic pushing because they deliberately don't want you to be notified because they're doing something dodgy like advertising their personal blocks out the company transit. I haven't actually done that one, but only because I was too lazy to. It also has somewhat limited support. Junos can do this. iOS 12.4 and equivalent versions in the other trains because Cisco like being stupid with version numbering. But there's usually very limited version control with it. So you have the last config that got uploaded. That's not always the most useful thing. Doing the pull method is actually much better. You have a central service that pulls the config on schedule. It can actually tell you on failures when a device is down unreachable, SSH has crashed on extremes, which is yet again. And it should also catch, in most cases, admin manipulation. And if you've got some networks that's critical, some networks, that's not so much. So the tool, largely of choice for doing this is something called ransom. What it is is a horrible mash of pearl, tickle, and expect. But the thing is it works better than just about anything else. URL there at the bottom. There'll be some more references at the end. What it can pull? IOS, just about all the variants. Junos, Greenos, Foundry, Ironware. Foundry is, of course, now brocade. The ethernet side of brocade, sorry. The ethernet side that's not converged of brocade. Extreme Extremeware, it doesn't really do extreme XOS unless you hack mercilessly at it. It can do quagga. So if you're running Linux-based routers, it can manage the config for those. Honestly, just use your existing host-based config versioning for that, probably. It can do a bunch more. If you have transmission gear and other larger carrier-type stuff, look in the distribution. You may be well-coded for. Your gear may end up being similar enough to something else that you can just modify. So what it does, Launch Firecron job, it uses something called C-Login, where the C is replaced by J or N or X-type things for other manufacturers to log in via telender.sh to devices, and act as a semi-obstraction layer for commands. You give it a little set of commands. It'll run it and come back to you. It runs the commands, stores the output in files, and then commits them to a VCS. By default, that's CVS. It can be Subversion. Someone's probably got a patch for Git out there. And then it emails Diff's to an alias. It also covers cases where routers come and go on from the configuration or if routers are unreachable. Essentially, unless your admin is being really deliberately trying to hide themselves, you'll probably catch their changes. Honestly, the easiest way to avoid sending out changes is to change the alias to send to you only for a change run and come back. Anything else will probably be caught. The logins. I need to recheck this, but I'm pretty sure it can only do password only. Doing SSH is probably doable. OK, so if the device supports SSH kids, you can probably make it work. Excellent. For iOS-type devices, with enable passwords, it also handles those. What you can do on devices that support it, like Koff, Juniper, is use login classes so your Rancid user has permission to view the entire configuration but nothing else. To be fair, you're probably giving the root password in some form, so it's not huge protection, but it at least means that people have to go to some effort to completely screw your network over. Now, so this obviously brings a huge amount of benefits. You get the configuration history. You also get hardware inventories. Rancid, by default, will do a dump of the environmental and system details. So your system crashes and the idiot forgot to update the hardware wiki or the sport contract wiki. But fortunately, all you needed is one Rancid run from history. That's that line card. That's that serial number. Hello, Cisco sender truck. You also get file system details. One of the interesting things that's happening with a lot of routers these days is things are being done in the file systems, not in the main config. Policies in a bunch of devices, commits groups, extensions on junipers. I'm sure there's a million other things are now no longer in the main config there in separate files. You can dump the file system, and at least you know if files have changed. You may not cat them all, but you at least know if file has changed. But the thing is, you've got the config and all this. So you can actually extract a bunch of information. So I've got a script for example that gives me all the version inventories. I take manufacturer essentially OS, which is relevant for the extreme gear, the model number, the version, the model number, of course, being relevant for Cisco gear, where the model number means you should be running completely different versions of code, and post name. This is all accurate as of a week or something ago. That's a good start, because then grepping, say, if I just want to know all my Junos devices that aren't running 10.0 at least, I can just grab it. That's a really great way to catch devices you missed. You might do basic config stuff. For example, this one, host name, a really dumb one. Is it pointing to our DNS servers? Is it pointing to our NTP servers? Is it does have the right SNMP community set? Very, very handy. Because while, say, your access switches might not be important in many environments for them to have the right clock, every now and again, you're doing something, and the ability for the clocks to be right is wonderful, or the ability for them to have DNS when you need to upgrade them. I've got another script. I won't show the output from it, because it's about 40 screen fools. A simple script, you go through all the interface addresses, all the loopback addresses, all the NAT pool addresses, for example, do a reverse DNS look up and display them. This one, you pretty much need to hand verify in a lot of cases, because there are often cases where you're not allowed to put the interface ID address in there, or a carrier has allocated the reverse DNS and won't change it if you ask. But it's still very useful. You will catch things that are broken. You will catch that you still have reverse DNS from a Cisco from 10 years ago, not the boundary that you have in today. Another really awesome one is HA configuration. It seems like load balancers and routers where you may deploy two identical devices, except for, say, four lines of config. Right a little bit of said, to isolate those four lines of config, diff them, and they should be identical. I have one that I use for foundry server irons. This is excellent, because it can be really helpful if someone does something stupid, or you think you get it right or you don't, or, hey, we're running a slightly different subtle subversion on one device versus the other. Very helpful. The one I like the most, though, an automatic network diagram. Not as pretty as one you draw with Vizio, but it's guaranteed to be accurate because if it's wrong, you don't have information about what's wrong. Therefore, you don't know the device. It shouldn't exist. Kind of being a trick there. Mine basically uses the same tool as the reverse DNS one to extract interface addresses, IP address, et cetera. Graph is. That's pretty much unreadable up there, but that's a network as it was someday, sometime. Obviously, Graph is, could probably use a better sort of rights and their style optimizer. This is the least broken one of the lot, at least in Debbie and Sable. So there are tricks you can do to try and break out, break it out a bit more and group things. This was as good as I got before I said good enough. It does somewhat randomly. It'll generate a beautiful diagram, somewhat randomly it won't, unfortunately, as you add and remove devices. You can also, of course, generate config for other tools. You can generate the config to add to Nagios, so it can go, it's a Juniper, monitor the system alarms. It's a Cisco. You add it to MRTG, CAC, Diamond, whatever, for your interface environmental monitoring for your historical graphs. Or you could auto generate DNS zones if, say, you don't want to do them manually for whatever reason. You can also use pushing. Because you have the tool to do the logins and send commands, you can use that as the basis for pushing new configurations, template, from template system. Now, I mentioned on discovery earlier, finding new devices, adding the empty management tools, have a serious think about whether you should do this. The reason I generally think you probably shouldn't is because there's often, especially in larger cases where you might have BGP and similar protocols in play, if you already have to manage multiple devices to add a device, having it, if you're having a config push out, it's auto discovery is something I think you should really think about before deploying. So once you've got your current config, the next step is starting to remove the human element of making changes to the config. The first simple step, have a wiki page or something similar. Most places generally seem to standardize on one vendor for routers, one vendor for switches, one vendor for Wi-Fi access points, et cetera. Keep your config templates for them. Persiscos, things like setting the escape character to not require five hands. Routing protocols, if you use ISIS, you might want to set the metrics. Well, you want to remember to set the metrics. You need to set the net address and the loop back. Setting SNMP, if you have extra read-only groups for partners or external users. A tool that's come out relatively recently designed for this quite specifically is Net-O-Matter Config Generator, which is essentially a template-based config generator. It's somewhat more generic than just network devices. It's an ARB core. It's essentially common to many systems. So Puppet users will be wonderful. Steve Walsh was telling me yesterday that someone's got a Puppet class for doing Cisco switches, things. Whatever, I love Puppet, but I'm still not going to use it for anything it has no idea about. So generally, you give these things templates and you give them a device information database, which contains every bit of uniqueness for that device, and then you merge them. So you manage bunch of things. Etsy Network Interfaces, Cisco Config, Policies, VLAN databases, DNS zones, you get the idea. Doing templates is fairly easy. You choose a device. You write a template that outputs the config for that device. For our stuff, remember that you may will have to have no commands to disable defaults. Choose a similar device, manipulate it. Choose a new device, manipulate it. Relatively easy. It's slow. The first device will take you a couple of days. The second device will take you another day or two. Eventually, you'll get a great template and you can add another switch by adding a couple of lines of config and everything's wonderful. It'll take a while. And then you move on to your firewalls and it'll take you a week and then a couple of days, et cetera. So there's another new tool which has come out recently. It's called Notch. It's a CLI abstraction layer written in Python. It's kind of primarily out of Google Sydney, but that's more because that's where a couple of developers have been. Some of them now aren't, but primarily out of Google Sydney. What makes it cool is two tools that have basically been developed. Punk, which is a Rancid replacement, and Mr. CLI, which is cluster SSH for routers. If you've never used cluster SSH, cluster SSH, list of machines, it opens N terminals and you can type sequential intern. This is more an in-order tool. It's kind of cute. If you're actually seriously looking to deploy any of these tools, I can strongly suggest those two talks, the two talks from Nanog there, and Notch, there's a little bit of information around. Hopefully we'll see some stuff come out in more detail about it. And I think I've got about three minutes for questions. If people have anything for anybody. Are you releasing any of the scripts or tools you've written for your own use so far with SchoolsNet? So the short version is some bits and pieces of it are in a case where I could release them. Some bits and pieces are in a case where they end up being so highly side dependent that I think giving people a list of the sort of things they should try and write tools for is very useful. Honestly, I've got a bunch of Nagios based stuff that I think is far more useful to release. But yeah, so the short answer is not really, but if people want to talk to me and want specifics, I can probably help them. Back there. The ideal really is something like configuration management system for network devices. We used to have a system like that called Build Config, but it required network people to be able to program in Perl. And since that suddenly, they didn't make Perl's requirements. So now people are doing things manually. Do you see these systems actually moving towards sort of a total automation of configuration? So the short answer is larger networks pretty much already do. It's a case of the thing you find with most things in IT, the larger the site, the more automation they have simply because efficient circuits. Ours is very large, diverse, and increasingly more manual. Yeah, it all depends on the people you have running it. If you've got people who somewhat comes from a system's background with automation, they'll generally add automation. Some people love playing with routers and digging around with individual configs. I enjoy it. I'd still rather automate 500 routers than to update them all by hand. It is somewhat of a mindset thing. Similar to the DevOps talks earlier today and yesterday, if you have people whose goal in life is to be on call 24-7 getting paid or some money to never get called, you will, it is highly like you will have a wonderful setup. Might not have all the features of the other guys' ones, but it'll be up. The same thing goes for the networks. I love adding features to my networks. Now that I have a large toy network, I can stop doing it to our production network in the middle of the day. And I think we're at time. One last question. One last question. Anybody else? Oh. Thank you. Oh.