 Morning everyone. So my talk is going to be about network monitoring. We free an open-source solutions. My name is Bennett Leong. I currently work for a company that sells network tests and measurement solutions but prior to that I do a dabble of Fabbit into network monitoring, the solutions revolving around this market as well as deploying hardware and systems tailored for high-speed packet capture. So I'm going to begin by asking a very simple question. How many of you have used and heard of Wireshark before? Great. Actually I'm done. So this is a very common list of questions that everybody you and I ask maybe some on a daily basis. Why is the network down again? Who's on my network? What devices is connected to the network right now? Why is the internet so slow or should I upgrade to a faster line? Questions that you constantly ask and if you don't have the information or the insights to help you make certain decisions or understands the problem you're stuck with constantly rebooting your routers, restarting your servers, things like that that it's just a stopgap measures. So we need to actually have the information as well as the insights for us to find out what the root cost is. So when we look at reasons why we want to actually monitor a network, we look at a couple of categories. So the first one is actually visibility and it's very true today because networks are getting more and more complex. Everybody is bringing on their phones, their iPads to work here. Everything is internet connected as well, the IoT. There's a lot more devices connecting to your network. Also there's a lot more applications moving to the web or the cloud. You used to store files directly on a file server. If you're a company working 10 years ago, you will notice there's a folder with your name created on it and that's why you dump all your files as a backup. Nowadays everybody pushes it up to Dropbox, to OneDrive and the data consumption, the bandwidth of all of these web or cloud connected apps has increased. And of course, you also have a lot of hardware and software. If you are in the network operations or if you are the IT guy of the company, you need to have a do an audit of what are the hardware and the software. So that's the visibility part. The second thing is about security. It's all about security. Before this, there was this talk about Honeypot. So unauthorized access to your network as well as compromised systems. You don't want your web camera to be part of a botnet. So you need to have insights to understand if your systems are actually properly secure. Are they doing what they're supposed to do and nobody is actually accessing it unauthorized. And finally, it's about troubleshooting. If I have a problem with my network, I need to find out the root cause and so that I can actually reduce the mean time to resolution. And for any companies or enterprise, you need to reduce the impact of the network performance either on your business or even on the user experience. So we kind of categorize kind of the need to monitor your network to these three categories. So before we jump into the entire network monitoring and then show you a graph of the diagram, I'm going to start with the analogy of a home network, home security system, which is very simple to understand. I think most of you would know. So we need to understand the fact that on the network or in this case, at home, there are many sources that you can actually collect to protect your home. You have CCTVs that allows you to actually view and capture images. You have motion sensors to help you detect if there's any changes, if there's any people walking past. You have door or window sensor detect if the door was open, if a window was broken. And you also have an alarm system that kind of ties everything together and allows you to actually give notification, call the cops or the security company, for example. So each of this solution provides a different perspective of the home. But at the end of the day, they all come together in improving security. But of course, giving you visibility as well, right of what's the status or what's the condition in your home. So in the same way, if you look at a standard, this is a very basic network where you have your router and your switch and access points. And then you have all your devices connected either directly to the switch or on wireless. So what do we actually monitor? So the most basic thing that we can monitor right now is raw data. And this one is very simple. If you have, I think a lot of most of you have used wire shark, you know this, right? If you just install wire shark, and you start capturing on your internet interface, you see raw packets. Okay, so it's easy to capture. It's hard to digest. And I think, you know, most of you know why as well, right? Then we look at flow data. So flow data is a lot a little bit more easier. It's generated by the network devices. So the network devices actually processes the information, the packets that's going through the device, and then it generates a flow of information. We gives you information like, you know, who's the source and destination, what port, what protocols, and some example is that flow IP fix. So if you have a switch or router that is managed, most of the time, you actually have this feature enabled, right? And finally, you also have machine data. So machine data are collected from your devices, from your servers. Okay, so they are usually application or function centric, right? They could be logs or counters from systems, devices and applications. So generally, this is how we see where can we get information from and how can we get a view of the network, right? So if we start at Wireshark, okay, it's all of you know this, right? It's a packet analyzer used for network troubleshooting analysis. Some of them use it for education. I know I attended classes when I was in university, and we actually use Wireshark as a tool to help you understand what's going on in the network. And you can see why, right? It's pretty straightforward. Okay, the software, as you run the capture, I actually just did this screenshot yesterday as I was preparing my slides, that you know, if I were to just type in, if I were to type in on my browser, right, to visit FOS, false agent, okay, what's going to happen is that I get a list of, list of packets. And at every single individual packets, I get a breakdown of all the data at each OSI level, right? So you have, you can see from the layer two, and then the IP layer, the TCP, and then as well as the HTTP, which is the content itself. So this is very straightforward, but for typical layman, or if you are, you know, if you want to get an overview of your network, it's not going to happen because you will be scrolling through thousands and thousands of packets, right? So then what you do is you look at flow data. Okay, with flow data, it summarizes, okay, and it gives you a table of information like this, where you have address A and an address B, right? What, what port is the, is the communication happening? What protocol it is? How many packets is going through? And what's the total size of this? And with this information, if you, so NTOB is actually an open source solution, all right? If you use NTOB to analyze flows, you can then create tables and graphs like this. So there's no graph shown in this screenshot, but you can actually then get a table like this, where this table shows you an active flow. And because of the summary information, and of course, NTOB is one of the very few organization or companies that still, that manages or maintain a open source DPI engine, right? As you know, if you look at DPI, a lot of the engines are actually commercial because it takes a lot of, you know, resources to actually build it. NTOB is one of the few ones that actually maintain a open source DPI engine. So that's something that you are interested in. You can actually go and look up on the website as well. So then you can actually look at active flows like this. And when we look, when we dig deeper, we can even look, we can even find out things like what application is it running? Is it visiting Google? It's the connection on Dropbox, for example, right? So that's flow data. And then you also have machine data. So machine data is when you have a lot of information coming from various sources. So just three basic examples here. The top is an Apache web server log. Okay, so if you, if somebody were to access my website, and this is the logs that I get to pull off from my web server. And then over here, you have something of assist log basis. So you can see that the format is actually different, right? The information that it's provided is also different. The nature of the information is different. You also have, for example, here is a Cisco firewall logs, right? Where again, you have typically just a date and, you know, some, some text. So how do we consolidate all of this data together, right? Just you can't go and look at each of this file individually. Or you could, but then it's very hard to actually draw the big picture. Okay, so one of the open source initiatives or, you know, the solutions that is widely used today, not just for network monitoring, but for many, many other reasons to process logs, right, is by the company called Elastic. And there is a stack that we call Elstack. It contains three of the solutions. Elastic search, which is the database as well as a search and indexing engine, right? You have log stash, which is kind of like the middleman that takes your sources from various different formats, put it in a nice standard form and stores it on the Elastic search engine. And then you have Kibana K, which is the web interface to allow you to kind of read from Elastic search and represent that into nice graphs or charts. So as you can see from here, that's just an example, right? It's not a direct representation. But this is a graph or a kind of a Kibana output of a web server. And you can now get easy representation of perhaps the status codes, what are some of the incoming hosts, request pages, etc. So this information allows you to really kind of consume the data a lot more easier. And because you're actually getting sources from various different machines, it also kind of draw a much bigger, clearer picture of the network. Okay. So other notable projects that I don't have time to go through, but I think most of you might have seen. So one is Cacti. So Cacti is very simple. It's all about counters, right? It has a polar that can pull any of the devices that either has an SNP or has an SNP interface, anything that you can actually get a counter value over time. And then you will store it in an RRD, which is a round robin database. Okay. And on the web interface, then you will just know gives you value over time and you can actually zoom in, zoom out, depending on how what's the granularity you set on the RRD database, right? SNOT, everybody knows this, right? They've been around for many, many years. I think established in 1998. They're still around. Okay, it's one of the most, I think reputable IPS solutions because they have constant updates as well as they constantly update their rule set to kind of give you a updated rule set of the track of the network, right? So that's SNOT. You also have Surikata. And Surikata is like SNOT. And there's a lot of papers and research out there right now that talks about SNOT versus Surikata. And one of the reasons Surikata came about is because SNOT was designed back in 1998, right? And there was a couple of limitations to that as compared to how sophisticated our network has become. So with Surikata, it's multi-threaded by design. So you can actually take advantage of all your new hardware that you have today, multi-cores, split it into various threads so that you can actually process more data, right? It also has additional features, the IP reputation, kind of give you an idea about where the IP address is coming, whether is it from a bad source, okay? And also kind of integrates a lot of other things. IPv6, for example, is built by default, is supported. So you don't need to enable certain fields or certain keys on SNOT, right? So that's Surikata. And finally is the Security Onion. I know it doesn't have a very nice logo, but most open source don't have great graphics, right? But what Security Onion does is that it's actually a Linux distro. And this is something that, you know, if you're interested, go and download it. It's an Ubuntu distro that kind of package all these network monitoring solutions together. So if you are in penetration or you're in security, you know about Kali and all those kind of things, it's, you know, the network monitoring equivalent of this, of Kali, right, for penetration testing. Okay. So now these are, you know, with all of these open source solutions, okay, for network monitoring, right? Why do we still not sometimes not hear about enterprises or companies not doing more monitoring the network, securing the network, right? And it's actually because of these four challenges. Okay, so one, it's kind of difficult, actually, if you don't know how to choose the right tool for the right job, okay, you know, your network is not as simple as that seven devices, right? It's a lot more complex than that. So you really need to know your network well in order to actually identify the right tool. You do require some level of technical knowledge to kind of deploy these solutions. It's you got to install the software, configure the interfaces, et cetera, et cetera. There's also fear of breaking the network, right? You don't want to be the guy skipping on dinners and working to midnight because you broke the network and you got to fix it before the boss comes in the next day, okay? And there's also policies, right? When maybe some companies, you cannot capture data and store it somewhere. And some companies you ask you require to actually store X amount of data for a period of time because of network audits. So these are, you know, challenges that if you putting a little bit of research and if there's a need for it, you can actually overcome and kind of learn how to actually overcome these challenges, right? So to wrap up, I am so on time, right? So to wrap up, network monitoring is important, right? It's getting a lot more complex right now. There's a lot more devices and everything. I can't stress how important getting visibility to your network is. We might not do packet capture across the board, but at least an understanding about what, you know, what devices you have, how much traffic, what's the pattern is important, okay? So to start, then what's the first way to get started? It's very simple. You identify what you need to monitor, okay? And then you associate the right tools to it. So at least that's the first step to get started. And there is a robust open source communities that manages all of that, you know, open source solutions that I've actually shared. And they have, you know, forums and wikis and a lot of tutorials and guides to help you through the process of installing and deploying a monitoring solution. So with that, thank you very much. Question and answer. Yes, hi. Now we have all this nice data in beautiful graphics. How do I start making sense of it, like saying? Is that spiked because everybody just went to the office to read the newspaper or is that spiked in the graph because somebody tries to take me? How do I set thresholds or alerts and stuff like that? So very good question. So that's why the first, well, because once you actually have that data, right, what you can do from that point on is that really understanding what the anomaly is. So if you only have counters, it doesn't tell you much if you have a spike in the, in the traffic, but you don't know what the flow information is, then you might have to do a little bit of guesswork. But if you have that spike and you can correlate that to perhaps a list of flows, then you can identify exactly, oh, maybe it's just, you know, Bennett streaming YouTube on the internet. Okay. So with that, you will then, you know, depending again on your enterprise or your network, what would be the right threshold? What would be the right threshold? It's there's no fixed answer. Okay. Because some companies have different policies, right? Some is just really, really used for monitoring. If one Bennett serve YouTube for, for the entire day at work, it's maybe not a problem. But if 10 Bennett does it, then that becomes a problem, right? So the idea of network monitoring is really kind of to empower you with that knowledge, that information so that you can actually take the next course of action. Okay. So you could, you know, using Antop, for example, actually set alerts and thresholds if it reaches a certain, you know, limit or a certain volume, sends an email because you won't be staring at that graph every single day, right? So you can actually set up thresholds and, you know, kind of give you alerts and notifications. But what you want to do with that data is, of course, So, compared to the enterprise products, how do you, how are the open source products, do they have some extra loss on the performance? Very good question. So, you know, so to compare it, open source solutions with what the enterprise solutions are out there, well, the biggest difference is actually support. And then that is really something that I would say, you know, a lot of people are willing to pay money for, okay? Because you, again, you know, one of the challenges is that once you break the network, you don't want to be the guy stuck in the office working to midnight, fixing it, right? While you're trying to get it back before the boss comes in in the morning. So, with a lot of enterprise, in terms of feature function sets, I can tell you it's really the same, right? You have a lot of good enterprise software out there that does what these guys do as well. But then the model with them is that they'll give you technical support, you got 24-7, you can ring people up and say, hey, I broke it, so how now, you know? So, but of course then the other part of it is that the the commercial solutions will then have more of a hardware support as well, right? Sometimes they will sell an appliance with the software, so you get that whole package solution. With open source, you know it's just a piece of software, they will tell you what their recommended hardware or platform is. Again, that's something that you will need to put in a little bit of effort and do a bit of research. So, that's the bit of a give-and-take comparison, right? So, so you should run it on the Edge device on the network? Yeah, so what you will do, so for M-top it's more on a wire capture, okay? So, there's a couple of ways you can actually deploy M-top. One is that you just put it right in between, so you can actually do it with a network tab, for example. Do you, I feel familiar with that network tab? Okay, let me just go back. Okay, so if let's us say I want to actually view all traffic that's going between the switch and the router with M-top, right? A network tab is a hardware device that connects the switch and the router together and at the same time mirrors the information out into my server or my appliance running M-top. So, that way I get 100% of the data, okay, by flows, by directional, into M-top. And then M-top will capture the packets, run its analysis, gives you all the pretty little graphs and the statistics, et cetera. So, that's one way. The second way that you can do this is that on the switch, if you're on a work group switch, you can actually do a mirror port, okay? So, there's a configuration, if you're on a Cisco and all those, you can actually create a mirror port and say that, okay, all the port, so the port 8, which is connected to the router, mirror it to port 7 and I connect port 7 to the M-top device and I'm only capturing one port, but by directional traffic. With that, you've got to keep in mind, if your port, if your link is 1 giga dps, right, and if you have total traffic of upload and download more than a gig, then you will have drop packets on the mirror port. You will not face that with the tag. So, there's some design considerations that you need to take into account. So, are there any known solutions for network rectify between internet ports? Oh yes, yes, well, if you want to use a Pi with the two ports, I haven't really got time to play, you know, around with a Pi, but if you can actually do a bridge, which what we usually do with the typical Linux configuration are, right, you can actually establish a bridge between two interfaces on the Linux device and with that, you can actually then very easily use it as like an inline tag. So, that's how almost firewalls or, you know, IPS solutions will work, where they are looking at traffic inline and so in normal mode, they will let the traffic flow between A and B, but if something were to happen, they can actually have the ability to actually stop the traffic. So, with an inline mode, you can actually do that. Okay, very good. All right then, thank you very much.