 Hello everyone. This is Rashid Feroz and today I'm going to talk about how we can achieve a high level of network inspection using VPC traffic mirroring, which is a feature provided by AWS along with Surikata and Z. So Surikata and ZCOR open source network security monitoring tools, which are very well known in the market and VPC traffic mirroring is a feature provided by AWS. Now, to give a context before we begin the stock, even though the architecture that we have will be showing in the presentation is using VPC traffic mirroring, which is a feature provided by AWS. But this whole setup can be cloud agnostic as well, irrespective of whether you are on AWS or Azure or GCP or on your own data center. Moving ahead, so this is about me. I'm working as a security engineer at CRED and I've been into cloud security since a long time. Before that I worked as a security consultant for multiple banks, financial entities and government agencies across the globe. So this was about me. Coming to the agenda of the talk. So this is the agenda of the talk for the session and we'll start with the problem statements that we have faced. And while trying to solve those problem statements, there are certain very specific pain points using a traditional approach that we will be talking about. And then I'll talk about VPC traffic mirroring and then we'll go into the implementation architecture, then we'll talk about why we have chosen Surikata and Z and then we'll conclude the whole talk. So these are the problems we started facing since last one year. Ever since COVID happened, the complete workforce for every organization started working from home and the only way for them to connect to the office infrastructure is through VPN. Now VPN has to be kept in a public subnet or in a DMZ. So VPN becomes a key target for attackers since the only way inside to the infrastructure is now via VPN. So what we have seen is that companies and organizations spend many resources and integrate commercial appliances and firewalls, ZDoS protection for the perimeter security. But when it comes to detecting threats on the internal network, I guess we are still lacking because it's kind of difficult to set up infrastructure to detect internal threats, which we will discuss about. So the major pain points that we have is that there is a lack of visibility over internet traffic and lack of visibility over VPN traffic as well. So in the last year 2020 and 2021, we have seen many, many attacks happening on VPN instance and there were zero days seen in the wild as well. For example, in the US federal agencies, VPN got hard and they got access, attackers got access to the internal network. And then there were restricts vulnerabilities seen in the wild, which was a zero day at that time. So coming to the next slide, these are some of the screenshots where you can clearly see in since last one and a half year, there has been increasing number of attacks in the VPNs in almost across every industry. So I'll talk a bit about when you approach this problem statement and try to solve it using a traditional solution, what kind of pain points you might encounter. The first thing is the log growth. If you try to ingest NetFlow logs or your network logs into a same solution, the log growth is huge even for a small scale organization. Now ingesting all those network logs into a log analysis tool or same solution will not always be feasible, especially for a small to medium scale organization because we're talking about huge log flow and it could easily go into terabytes or petabytes. So we have multiple issues that we have faced. For example, correlation and analysis of such huge data sets is a nightmare, especially for small security teams. So let's say if an incident happens, the typical response is to do a threat hunting or try to analyze what happened by analyzing the network logs. And when an event happens, if you try to analyze terabytes and terabytes of flow logs or network logs, it would be kind of very difficult or almost impossible to find out in a short period of time as what exactly has happened. Because VPC flow logs generate multiple log lines for simple packet transmission. Even for TCP handshake, it will generate one or two log lines. So this is a meme that I made and it stands very, very true for again small to medium scale organizations where the security team is small and the log growth is huge. So let's say if you want to do a log analysis from your last 30 days, we are easily considering around 30 to 50 terabytes of data and again one terabyte per day is an assumption I have taken for a small scale organization. It can be less or even much more depending on your organization and network traffic size. So this can be a very daunting task for a stock analyst or people who implement same into their org to do such a correlation analysis on such huge data set, because there are the scaling challenge is huge. In order to scale your same solution or log analysis tool to such huge scale is a daunting task. So then you have a cost problem when it comes to commercial same solutions ingesting such real data and then you have resource branches. So there are a lot of issues when it comes to doing threat hunting or detecting threats via analyzing network logs. So the question which comes to our mind is, is there a solution which can be worse deeper network visibility and help us to detect threats within our internal infrastructure with a minimal and easy to use setup. So here is the solution introducing VPC traffic mirroring along with Surikata and Zeke. Again VPC traffic mirroring is a feature provided by AWS which we will talk about and Surikata or Zeke are open source network security monitoring tools available in the market. Before we go ahead let's first understand what is VPC traffic mirroring. So VPC traffic mirroring is a feature provided by AWS using which we can copy network traffic from an elastic network interface of an Amazon EC2 instance to a destination. The destination could be an EC2, destination could be a load balancer, etc. Now there are three main key elements of VPC traffic mirroring. One is source, another one is filter, another one is the target for the destination. So if you see in the picture below, the source A on the left side is actually sending the mirror traffic to target D. So how it works is, whatever traffic is flowing through source A whether incoming or outgoing, what VPC traffic mirroring will do, it will copy the exact traffic, encapsulate the whole traffic into VX line packets which is an encapsulation protocol and send that traffic into a target D which is our mirroring target. And target D could be running these tools for example Surikata and Zeke. So they could be multiple sources and multiple destinations in this process. So let's talk a bit about how the whole thing works. This is a very simplistic architecture that I have shown here. It might not be this simple in a real-time scenario. This was the sake of understanding. On the left side, this is the source and in the middle is the destination for VPC traffic mirroring. So let's say we want to monitor certain network interface or certain instances. So we will implement this VPC traffic mirroring where we will choose those instances or those load balancers as a source. And then VPC traffic mirroring will forward all that traffic to a particular instance or a load balancer of our choice. And on that particular machine we will be running our NSM tools. So these NSM tools will generate some logs which we will be pushing into Elasticsearch or you can push it to your favorite same solution or log analysis tool which is used in your organization. So coming to why do we even use Surikata, what advantages it has. So it is a very powerful open source network security modeling tool and it has capabilities of scaling and high performance where it can handle gigabits of data per second very easily. So the way Surikata works it listens to that interface. So on that interface the VPC traffic from the source is getting mirror to destination and Surikata is trying to listen to that interface capturing on the packets and on the fly it does all the decoding and parsing job for you. So you don't have to figure out exactly what protocol is in use, what files are being transferred. It will do all those things for you. So all the applicationally decoding protocol or decoding it will do it for you and generate the logs. Another very important and interesting feature Surikata has that by default it supports tunnel decoding which is VXLAN or GRE. So VXLAN is an encapsulation protocol used by AWS for VPC traffic mirroring. So when you mirror the traffic from the source to destination, AWS encapsulates it in a VXLAN protocol and then forwards the traffic to a destination. So Surikata has this support for automatic VXLAN decoding. So that the source in the destination IP will always be the same as what it is seen in the source. So it also has a HTTP parser which can parse your HTTP request and analyze all the HTTP headers and get requests. Then you can also extract files transferred over your network using Surikata. Let's say if somebody is trying to download a malware via your network, internal network, what Surikata can do is that it will identify that a file is being transferred. So it will save all those packets associated with that file and generate the file and system itself which you can forward to a sandbox for further malware analysis. So this Surikata works on a rule-based alerting engine where the rules are written to detect any sort of anomalies. So by default open source rules are available, but you can also use commercial Surikata rule sets from the market which will be paid. So the way it detection logic works is you can write a lower script and it's kind of very easy to write your own customer scripts for detection. And you can even use MISP to match the IOCs against your DNS request or URI or other fields. The best part about this is that it generates the output in JSON. So you can easily pick up the JSON output and feed into your elastic search or whatever same solution that you are using for visualization and alerting. So coming to the next part which is why do we want to use Z? So Surikata and Z are two different tools and both are categorized as network security monitoring, but there's a primary difference between Surikata and Z. Surikata is primarily a signature-based detection tool, but Z is more of a very powerful network analyzer. So what it does, it converts all the traffic captured via Z into a series of events. For example, let's say if I'm transferring 100 megabytes from a social destination within my internal traffic, what Z will do is that instead of generating 1000 loglines as you would see in VPC blogs, what Z will do, it will combine all the spackets into one simple logline that this is the source, this is destination and this guy actually transferred 100 MB over your network. So you can send a Z output to your elastic search or your same solution and it is very intuitive and very simple and condensed form of I would say flow logs which can be used for threat hunting. So it gives you a more perspective. It's more easy to do an analysis and it's more easy to piece together what's happening in my network. So how have we solved for this? We have used VPC traffic mirroring in AWS to forward traffic from our VPN to the chosen destination. Now Surikata and Zeke works independently and they consume the mirror traffic. So Surikata analyzes the network traffic and detects any attack patterns or signatures that could signify a network intrusion. So whenever the traffic matches with any attack patterns which is defined in the Surikata rules, then it raises an alarm. So Zeke does not generate alerts on its own. Zeke captures all the traffic and generates key insights on those traffic. For example, let's say if you capture 24-hour traffic using Zeke, it will generate logs for every single protocol SSH, SMB, SNMP or just the connection logs. So it's easier to go through Zeke logs than comparing to network logs. Now you can use Firebase for LogStash to forward the logs generated by Surikata and Zeke which is in JSON to forward it to your elastic cluster for visualization and analytics purpose. So this is more of a realistic architecture that you might implement in your organization. Again, it depends how it is set up. So if you see on the left side, we have an instance T3 which is the VPN server that we're using and it is in a DMG zone. So employees will be connected to this VPN server. So what you can do is you can use VPC traffic mirroring to forward or mirror the traffic from VPN server to a chosen destination. And your destination could be an EC2 instance or a load balancer. In our case, Heritzen network load balancer just forwarding the traffic to Surikata instances. Surikata has their own rule engine which detects attacks by analyzing the traffic. And then using FileBate, you send the logs generated by Surikata to LogStash and then to elastic cluster. So what are the use cases? So the use cases could be that one is that it detects attacks of the VPN instance from outside and then it can also detect malicious traffic flowing through VPN. Let's say if some employee gets compromised and that attacker is trying to enumerate our resources through VPN or trying to do some attacks via VPN traffic, then we would be able to detect that using this setup. It might also detect zero-day attacks on the VPN instance because if the payloads are not encrypted, we can easily see that somebody is trying to do some attacks on a VPN instance. And using Zeke, you can do another ton of things. For example, you can detect any mass data extraction from your system and using Zeke, it can even generate even logs which you can consume for threat hunting. So the way I see it as working is let's say in your whole environment, there are certain very key instances where you process and store sensitive data. So what you can do instead of capturing the whole traffic, you can use UPC traffic mirroring to mirror the traffic from those key instances or those key load balancers and then forward it to a Surikata or Z cluster and then so that will give you a better understanding of what's happening in your network. So it's easier to make sense what's happening in your network if you use such tools. So these are some of the alerts generated using Surikata. The first one, if you see it's a Netgear remote command institution as detected by the signature in Surikata. And this attack has come from China where somebody is trying to do a remote command institution if this succeeds that attack might be a reversal, which is detected by Surikata. And then on the bottom, you will see different alert signatures detected by Surikata and what is the IP, what is the location, what exactly, what category it falls into. So this is just a visualization of what Surikata is capable to do. Again, it works on the signature that you have defined, the rules that you have defined within Surikata. So if there is an attack which does not match the rules, then Surikata might not detect that. Coming to Zeke, so by default Zeke does not generate any of the connection logs, I mean alert events. So by default Zeke does not generate alerts as compared to Surikata, but it generates log files after analyzing all the network packets. So you can consume those log files and build your own use cases. So your use case might vary from my use case depending on what you're trying to achieve. So here is one of the use cases that we have achieved using parsing Zeke logs and that is mass data extraction using VPN. So if any employee is trying to extract mass data, you can define what do you mean by mass data, it could be greater than 100 MB or it could be greater than 500 MB from your VPN instance. Then I will immediately get an alert. So this person from this IP address has extracted this 100 MB of data. And when you see recent data breaches where terabytes and terabytes of data were leaked from the systems or data warehouses. So this can be very useful when it comes to detecting any sort of mass data extraction. Now it could be legit employee or it could be an attacker as well. So what are the limitations? So for sure Surikata can't detect attacks on encrypted channels because it's not possible to read what's happening on encrypted channels. But again, it depends how your setup is and where you are trying to listen to the traffic. A lot of times what I've seen in many organizations is that the SSL connection ends at the load balancer itself and after the load balancer the traffic flows in HTTP. Again, it depends how your organization has set up the whole infrastructure. Now if you listen tap onto the traffic, which is moving within your infrastructure after the load balancer, then we can easily figure out what attacks are happening on your network. But again, even if your internal traffic is encrypted, it will be difficult to detect attacks, but it can give you key insights of what IP it is trying to do. So there are certain alternatives to VPC traffic mirroring if you're not on AWS. If you're on GCP, you can use packet mirroring, which gives you the same abilities. If you're on Azure, you can use virtual network tap. If you're not on GCP and Azure, if you're on a custom DC, you can always use port mirroring and span for achieving the same as what we have done. So coming to the conclusion, what we just talked about in this session is that we generally organizations tend to focus a lot more on the external angle from a perimeter security angle. We might put firewalls and different appliances on the perimeter, but generally it becomes very difficult to get a overall bird's eye view on what's happening in your internal network. And in order to do that, what is the general practice is that security teams tend to consume all the network logs into a same solution. But when the log size is very, very huge, it goes into terabytes and petabytes. It gets very difficult, especially for small and medium security teams to ingest those logs into a log analysis tool from a scalability perspective or post perspective or resource perspective and make some sense out of it. So what we have done here is that using suricata, Zeke and traffic mirroring, you can get a sense of what's happening in your network, easily detect anomalies and also generate more eventful or stateful network logs using Zeke. So I'm not saying that you should not use network flow logs for threat hunting or you should not ingest network flow logs into your SIM. Even we ingest VPC flow logs from AWS to our SIM solution because it serves certain use cases. So I'm not saying it's a bad practice to ingest network logs into your SIM solution. But if you want to have an easy to use setup with a minimal infrastructure, then this is something you can easily configure within a day or so. That's what it was about. Thank you for attending the session. So thank you, Rashid. That was very informative. Is there anything you would like to add on to that talk or any updates that you all have made in the architecture, any changes? Anusha, I don't think we have made any changes, but I would want to talk in general about traffic mirroring. So there are a lot of other use cases I have seen in the last one year using traffic mirroring. So what I discussed in the talk was majorly around how to get more insights, how to detect security threats, right, using Surikata and Zeke. One of the most interesting things we did was we had a lack of visibility on who's extracting data from our environment and using Zeke. We got to know that basically which employee it could be an employer, the employee might be compromised. So what employees downloading what amount of data in MBs or gigabytes from our data stores. So given that we already have very strict access control measures, but still there could be some non confidential data which might be important to us as well. Or it could be some sort of confidential data as well. So that has served the purpose really well. We have great visibility on who is doing what in terms of data extraction. Other use cases I would want to talk about is I have seen this use case in forensics as well. For example, if an instance is compromised and an instance is trying to communicate to a command and control server, right, on a traditional ransomware or a botnet architecture, and you want to analyze the traffic which is flowing from the instance, you can isolate the instance, remove it from the production, and you can implement traffic metering, and you can analyze the traffic without logging into the instance. Because a lot of times you might not have access to the instance because of some reason and traffic metering would give you some insights as what that specific malware a bot is exactly trying to do. That is one use case that I found to be useful. Another use case I want to talk about is I've seen a lot of products that we product also using traffic metering right now. For example, we were trying to evaluate an API security product. So a lot of times, especially in smaller companies, there could be some API changes being made go live without notifying security teams as it could be very minor change or there could be some reason to go live without notifying security teams. But for us, it might cost security lapses. So we wanted to basically make sure that any minor changes which is happening in the API should not escape our eyes, and we were trying to evaluate a security product which gives you any changes which is happening on the API structure. So interestingly, what this API product does is it uses traffic metering to ingest traffic from your cloud front to ELB where all the ingress traffic comes our traffic comes and figures out by checking the parameters, if there is any new API which they can add into the database and notify us if we already don't know about it. So this is one of the interesting use case. Again, it's not the best way to solve for this problem, but traffic metering can be applied in all these different use cases to serve your purposes. So yeah, I think that's what I wanted to convey that apart from getting insights on network traffic metering can be used in other use cases as well.