 OK, everyone, welcome. Thank you for coming to this talk about Infernal Analytics and tracing for EBPF with EBPF for OpenStack Clouds. Specifically, I'm going to be talking about how to use the tracing capabilities of EBPF, along with some machine learning techniques to secure microservices. So my name is Brennan Blanco. I'm an employee at Plumgrid. Most of the work was done by Ali Kayam, also Plumgrid. He's not able to make it, but I'll be presenting on behalf of the work that he's done. So that's why his name is up on the slide. And we're both doing this work on behalf of the IAVISOR project, the t-shirt here. And the IAVISOR project is a Linux Foundation collaborative project composed of the companies with those logos. And IAVISOR project, just to give a little bit of background before I jump into the material, IAVISOR project is a community and a set of tools that people in that community use for doing high performance IO applications, from providing networking functionality, to doing tracing, security, all types of new infrastructure applications using these tools with a mind towards security and performance and IO. Now, so the premise of this talk, we want to look at microservices. And when we think of microservices in terms of security, you think of the lifecycle of those services. They come and go very quickly. And they have some interesting characteristics that are unique to these services that kind of change the way that we have to think about security. For instance, services run on a shared kernel might be collated with other tenants or users. So shared kernel means it's lightweight, but it also means there is a larger attack surface potentially if you don't think about it properly. And it's nice that now developers are turning into their own sys admins. It speeds up the lifecycle. So that self-service is great, but does that mean they also have to be a security expert? We'll see. The shared infrastructure now that all of these services are running on are, I mean, we're making great use of the physical resources, but do we have to be worried about users as part of the same organization compromising one another intentionally or otherwise? And as we go through this fast development cycle, are we compromising some of the assumptions that we were making when we were working in VMs or in the cloud where we could assume nothing about the VM and treat it as a source of potential danger and be able to isolate it very easily? Now, who do we trust? And if we want to secure these microservices, where should we put that security? Now, the answer is you have to defend in all the layers. Everyone should be thinking about security. No one is free from responsibility, including the developer. But as open stack infrastructure operators, we have a responsibility, at least, to think about our piece. So in this talk, we'll look at how to provide security for microservices as an operator. So we'll focus on the right-hand side here, the infrastructure operator. What's in, if we think of maybe the security as a service concept that the infrastructure operators might provide, what should be the characteristics of that? The first thing is it should be transparent. You don't want to reduce the capabilities of the microservice that's running. It shouldn't impact the performance or any of the functionality that these developers are expecting to get from their platform. So it needs to be kind of invisible. And it needs to be generic. You don't know what microservices are running. You want a platform that adapts to all types of use cases without having to special case everything. So the way in which you provide security as a service should be something that's flexible and generic. And again, I'll re-emphasize it needs to be efficient. Besides the transparency of functionality, it also needs to be as close to zero overhead as possible. Otherwise, someone's going to go to a different platform that's more efficient, even if they're willing to give up security. Security is something that you shouldn't have to compromise on any of these things. So now, as an infrastructure operator, what are the things you can know about those microservices while still treating them generically, while looking at the black box of that service that's running, and still being able to learn something about it, and do something about when it's performing insecurely? What can you look at? You can look at API calls. So if there's a REST API or an RPC API, you can inspect those, but most likely they're going to be encrypted. So you have to maybe put some rough guidelines on what you can look at, so the contents, or maybe worst case, the length of the requests and responses. Generically, you can look at traffic on the whole. Packets in, packets out. Everything is networked these days. Everything is usually, we'll probably have a disk associated with it. So you should be able to look at disk.io as one of the metrics feeding into your profile. And the tenants that are actually there, you should think of that as the IDs of the services that are running have meaning. So who is running the service is important. So in Linux, those will correspond to namespaces, or process ID names, or C groups that you can, with some little bit of legwork, tie back to the service that's running. And again, you want to be able to gather these metrics transparently and efficiently. So to do those things, we have been working on this experiment to see if we can use iAdvisor to extract some of those metrics and see what we can build. So the thing that we'll be showing here is not a product. It's not a finished thing by any measure. It's an open source project. And we're just looking to see what the building blocks can provide. So with that, we have the set of tools that's already there in Linux. And it has the ability to capture API calls, capture network IO, capture system calls, perform some analysis from them, and to do so very lightweight. So we'll use those tools to build this kind of a feedback cycle between the applications that are running, whatever they may be, and the infrastructure operator to be able to look at those black boxes and to learn something about them and potentially act. Now, if you're treating the services that are running as a black box, you don't really know anything about them. So as an operator, how can you figure what's good and what's bad about those services? What's a particular profile for one service in terms of network and disk accesses? Maybe normal for one service. But if you apply that profile to a different service, it will look abnormal, even though they're unrelated services. So you have to decouple and put a blindfold over yourself to make this judgment about what's good and what's bad and rely on, instead, machine learning to take these metrics, which we'll be gathering and feeding into machine learning algorithm, and seeing if we can get a high accuracy in terms of detecting threats and also not polluting ourselves with noise. So we have this workflow of monitoring the microservices, both in terms of API and traffic, along with disk and memory accesses, feeding them into a collector, and then taking the output of that and feeding it through machine learning and seeing if we can detect threats. That's the experiment. So changing gears a little bit. The tools in Iovizer let us, in a very lightweight way, start to monitor the attack surface of Linux and to gather those metrics in a very customizable way. So here we're just showing a few code snippets of how you can kind of script some of this. And these tools we'll show at the end. We have GitHub and so on. You can see the full example. We'll show that you can attach some of these functions that we'll write to different Linux internals to collect things, for instance, as the VFS. So VFS in Linux, a virtual file system, it's kind of the abstraction layer to all of the block devices or network devices that do block access. So it might be interesting to gather the metrics related to block writes using the VFS API. Similarly, you can collect the packet and byte count from different interfaces and take those and, let's say, feed them into a histogram. So you might be able to take some metrics and do some analysis on those from the script or from within the kernel itself. So things like, at the end of the year, you can do calculations such as latency, latency of virtual file system calls. So that's not something that's generally reported by top or IOTOP or some of the kernel APIs on a per function basis. So it'd be useful to be able to capture those and tie them to the particular microservice to be able to start to build these metrics. Here's another snippet. So this is going down another level. So the tools that we provide let you really get into some of the details of how Linux is running, how the traffic, for instance, is flowing through the kernel. You can write a very customizable parser. And that's something that we've been doing to add into the Iovizer tool set. You won't necessarily have to do. But I'm just putting this here to show the power of the tool chain and the types of things that you can collect. So we have two examples that we'll show. And the first of these is to look at OpenStack as a microservice itself. OpenStack these days is deployed as a series of containers. You have a Glantz container. You have a Keystone container and so on. So we did one experiment to see if we could learn something about OpenStack. Even though you could inspect it through just reading the Python code and adding your own logic, let's treat that as a black box and see what we can learn. It was a tool that we already knew. So it was a good starting place. And so what we did, we used the Iovizer tools to build the security profile. We let OpenStack run normally, collecting the metrics for VFS reason rights and the length of API calls that are going into the different services. And we took these profiles based on what the service in question was. So you had a different set of profile for Glantz, a different set of profile for Keystone, and so on. And we're going to focus on Keystone. So we took those profiles for all of the OpenStack services. And then we just created a simple password attack scenario that would start to just try to guess passwords to log into Keystone at kind of a low threshold, not completely overloading Keystone, but doing something that's a little bit out of the normal. And we kept that up for a little while. And so taking those, you feed the outputs of the good profile. And what we did was create a rock curve that will tune to minimize the false positive and increase the true negative. And the results for that were pretty good. So it was a pretty simple test. And we got a 97% detection rate based on that machine learning algorithm. So that was, I think, a successful test. And we tried one other example, which was a database container running within Docker, running MySQL. And what we did was we played a series of SQL queries, the good profile, and monitored all of the packets on the network, their size and length. And also the VFS reads and writes to see how particular types of queries might relate to particular reads from the disk or writes to the disk. And if you end up seeing some query that you expect to produce a low number of reads and it produces a high number of reads, that's something that is potentially outside of your curve with some probability. And these are all done within the kernel. So we keep these statistics and aggregate them over time. And then we injected or ran a query to do an SQL. So we had some queries that were SQL injection attacks, which extracted large segments of the database back to the malicious user, so it was trying to collect all the information inside the database. And that was while the benign traffic was running. And we were able to detect those as well. A little bit more false positive. So this was not quite as accurate. So there's still a little bit of noise there. So there's some work to be done. And there's also a dashboard that came along with this. So we fed this time series database of analytics that we're capturing from the kernel. And you can see here the four spikes at the bottom are the algorithm detecting the attacks. So this is when it went across that threshold of saying that this particular correlation of network traffic and disk accesses is abnormal. So at this point, we can see how this works. So here we have two shells. And on the left, we have our Docker container. We'll start that up. And we'll start a monitor on the interface for that container. And every five seconds, that's the five parameter there. Here's the name of the interface. We're going to be collecting those stats. And as well in the background, the VFS calls are also being monitored. And here we're just showing the log of those data points being collected. Now I don't have, side by side with this, the CPU utilization or the overhead of these monitorings. We're starting the queries in the background. So we don't have the CPU utilization of that. But the overhead is in the 0.01% overhead for the monitoring that's being done. And as compared to some other internal analytics tools like Perf or EscapeMe, but the overhead is very small. And you can just do the summarization of the statistics of, let's say, reads per second, bytes per second, and so on. And just only on demand from user space will you extract those using a system call. So we can see here the summarization of those different values printed into a log from the little Python script that's running on the left there. So we can see here the particular statistics that we're keeping. So VFS reads and writes, TCP traffic. And then we're going to start next the SQL injection. So we'll start the attack. And you won't see anything here. This is just, on the side, we'll be feeding it into the machine learning algorithm to make the deduction about whether this is good or bad, or if there's anything abnormal going on. We'll switch that real quick. So here we see, again, the same dashboard from the query that we're just running. So this is something that you can update in real time. It's a very lightweight analysis that you do. So in conclusion, we set out, again, a research experiment to see what some of the components of security as a service as an infrastructure operator might be. And let's look at the transparency aspect, for instance. The application was not invoked at all. We didn't instrument anything inside of MySQL or any of the OpenStack services. This was just something we started running off to the side. Didn't have to restart anything or recompile the kernel or load any kernel modules. This is using functionality that's already there in recent kernels. So EBPF is something that we rely on that lets you kind of script inside the kernel. And we saw some of that in the code snippets earlier. And so the developer is completely unaware of the fact that they're being monitored. The tools that we use rely on a GitHub repository that's under the I-visor umbrella that provides some of the tools and use cases. So in terms of how generic this is, we showed two different applications, two different types of microservices, one which is the OpenStack services composed of all the different types of binaries that are running inside that, as well as something that was focused on MySQL. But again, the tool didn't change. The tool was generic. It was just looking at a generic set of kernel properties. And we have some future work to see, again, that this scales to other types of attacks and is generic across those as well. So trying to look for DNS types of attacks, which is kind of relevant recently, things like ransomware. So trying to see if the training set or the profile generation works for those types of use cases. And again, the efficiency for this, it's very low overhead. The probes that are running inside the kernel are pretty seamless. We're not dumping data from kernel into user space. And looking at it later, it's something that you can look at right inside the kernel, do the summarization. It's not sampling based. It's something that you can be completely accurate in all of the metrics that you're keeping. So you're not going to throw away data or miss something. And it's completely scriptable. So you can decide from your application from user space how often you want to try to detect the various anomalies. But even while you're not pulling, the data is still being generated. So you're, again, not losing anything. And the machine learning part is something that we do off box. So we can take these very, very lightweight summarizations, probably just a megabyte or something very small, which is also tunable. And ship it off, let the analysis run, maybe even across a cluster. You can do maybe a core screen analysis and do a heavyweight one later and archive the data for whatever future use case you might want to look for. But that's all invisible to the nodes where the initial analysis is running on. And so here's some analysis that we have on the overhead of maybe the scalability factors of some of the things that we're monitoring, where, for instance, the VFS read, it's a hash table that you keep binned based on the time and as well the number of processes that you're keeping track of. So there's a finite amount of processes that you'll need in Linux. And this will scale to that number. It's just a compact hash table based on process ID. And that's the gist of the presentation. So if you're interested in looking at the tools that we just showed, Ali has these on his GitHub. We might be moving that over to the Iovizer GitHub in the short future as well for all of the Iovizer tools and the various mailing lists and so on. You can see the details up there. And with that, any questions? I'll go back to that slide in a second. So any questions? I'll leave this up for people to look at. But I have a microphone here. And feel free to walk us around. So the question is, what kind of Docker networking used in your test? Is it like with base or does it support OBS? This was, I think, a simple Linux bridge. But that's kind of tangential to the tool itself. I mean, these tools can be attached to generic vEaths. They can be used to monitor kernel functions irrespective of the network device type. It's very flexible. Do you have a solution for when you're using kernel bypass technology like DPDK and SRILV? So within the set of hooks that you get, with BPF, besides attaching to kernel functions, you can also attach to user space libraries. So you can use this as a user space library monitoring. Just like as a crazy example, on the performance monitoring side, someone built a memory leak detector by putting a probe in libc at the malloc and free calls and was able to build a memory leak detector based on doing some analysis within the BPF program. So I don't know the details of the DPDK library itself, so I couldn't tell you what functions to instrument, but it is possible to instrument most any function. As long as it's in a library of some sort. In terms of networking, are there any limitations about the depth that you can get into the package? You seem to be counting amount of traffic, but could you count type of packets, which ports are going to any type of networking information or the traffic? Yeah, so you could, for instance, build a simple packet parser and include TCP packets, but not UDP packets, or try to figure out what the URL is that's being requested, assuming it's not encrypted. Some of those assumptions are why we choose to treat this as a black box and make it more generic, but if the data is there, you could build a parser for those things. With the caveat that the more work you do, I mean, you do have to pay for some of that in performance, but it's a very efficient library, so it can accommodate those extra use cases, I think, pretty well. Okay, so if you have any further questions, feel free to come up if you want to ask in private. Otherwise, thank you very much.