 Hey everyone and welcome to this webinar by the CNCF about strengthening your Kubernetes and container security without losing visibility. We're going to use the CNCF project Falco and the advisor today for this purpose and we're going to present how these two projects can interact with each other. First of all, I would like to introduce myself. I am Luca and I'm an open source engineer at Sysdig. I'm one of the maintainers for the Falco project and my passion when it comes to computing is security in all its forms. So whether it's vulnerabilities, how to make system more secure, how to identify with spots and correct any security issues or detect potential threats at every level, I'm all for it. You can find my code and my bugs on GitHub or you can find myself along with all the other maintainers and the Falco community on Kubernetes Slack in the Falco channel. And I'll let now Nicholas introduce himself. Hey everybody, my name is Nick. I'm a software engineer at Google and I've been working on GVisor for the past couple of years. I'm passionate about containers and security and operating systems. You can find me on GitHub at the link there, github.com. And I also hang out in the GVisor Gitter Channel along with most of the other GVisor maintainers. So yeah, happy to talk to you. Thanks a lot. I'm very happy to present with you today. And also I'm very happy for the work that we've done along with both the Falco team and the GVisor team. On the Falco side, I've had the pleasure of working with several people and I actually want to give a shout out to two members of the team, specifically Lorenzo Susini who is working at Sysdig with me. He actually joined us as an intern and he did a fantastic job on several projects including the GVisor project, enough that now he works full-time on open sources software, which I'm very happy about. And also I've had the pleasure of working with Greg who has been working on Falco for years now and he was very important in getting some base refactoring going to allow this support to be built smoothly and with code that, you know, it's actually well written in terms of engineering and not a complete hack like some could do. Also I've had the pleasure of working with the team that you see in the background for Falco that are some of the Falco maintainers who were very helpful in helping us getting this feature upstream. Also I really want to thank for the GVisor team Fabricio who is the person on the left on the GVisor team picture who has been for the most part our main point of contact for this project and he was fantastic in implementing, designing solutions to really get this project to work smoothly. It's thanks a lot Fabricio. And Nick, if you want to talk a bit more about the GVisor team, the excellent GVisor team. Yeah, you know, we've got a lot of a team that's been working really hard on GVisor for a while now. There's about half of us pictured here. We're fairly remote, fairly distributed I guess. So it's hard to get a picture of everyone together. But yeah, Fabricio did most of the work for the Falco integration that we're presenting today. Also, Shambavi Strivastava and Steve Silva helped out a lot too. Cool, so let's take a look at what we'll be talking about today. So today we're going to tell you why sandboxing containers and applying security at the same time is actually a challenge, rather challenging. Then we're going to introduce both the Falco project and GVisor. And then, of course, some of the audience would be a bit more familiar with Falco because it's a CNCF project. But many maybe have not tried GVisor and you will learn how cool it is. And then we're going to explain how the GVisor and Falco integration that we built is actually working. Then we can share something more about how it is as a user, what can you get out of it and how you can interact with the open-source communities that are driving this project. So first of all, why is container sandboxing and security a challenge? So Nick will tell us much more about GVisor and how it works. But for the uninitiated, think of GVisor as a mechanism to sandbox and isolate your container applications and workloads from the host operating system. So as a security person myself, I actually love this feature. And I think it's really one of the most important security features that you can have that is a tech surface reduction by isolation. GVisor is one mechanism and one technology that you can use to prevent your workload from performing potentially dangerous operations and to have better control of what you run. And this, especially if you think about highly sensitive environments, it's a very important capability to have and every security person will be able to confirm. So this is great and Falco does something for security but it's on a bit of a different angle. Falco is able to detect anomalies, is able to detect what you want. Falco is able to gather events from your applications, from your hosts, from the kernel and is able to report what you are interested in to you. So it's more about the visibility. It acts like a security camera for your servers if you think about it. But actually sometimes we will see it in a bit more details. If you try to use such a technology in an environment that is deeply isolated with the heightened security such as one that works with GVisor, you are actually going to get problems because the security controls that you put in place can actually make it harder for you to see what's going on inside your containers. So we work to actually get no compromise here and be able to both have a more secure environment and get the visibility. And the first, we are very happy to say that the first tool that is able to monitor GVisor workloads is actually Falco and we'll tell you how it works. So first of all, if you're not familiar, I would be happy to spend a couple of minutes just to give you an overview of Falco. Falco is a project that is the first cloud native runtime security project that has been adopted by the CNCF. So Falco was originally created at Sysig and then it's now incubating at the cloud native computing foundation. It was adopted a few years ago and it grew tremendously since its adoption and throughout its incubation at the CNCF. More I work on Falco, the more I realized that probably in every, everyone uses some kind of big cloud deployed systems from companies, big and small. And at least one system that you're currently using is probably monitored by Falco since its adoption is so big and so important in the security landscape. And also back in the day, it used to be just one project from Sysig and today it enjoys a lot of contributions from a lot of different people and maintainers that are from many different companies for many different reasons. And it's very cool to see these working to see people from different backgrounds working together on the Falco project. So if you've never used Falco, its most basic usage is like this. So basically Falco will take all the system calls that are happening on your system and will alert you based on a set of rules that you can customize and you can specify it with a very simple language. So its basic idea is that you will get warnings of notices or errors about what you want. As you can see, every event is also automatically enriched with all possible informations from containers. So any metadata that might come from your container engine plus metadata from your Kubernetes environment. So your pods, your clusters and all this information is natively available Falco. Falco is not just a command line application but it's got an environment and an ecosystem of applications, for example, of the application that can connect to it. So for example, we've got Falco Sidekick which is a cool piece of software that you can use to forward your Falco events to any other event processing system that you might have being at the CM or email and Slack integration. And also it's got a very cool UI that I hope I'll be able to show you a bit later. So the basic idea, if we just scratch a bit under the surface of Falco, we will see that Falco itself is composed by two main parts. One is the Falco along with its own rule engine. And the other are the underlying libraries that allow Falco to work. The two libraries are called the LibsSins and the LibsCAP and collectively are called the Falco libraries. If you have never heard about LibsCAP but if you have used the packet captures in the past you can think about it as exactly the same as a packet captures but for system calls. So it's got a data source which is normally a kernel model or an EBPF probe that you install in a kernel and that is going to forward events to a user space component which is inside LibsCAP that will simply collect the events. Then what happens on top will be a lot of analysis and inspection by the LibsSins module that and that will eventually end up in the powerful and flexible Falco rule engine that will allow you to specify these are all these kind of rules that you can see. The rules are easy to express and are fully customizable. So just to show a diagram of a very simple Falco deployment normally you are going to run Falco on directly on your Linux kernel through its kernel module or EBPF probe and it's going and that one instance of Falco will be able to capture that can be of course the one instance can be installed via either a container or a demon setting case of Kubernetes or a regular process on a machine the way it's more fit for your production environment. Then it will be able to run Falco it's going to capture every event coming from all the containers installed all the Kubernetes pod and every process that runs on the system. Now this is the basic way to run Falco and I will let Nick tell us about how this environment changes that when we decided to secure our application our processes with Gevizer. Thanks Luca. So Gevizer is a sandbox for containers and as Luca mentioned, Gevizer's primary goal is to isolate containers from the host machine that they're running on. We want to prevent container escapes and prevent your infrastructure. The way Gevizer is distributed is just as a simple OCI runtime and so it integrates pretty seamlessly with Kubernetes and Docker and is quite easy to use in your existing deployments. Gevizer was developed at Google and used at scale. It powers a lot of different cloud products App Engine, Cloud Run, Cloud Functions and GKE is Google's hosted Kubernetes engine and there's also a product called GKE Sandbox that makes it really easy to run your pods that are running on GKE to run them inside of Gevizer Sandboxes pretty easily. It's also used by a variety of different projects inside of Google that I can't say too much about but Gevizer is also open source. Gevizer.dev is our homepage and all of our code and issues and everything are on GitHub at github.com, Google Gevizer. So it's a great way to get in touch with us there as well. So before getting into how Gevizer works we'll maybe say a little bit about why it exists. So with standard Linux containers those containers are interacting directly with the host kernel. There's a couple of isolation mechanisms in place like namespaces and C groups but fundamentally those containers, those applications are making system calls to the host Linux kernel. Linux is a fabulous piece of software. I mean, it really is but it's also very large attack surface. There's a lot of system calls. There's a lot of stuff going on in that proc file system and there's a lot of vulnerabilities. So since 1999 there have been over 2,000 CVEs in the Linux kernel and 250 of those have been privileged escalations and it only takes one bad Linux bug to get a container escape. So if there's one system call with a vulnerability containers can exploit that to gain access to the host and from there malicious workloads can attack your infrastructure or attack other containerized workloads. So this is something that we wanna prevent and this is why GVisor exists. So GVisor the architecture is a little bit more complicated. There's a few different components. The first one we'll talk about is this GVisor kernel. Next slide, yeah. So this is the one that's in bold right there kind of on the left hand side. This is the main component of GVisor. This is a fully Linux compatible kernel. So we've taken all of the system calls that Linux has and we've implemented them ourselves in GVisor. And I really wanna stress these are full system call implementations. This isn't just like a pass through or a filter. When the application is running and it's making system calls that thinks it's interacting with the host kernel it's actually interacting with the GVisor kernel. So we've implemented all of the Linux system calls. We've also implemented all of the common Linux file systems like PROC DEV, Sysafas, Tempafas. This is a real kernel. So there's all of the kernel primitives that you'd expect. There's memory management, few texas, timers, VDSO. There's actually a full TCP IP stack that we run inside of the GVisor kernel. GVisor kernel is written in go. We chose that for the memory and type safety that go provides. By getting rid of use after free or out of bounds bugs you can sort of write off a large class of bugs right there. Of course we still have to be careful and not everything is safe but by using a memory and type safe language we get some more guarantees of safety. Go also has great concurrency primitives that make it a good fit for operating systems. And go as a language is also just very productive and a fun language to work with. It's really great. Then the last thing I'll say maybe about the kernel is this kernel runs in user space alongside the container. So from the host's point of view the host Linux kernel just sees GVisor and the container running together. It just looks like any other process from the host's point of view. A couple of other components that I'll mention the containerized workload that the GVisor kernel plus the container run inside of a set comp sandbox. So set comp is a utility that Linux provides to allow you to restrict the number of system calls or restrict the set of system calls that a workload can make. So the container and GVisor together are only allowed to make a very small set of system calls to the host. Basically those needed by the go runtime plus a few others. If you go on our GitHub you can actually see what the list of system calls allowed is. It's very restricted. So for example, the container and kernel are not allowed to open a file. They're not allowed to create a socket. So they're quite restricted. If the container does, I mean containers often open files those container opens a file the GVisor kernel will intercept that system call and it will forward it to this file system proxy that runs outside of the sandbox. So the file system proxy is responsible for handling all of the file system access. So all of the opens, statting, walking, file system paths, et cetera. This provides another layer of security around GVisor. It's another area where we can sort of dictate what the sandbox can do. And the file system proxy also has its own set of set count filters. So, you know, all of this is, of course it's allowed to open files unlike the GVisor kernel. But all of this is about providing multiple layers of security around the containerized workloads. Yeah, so that's sort of what this picture here is showing. So these multiple layers lead to defense in depth. So the GVisor kernel itself is the first layer of defense. Actually our security posture, we assume that the GVisor kernel is compromised. Even if that is the case, we still have enough layers to protect the host and infrastructure. We've got the set count filters that I just mentioned. So you can't open a file or create a socket. The entire kernel and workload runs as UID and GID nobody with no capabilities. There's namespaces around those. These things run an empty username space. And lastly around all of that is the pod C group. So all of these different layers make it harder and harder for a malicious workload to break out of its container and reach the host. Yeah, so I mentioned that GVisor is distributed as an OCI runtime that plugs in to Kubernetes. GVisor can run on bare metal and it also works inside of a VM with nested virtualization. So that makes GVisor a really good fit for your cloud Kubernetes deployments. Yeah, and then so all of these mechanisms I just talked about are about preventing container escapes, which GVisor is pretty good at, but prevention is not detection. So while GVisor can prevent these container escapes, we never had a mechanism to actually detect when a malicious workload was trying to escape. And that's why we're really excited about the Falco integration because that's what we can provide now. Thanks Nick, I think now we have all the information to understand why our project has come to life and what's the problem that we had when we first started to monitor container workloads that were actually a sandbox within GVisor. As Nick explained before, actually the interactions with the kernel in case of a GVisor sandbox are very different from the regular interaction that you get with native process. So what would happen if we try to just plug Falco as out of the box as we always do into such an environment that well, we'll get a very confused Falco because if you remember GVisor not only is a different process, so instead of actually having our normal process, we're going to have a GVisor kernel process, so which is a different binary, it's written in Go, it does other operations, that it has a different set of interactions with the Linux kernel than the normal application would have and the normal process would have. And also it does indeed execute some of the operations that our process will do, but it executes them in a very filtered way and also with second filters already in place, which means that whatever Falco will see at the end is something, but it's definitely not what our original process does. And as Nick correctly mentioned before, what we want to do is that we want to detect suspicious behavior that is executed by potentially compromised application or potentially malicious application or even something that we are actually trying to troubleshoot because Falco can be used for many different types of use cases. So whatever misbehavior we have in mind, we want to get it straight from the application. And we cannot even rely on the capabilities that Falco has to, for example, tell us about file system usage or system codes that interact with the file system because even the file system to get the right security properties is a proxy that in divisor. So how do we actually solve this problem? Well, first of all, if you recall, Falco itself is not the model, it's composed by the different set of pieces that interact together. And the piece that we're actually interested in is Lipscap, that is the foundation on top of which everything is built. So Lipscap inside of itself isn't that complex. I mean, the mechanism that it has to collect events can be very complex, but itself it acts as a collector events exactly like as I mentioned before, a packet capture. You've got a system called capture. So what we need to do is we needed to refactor and to work on Lipscap in such a way that is able to accept a new source of that system called events that are coming in not from kernel, but directly from a user space from the divisor kernel itself, which will be able to send them in some way. We have decided to use a Unix domain socket for this and inside it's talking protobuf to extract events from the divisor kernel as they happen directly from the application and reach them on the divisor side with metadata about container and every information that is not available straight into the host and the Linux kernel that Falco usually queries to understand. And then it will forward them straight into Falco, which on the other side will act as a server, receive the connection from the divisor kernel and will be able to read the format that divisor provides and convert it into the Lipscap internal representation that can then be used by anything on top of it. And this is pretty cool I think for several reasons including the fact that this happening at a lower level, any other project that uses the Falco libraries which are several of them since the Falco libraries are open source and they're used as a base for not just Falco but also other research projects and other open source software will be able to enjoy this integration and will be able to seamlessly connect to divisor and to read the events that are happening in the sentences as if they were happening directly on the kernel. Also, if you recall how Falco normally works with one instance of Falco per node, this integration allows it to just work in the same way as it does with a regular kernel. So you will be able to spin up a Falco instance in your node and we'll show you how in a second. And then this will be able to connect to any number of sandboxes that are running that are managed by divisor via the same server as the socket is the same and Falco will active the server and will handle everything. And that doesn't break user experience in any ways because that's exactly what Falco does when it monitors several pods or several containers within the same system and the same cloud node. So now I'm eager to show you how this actually works in a real system. So we have a cloud instance here and we have installed both run a C which is the tool that is used to monitor to actually make the magic that Nick described for divisor happen and we also installed the Falco. So we can check first of all that the versions of our tools are correct. So I've got Falco version that needs to be at least 032.1 and we've got to 032.2 which is the latest at the time of speaking. And then we've got run a C as well, run a C version. And the run a C version is expressed in this case as a timestamp and as long as there's a newer than the fourth of July, 2022 it's going to work with this integration. So I'm now going to start my Falco instance and on my terminal I will add a couple of parameters that will allow us to enable the integration. First of all, I will need to specify a configuration a divisor configuration. Now for the whole instructions about how to set this up you can refer to the tutorials that are present on the divisor blog and on the Falco blog. But just to show you, I have simply installed a little configuration file which is Falco.init.json which is a file that instructs divisor to collect some types of system calls and to forward them straight to Falco. I've also configured divisor via the run a C install command to receive this configuration when it starts in pods. And so we can see the configuration into my Docker demo is installed this configuration into my Docker demo configuration. So if I specify as a runtime a runtime that I call the run a C Falco then this integration will be online. So we're going to start Falco. Falco is going to start and tell us that the event collection from divisor is enabled and it has received our configuration file. The configuration file is shared between Falco and divisor so that the two pieces of software can make sure that they are aligned in terms of systems that they collect and the socket path and all that tunable configuration. So I am now going to run a pod directly via via our my regular Docker. So Docker run. I'm going to set the runtime as run a C Falco that is the one that we've configured here. And I'm going to run my image which is just any image that we have and start the top. This workload is now running under divisor and it's also monitored by Falco. If you don't believe that this is running under divisor let's take a look at this, the message. And I think this is something that I'm not expecting in a regular workload. This is actually telling me that divisor is enabled. So let's try to take a look and see what a suspicious event might be. So I'm going to first open the ETC shadow file which is not something that the regular user does. And as we can see, this has generated an event in Falco. I have used the JSON format to take a look at the events but we can, as I'll show you, we can see these events in whatever format we see fit. So we can see that the program cat has open ETC shadow and then we've got the PID, we've got the container ID and everything, the container ID just works seamlessly as if the container was running into our application. Another thing that could be suspicious is the fact that some versions of netcat are having an E flag that not a version support but for all the old school hackers out there we all know that you were able to just do this and with adding some bunch of parameters to do that. My version doesn't support it but Falco will still detect our attempt even if it doesn't do anything but it will still detect our suspicious attack. Also, I would love to show you how this looks also in a different user interface. So I've got the Falco sidekick UI over here and as we can see, all our events have been collected. We've got our warning events. We've got the our netcat remote code execution attempt in our container. We've got the read of the ETC shadow file that we've got here along with the file name along with all the context information that we might want. We might filter by containers here and so we can take a look at everything that this container has done. And this works in the same way whether you're using whether we're using divisor or not. I hope you enjoyed this little trip inside Falco and divisor. So if you have been following the Falco project for a little while, you might know that back at the beginning probably at the time when the CNCF adopted it that was, I believe, quite a few years ago at least four years ago, there was only one way to capture system calls and that was it. So you've got your Lipscap, your kernel module and a model on top of it that would just run the rule engine. Then later, it got support for Kubernetes audit events which are events that are not system calls are just something different. And after a while, it also got support for EBPF which is a very cool technology that allows you to write code that runs directly into the Linux kernel without having to write the kernel module in a much safer way. This comes from network and now works for system call and it's a very exciting technology that Falco has fully supported since years now. Then later, we decided or at least the Falco community has decided that just having Kubernetes audit event wasn't enough. And now there is a full-fledged the plugin framework where you can upload whatever data you want into a plugin that will be consumed by Falco and will let you interact with the same language and with the same rule engine that you know a lot from system calls but it will work for any event. For example, the cloud events. We have open-source plugin for Cloudflare, for example the one for the Kubernetes audit event. You could think about something for GCP, for example or there's a and this is very new and there's a lot of exciting things that are happening in the community on there. And now we were able to refactor it even more to make it even more modular to add a new system engine and new way of collecting events. And so thanks to our collaborations thanks to the community and thanks to the work done within the CNCF of Falco is that I can see since I've been following Falco for a while Falco really growing and getting more and more capabilities by every release pretty much. We would like to now tell you just a couple of things about this integration project. As you can see it's working with the new versions of Falco and advisor and it's I think the latest brand new feature that was added in the latest versions of Falco. So we are all very excited as it's new and cool. But the way it works is a little bit different from the usual kernel collection event because right now since all the captures are happening in user space every system call will actually need to be supported on both the divisor side and the Falco side because divisor will need to be able to transmit the system called data and Falco will be able needs to be able to receive it and convert it into its internal representation. Falco itself has support for a lot of events that are more than 290 different system calls and events that you can use in your security rules. The Falco and advisor integration does not support all these events right now but we have chosen a subset of them that are the most common the ones that as you could see can be most useful in the default rules and in the rules that most security engineers are interested in. But of course there's room for many more and it's possible to, I don't think it's possible to have the full coverage of events just because divisor and Falco work in a slightly different way. Actually divisor works in a slightly different way than a regular Linux kernel and the events that divisor processes are sometimes a tiny bit different from them but I believe that the majority of events can be imported and just with support from both sides can work directly in divisor. The good news is that it's easier to implement an event in the open source divisor and open source Falco on the user space than it is to for example write kernel code or eBPF code to collect a new event because you don't really have to mess with kernel. You don't have to think about all the compatibilities with all the different versions of the kernel which is one massive pain point when you are dealing with directly with Linux kernel as a developer. You don't have to fight with the eBPF verify which is very cool but also very picky about your code. And so if for both the divisor team and the Falco team and actually any contributor that is interested it should be easy along with our help getting support for more events in case we see that we need that. So our last call in here is to invite you to join the community if you're interested and contribute both divisor and Falco have thriving community and the Falco one is supported by the CNCF. So, you know, it has some pretty high standards in terms of inclusivity and the openness of everyone pretty much everyone's back. In terms of usage, we have actually seen usage from early adopters of this technology. For example, Mercade which is a global marketplace for buying and selling based in Japan is actually a shared user between divisor and Falco and uses both technologies and does not really want to compromise and pick one over the other when it comes to securing their environments and their containers and their applications. So Mercade has had the chance of trying this out and especially in the environment where they operate that it's highly regulated they want the best in terms of security. And so they were very happy to see how this can drastically improve container security for their use cases. Also, I would like to just as to recap what we have discussed today that when it comes to security we want to detect anonymous behaviors within workloads and we want to integrate our security system with the existing rules and the open source open source rules and the open source technologies as much as we can. We want to monitor our systems and also we want to have workloads that are natively sandboxed and they are actually prevented from doing some operations that might actually impact our systems at a higher level and make it a simple security incident escalate. So we can use Falco to monitor container nodes and still enjoy high security provided by divisor. I would like to add just a couple of links to both the divisor website on divisor.dev. You've got a cool tutorial that guides you step by step to get an environment just like the one that I showed you that has both divisor and Falco support and you can find out more on the Falco blog as well that we'll go into a bit more details about how you might want to configure Falco in such an environment. And if you are not familiar with divisor I would encourage you to take a look at divisor.dev since there is a lot of cool information about how this technology works and how to contribute in case you're interested and in case you want to learn more about Falco, a book recently came out from O'Reilly that you can get that was written by a lot of the original author of Falco and Falco containers to learn more about the technology. And regarding Falco, I don't think I need to add it to add much more since it's a CNCF project. We know you just go on falco.org and you will find our community on Twitter that you will find a Slack channel. We hold weekly meetings that everyone is welcome to participate in. And I think maybe Nick wants to share a bit more about contributing and the community around divisor as well. Sure, we've plugged to divisor.dev a few times already but it's a great place to go. There's a getting started guide if you want to just download divisor and start experimenting with Docker or if you want to integrate it into your existing Kubernetes deployments. All of those guides are there at divisor.dev. There's also some really good architecture guides if you're curious about some of the things I talked about today and you want to learn more. We've got a lot of documentation on divisor.dev. All of our code and issues and pull requests from the community go through GitHub. So that's if you search for divisor GitHub, you will find it. And then most of us hang out in the divisor Gitter chat room. So that's another good way to get in touch and ask questions, meet some of the engineers working on divisor. Thank you everyone for listening. Thanks a lot Nick. It was great working on this project really. Likewise, yeah. Thank you everyone for attending.