 Hello everyone and welcome to the maintainer track for Falco avoid a little wind and catch the jet stream using Falco to detect attackers and compliance violation I am Luca and I work as a software engineer at Sysdig and I'm joined here And I'm also a Falco maintainer and I'm joined here with the other awesome Falco maintainers We have Carlos who is a software engineer and very expert coffee maker at Sheingart and also I'm joined by Jason Also a software engineer at Sysdig and Melissa a security engineer at Apple so Today we'll be talking about Falco and I guess that if you're in the maintainer track for Falco, you know What what we're talking about but anyways as a refresher Falco is an open source security project for threat detection across Kubernetes Containers hosts and the cloud it started as a project to detect any suspicious events via rules in system call By rules using system calls in your hosts and your containerized environment And then gain the ability to inspect Kubernetes as well And then now with its powerful plug-in system is extended to the cloud and pretty much every sort of event that you can think about it Falco is currently Currently a CNCF incubation level project, but I have some Exciting news to share that the Falco maintainer and contributor community have worked very hard to get the Falco project closer to graduation and indeed we we made a lot of progress and I am very happy to say that we have passed I think what is the most important part of the graduation project a process which is Knowing that the CNCF technical oversight committee has decided that Falco is mature enough to really proceed to graduation. Of course We are still working a lot with a community to get it to complete graduation, but really We just wanted to take this time to say thank you to Actually you the the Not just the code contributors, but the people that use Falco the adopters and everyone that made it actually successful And not just a simple open-source project that someone at the company started one day. So just stay tuned for Graduated the version Well for a graduated in the CNCF Falco but anyways Falco is not just talking about graduation Falco the Falco project has been working a lot to Be easier to use for everyone to have better detection capability and to have better security and and all these things and I would like to to ask Melissa and to give the word to Melissa who has worked a lot to About something that has been a big Point that people needed to learn about Falco that is the rules How can we make them easier to use and easier to contribute to? Thanks, and I also have more thank yous here So we are very grateful to our community for contributing nearly 90 Falco rules That's pretty impressive And we're also happy to announce that we have introduced a formal rules maturity framework to make it easier for everyone to Navigate and use the Falco rules. We also added a new guide to our website on how to adopt Falco rules in production We hope that all these efforts promote further deconsistency ease of use Adoption and innovation so that everyone from Experienced security engineers to newcomers to the field can get the most out of the upstream Falco rules Falco rules are now split into different files according to their maturity level The most mature and stable rules are included in the default Falco package They at rest product threats provides universal system level detections and follow best practices for optimal robustness Incubating rules still at rest relevant threats They provide a certain level of robustness and you can say that they cater to more specific use cases Sandbox rules are more experimental. They can serve as an inspiration and they also adhere to the minimum acceptance criteria Occasionally we do deprecate Falco rules Sometimes it can be confusing to understand why one rule is considered to be more stable or more relevant than another rule Therefore, we would like to emphasize that this is a best effort in our part and ultimately you have to decide which Falco rules are relevant to your environment and most likely you still have to create a lot of custom rules Falco now also has a formal rule style guide The description needs to be a little bit more informative than just one sentence and also include some tuning advice The condition statement needs to be syntactically correct for performance and consistency reasons We ask you to first place the event types followed by all of your positive expression filters and lastly you would place your exclusions such as tuning lists or macros Falco's output fields are now also much more consistent across rules And I would like to draw your attention to the special placeholder field container info If you run Falco with certain command line flags, which we go into more detail in our guides It will automatically resolve to the most important Kubernetes and container fields such as namespace podname container name tag and please keep in mind you can always manually add any field that we're supporting The tags property now also includes much more information again here We see the maturity level but also the minor attack phase and if a rule is applicable to a compliance use case It also contains new tags in that regard The main message of the slide is that you have to allocate a lot of engineering resources Not just to adopt the upstream Falco rules, but also create your own custom rules So basically expect that cycle of experimentation deploying Tuning to never stop and it's all about finding the right balance between the scope and the noise And also keep in mind that every Falco release will contain new capabilities That will empower you to create even more powerful Falco rules To wrap up the Falco rules update by default Falco matches the first condition statement that evaluates to true Sometimes rules they can overshadow each other now You can choose between the default first match wins or match all Falco rules Most likely we'll probably have a little bit of a performance hit here Talking about performance security monitoring never has unlimited security budgeting Therefore we take performance very seriously and there have been a couple of exciting updates in that space as well Falco CPU usage is highly correlated with how massive your servers are and the number and frequency of System calls under the hood Falco monitors all system calls from each of your Falco rule But we also add extra system calls for Falco's internal estate The default configuration configuration is a very conservative now We provide you as an end user full Overwrite control control so you can tailor Falco even more to your use cases and most likely you will be able to reduce CPU utilization Talking about CPU utilization Falco now natively supports a very powerful metrics framework Besides the traditional CPU and memory usages. We also expose more sophisticated metrics such as lippy pf stats or all of our internal event counters or drop counters And we're also very proud to announce that Falco has been selected as the first project for the CNCF tech Environmental sustainability and green reviews effort these efforts involve testing Falco Against a CNCF host a test Kubernetes cluster with synthetic workloads. That way we will be able to report how Falco's usage Usage changes between releases and also how much Falco was using on that particular test cluster Falco's kernel driver modernization marks yet another milestone to celebrate We could probably give an entire talk just about the modern ebpf Therefore, I'll keep it brief in the past. You had to compile Falco's kernel driver for each kernel release That means that every time you upgrade your kernel You had to make sure that the new kernel driver was available Sometimes this would sacrifice production stability now with the modern ebpf driver. None of this is needed anymore In fact, you don't even need to pass the kernel headers when you compile the driver And all this is made possible by the compile once run everywhere feature your kernel needs to be greater equal 5.8 or Support have back ported btf support and in summary This is a real joy from a dev ops and testing perspective and Luca has more to talk about kernel versions Thank you, Melissa and Yes, indeed. It's it's great to have another option to to run Falco in your kernel, but Colonel even being Relatively happy about the fact that we think our software works on most of the kernel is not that easy If you were here last year in Detroit when we were talking about Falco First of all, you were a hero because I think we were talking on a Friday afternoon So in that case kudos to you. Absolutely, but what we were saying among other things is that When we you try to test something like Falco that needs to operate at the kernel level plus with low-level user space component you find the lions and dragons in everything that you want to test and Specifically, we have improved so much in the last year and a half I think on testing but if we want to think about what are the dragons that you find that when you try to actually test something in the kernel It's mostly the fact that when when you try to test something in the same system that you're instrumenting You're going to have some interesting challenges to face and and not only that if you think about writing kernel code Pretty much from a developer standpoint in Linux. You can think about the Kernel community of having one rule the fact that they don't break user space, so they don't break the Boundary of the system the system call interface, but when you are inside the kernel Between every version of the Linux kernel plus the vendor patches that also exist You pretty much can see everything changed and that's normal So from our standpoint where we have to develop a module and the BPF probe We need to make sure that our software works with a kernel that many people have We would like we would love to have it working with all kernels, but before Melissa also mentioned the fact that we love our new eBPF probe that doesn't require you to build and doesn't require Us to build a specific version of the probe for every system by using quarry But eBPF poses an additional challenge that's called the eBPF verifier Which is this piece of code in the kernel that will tell you if your eBPF program is safe enough to run and That thing is also fragmented because every distribution has its own Version of the verifier and every kernel version has its own version So something that could be safe for my kernel could not be safe for yours and we want to test this This ended up as a very cool project that Has now that we built with an entire testing framework We have Something that is able to orchestrate the virtual machines to boot directly Small machines with a lot of the custom versions of the kernels based on both distributions and version Alongside the developer experience that should be good enough for people contributors and maintainers to be able to reproduce the bug and to be able to fix it So it would be I think we definitely don't have Enough time to talk about how this is actually designed So I would encourage if you're interested to read our blog post up because we we added a lot of detail But the the result to us We we have a certain sense of beauty for us It's beautiful because we can get more than 20 versions of kernel and and the distributions tested continuously with two CPU architectures and a lot of different configuration module BPF and Verifier and and everything there. Thanks a lot for this project for a CNCF actually for giving us access to an equinex metal cluster that is a bare metal cluster that allows us to actually learn virtual machines So but Falco is not just being tested but also wants to improve its ability to detect the things and That that ability can be sometimes something simple as that What would happen if I run this command? What would Falco say about this command? I mean if I I just sim linked a binary into another cat sim linked into dog And if I didn't show you the first line It would be fine to answer in the way that Falco before 036 was answering So that's technically correct We have a process name that is dog and XC and XC path that are the same that that's right But if you are doing security detection with that you are probably interested in knowing what if there was a sim link you probably want to see it and As a feature it since Falco 036 we improved the ability for Falco to Resolve these cases and so you can see that we changed the meaning of the XC path field to something that is That according to community feedback is actually far more intuitive for security professionals rather than what was displayed before so we now actually have a Distinction between dog and cat in this example and also another Comment that we had is for another thing that it's even more subtle in the in the kernel Is the fact that if you have ever run? PS tree on your system for example if you're running a container or something you will notice that In in the Linux in a Linux system every process is organized in a hierarchical fashion. There is the one parent of them all that is the init process Can be system D can be depending on the on the system that can be something and everything is a descendant of that and Here I just ran a container and everything was nice in container land There was a container this shim and then there was an engine X that I executed And then of course I went there with my user terminal and I tried to do a docker exec And this is just an example of something that that can happen What do I expect with ps3? Of course, I see that there are two Processes sibling to each other that are actually running in In in our container, but that's not actually what happens in when I spawn these these exact because what happens in that in that case is that Actually, there is another process that is run see that is very very used in a lot of container engines and this process will actually exit and has Child that is my shell in this operation There are certain an amount of re-parenting operation that happened in the kernel that made it so that Falco from its view in the events Would sometimes create a broken link so at the detection time when you wanted to know what actually spawned that shell You couldn't know because the Falco didn't know the the parent in this case in this version We made we did a lot of work to make it better to have better Detections so that when you try to print the field about all the ancestors of a process that you actually have something that resembles more what you intuitively would expect by running ps3 and Falco and as as usual I always love when I can improve the security of the Falco project itself This is a never-ending work. I will not be done anytime soon and neither will be all as maintainer but anyways this time we Worked a lot on some supply chain security topics We created the signatures for not just container images But also plugins and rules that we distribute for the to do this we use that technology which is a cosine which is a pretty popular nowadays and for good reasons I think and Also, we added the distal as image so that you can actually have a Falco image that runs in your system With the zero CVs. That's because it has much less things I would love to tell you all about this but in the maintainer community We have Carlos who is a maintainer for some of these cool projects that we have used to build upon so Carlos would you like to tell us how we made Falco better with with this? Yeah This is more of a project level updates like we are signing The images and blobs across the entire org for Falco It's not only for Falco, but all the other projects as well the Falco CTL Falco South Geek and others Like before I continue like I just to understand like who knows what is the six Torco signed to Can raise your hand Okay, I'm gonna Give a quick an explanation cos sign is a tool to sign image and blobs and that use the six star Public infrastructure that is running almost like for now. I almost two years now in a public space and the six star organization have is backed by the Linux foundation as well and With that we are signing our Let's go here We are signing not only the image you guys are consuming but also the the blobs and we are now signing also the plugins and rules and if you use Falco CTL to install your plugins it's a Falco CTL also Before installing that in your cluster it can detect check the signatures if that the plugin you are told to install It's value. It's signed and verified all the credentials for that with that like you can use Falco CTL to Install the the plugin and verify the signature, but also you can use cos sign tool to Verify the the signature for the rules and the plugin itself You can take a look on that on the blog that was written that explain a little bit more and also like I can Talk about that later on as well The next one that we like it did I think this is like a huge upgrade we Are using the wolfie wolfie. It's a open source tool like the org that produces image and package and the wolfies like we call it the undisturbed Distribution like because that there is no kernel in the image It's just the package and that is like a read we can call it like that as a v2 of the distro less And the Falco now we are building that in the distro less way as well that makes the The image itself like is smaller and like as look at mention like with near zero CVs And if we find any CV we can like rebuild that like pretty fast as well and you we are I think we are publishing both Falco distro less than the normal Image that everybody was consuming the past but you can try out the Disney one here and provide feedback to us if it's fitting your workload space and Last but not least we also did like a lot of documentation improvements. We revamp our Website like you can check in the folk org. There's a lot of new stuff there like there's a lot of How to that is like we are glossary and a lot of things that Examples and things about rules and plugins that you can use if you go there and find the That is like not correct or missing things like we are waiting for you Like to open issues and even PRs to fix that for us as well and help us to make it better Okay Thank You Carlos. So Speaking about the plugin feature which was mentioned a couple of times already I wanted to give like a bit of retrospective what happens here so we first launched the feature almost two years ago, I think six releases ago for sure and the Main goal of that was to support the new kinds of events in the specific We leveraged this feature to implement cloud logs threat detection So for example, the first use case was AWS cloud trail Kubernetes audit logs, and then we got to get up an octa Plus extracting new data fields to be used for in folk rules for those kind of events that open many new use cases for the project But at the same time the core goal falco is still in point and system security So this led to the intention of renovating the plugin system to support Most of the feature that the core code of falco already do and basically become a real extension of frameworks for falco itself basically Speaking about capabilities that we added briefly. We now are fully compatible with system calls and kernel events So now you can create plugins that hooks Bigly hook into the system call event stream and are capable of generating those and extracting new data fields from those Plus they can do stateful detections So just falco itself and keep a maintain internal state and share it with falco and other plugins as well For example now a plugin can access all the thread information and you know the process life cycle information that falco collects natively and also reach it Another minor thing that people don't know we do under the hood is that we communicate for example We the container runtime and collect information as synchronously and put it back inside event stream so that falco can update the state with that plugins can do that as well Doesn't mean anything I think Without few examples of what the new use cases can be so think for example Kubernetes meta data enrichment whenever you receive a file collards You can have information about You know the Kubernetes objects that are related or at least something about the cluster or the deployment So this Implementation is very historical and we maintain it since years. Let's say, but we have Performance concerns the scalability concerns on big clusters and says falco is still the probably most widely adopt open source tool for threat detection Kubernetes we wanted to make this first class it is an integration better and reinventing this as some challenges because This needs to observe all the system calls related to process life cycle It needs to maintain internal state about the Kubernetes objects and it's observed and plus He needs to communicate that information to falco as synchronously after communication with the API server Look, this is exactly what the plug-in system offers nowadays And this is exactly what we are doing in the re-implementation phase every design phase of this feature So for now on we will drop in the next falco release We will which will come in January the legacy implementation that we have and we are in the development process of creating a New more scalable and more optimized one Leveraging the plug-in system if you don't want it you can not use it basically and if you want to use it It will be an add-on that you can use on falco or dynamically another thing Which is actually proposed by Melissa is a statistic around time anomaly detection framework Which is in the working and under proposal Which will basically allow you to see what's normal in your workload Let's say automatically and dynamically and that will probably leverage the plug-in system as well So the long-term vision of this is for the core of falco to become more more Minimal to reduce to the bare minimum the footprint it has on your system for the whole case at least and then have a Wide ecosystem of many Individual and minimal plug-in play modules that you can just add on depending on your use case The end goal of this is customizability, which is the one thing that you know adopters ask us the most given that The range of use cases of falco is very diverse in the ecosystem Non-technical updates involve the plug is now being signed like mentioned before Which is like a very good thing for our automatic distribution system from us, you know a trust perspective And now we also made on top the top top of this the publishing and testing pipelines in our CI more robust and resilient So now we're capable of You know retrieving errors very early on and you can expect whenever you listen to our latest updates with Falco CTL That everything is sound we are we are much more reliable Then the C++ SDK for writing plugins has been reinvented as well for performance basically so now this maximizes performance buying lining code whenever possible and Given the control on memory and everything the C++ gives you this is also the SDK that Has enabled access to all the new capabilities of the plug-in framework. It's still not official in you know our Officiality framework we added in our governance recently, but it's heavily optimized and fully functional So give it a try and last but not least we have a new plug-in our family Which was actually contributed by a newcomer in our community. It was very cool and now he committed to maintain it From now on which involves threat detection for GCP audit logs Now this one I'm pretty excited about so Falco participated in Google Summer of project this year and that these Definitely starts our first steps into the world of WebAssembly, which is not very common You know in the CNCF landscape yet. We we try to experiment to the best of our Of our possibilities So the problem we are trying to solve here is that Falco rules people like them because they're easy to understand and more You know much expressive but you're probably noticed if you try to write them that they're not that easy to develop Mostly because it's a try and error process So you use your favorite idea to do that you dump the rules in Falco for syntactic checks If that doesn't work go back to step one assuming that everything is good Use them in Falco trying to see if they trigger in the situation that you expect them to do so If not go back to step one and falco is the one tool in the middle that allows to do everything through the CLI It's not very handy and this I think sort of makes the learning curve not favorable for newcomers So now we are capable of running Falco natively in the browser because WebAssembly is a completion target that we officially support in our packages and We gained a new maintainer in the process and this solution is a web ID that is now deploying in our community And it's totally back-endless because Falco really runs and you can try that in the QR code with your phone It works on mobile too The long-term vision for this project is to further integrate it in our website and use that for more in-depth documentation snippets and tutorials For the future What are we focusing on for you know the near-term roadmap as I said the integration with the Kubernetes We're trying to put some love in that because we recognize the importance that we have for us as an integration and The redesign of implementation of the clients for metadata enrichment is the proof of that And then it the biggest concern that we have and goal is the roadmap to version 1.0 Which involves probably mostly quality improvement and optimization We want Falco to be reliable and to be as fast as possible to reduce the bare minimum they overhead on your system and all the things that I mentioned before are basically Some parts of this process, but that's our focus then threat detection We want the threat detection capability of Falco to be marked, you know more and more sophisticated at every release A few things are are really rementioned being mentioned by the others like the significant resolution for executions We now also care about file list Executions by listening to the MMFD Assistant calls and we are trying to go faster and faster in the development of these things and then he's of usage Working on the website trying to create more default use cases make life easier to adopters for adopters to develop Falco rules We want the learning curve of you know much technical project like Falco to be as optimal as possible for people Then please don't forget to reach out to us You can find all of us plus the other mountainous and all the contributors and people in the community in our feature channels In the mailing list as well, and we meet every Wednesday on the Wiki community community call we discuss You know bugs feature requests planning and all that We're also going to do a sign-off of the Falco book tomorrow From lores is one of the author and also the founder of the project historically. So feel free to find out and ask us about that Thank you for listening. I think it's time for Q&A Any questions Okay, thank you