 Good afternoon everyone and welcome to the Falco maintainer track. We have a full house here and five maintainers myself I'm Alyssa Luca Jason Carlos and Hendrick. We have a lot of content to cover Let's get started if you have never heard of Falco think of it as security cameras for your operating system To detect harmful behavior Falco uses rules to define behavior allowing you to customize them for your needs From a technical perspective Falco resides low understack at the Linux kernel layer using modern technologies like extended Berkeley packet filter EBPF, it's a pretty cool way of doing kernel programming which raises the question Why do we need kernel programming in the first place looking back to what EBPF does? It can attach to Linux kernel trace points to monitor system calls and more You may ask yourself what our system calls and why do we need them to detect cyber intrusions? So if you really think about cyber security if you attack The system like the way think of an operating system at the lower kernel level with all the applications running on top and There's an analogy you can think about it if you've watched stranger things on Netflix Imagine it like being an underworld and an upper world with a gateway in between so system calls are the API or largest gateway to ask the kernel for permission to interact with hardware such as accessing memory or opening a file from disk Falco hooks into system calls to detect malicious behavior and Send an alert from the precise location on the stack where it's the most beneficial to send an alert from and furthermore Falco lets you customize your deployment with your custom rules and that way you can determine What is happening in your environment? And we as maintainers we benefit from Falco to make our deployment more stable and scale it out more and There is also another aspect to it if you want to So besides system calls there is a lot more to Falco Starting from the past year Falco introduced a new plug-in system that basically allows everyone to develop extension for Falco and Basically adapt the tool to many more use cases. This mainly gives us the opportunity to pipe new data sources into Falco and write security rules on top of those One specifically important new data source type is for example cloud logging So that you potentially can use that to write security rules in services like Amazon cloud trail or Kubernetes audit logs Starting about examples of those integrations those two that I mentioned were one of the first ones We also have one for GitHub and one for from octa just to mention some of the few the official ones that we support And there are others maintained currently and developed by the community. There's gonna they're gonna be available in the near future Now Luca, can you tell us more about your experience about being a Falco maintainer? Thanks Jason it's really of course an honor and it's been awesome being a Falco maintainer and being able to maintain such as Sensitive and cool project and if there's one thing where I hope I'll be able to get this Super smart and cool people and help them and have the community have the users I think it's probably sleeping a tiny bit better at night Not by just being boring but by caring about the security of the Falco project so You we know like Melissa just said that Falco needs to monitor system calls needs to run with high privileges and this is Something that you need to do. You cannot really get around this problem. So We need to care about the security of the Falco project itself of our code of everything more than maybe other workloads that we are running on our cluster because You know one vulnerability in Falco could be much scarier than a regular vulnerability. So I am very happy to help lead to help Organize and to contribute to every security project that we have going on in Falco for example We just concluded an audit with Quartz lab that's Security company with security experts that did a great case study where you can see how First of all our code base is from a security standpoint and on the other hand You can see how you we can apply even more static analysis And we have now like 20 checks but we can add more and how we can build dynamic analysis Which is super challenge to do if there are some dynamic analysis experts and enthusiasts here I really like to talk because that's going to be a very cool project And you can see the case study in the in the report that that they made for us. So And and of course, so we are learning really a lot about every type of security We all heard about supply chain security, of course, we know And we want to get Falco to get the best and the most up-to-date supply chain security standards and tools in everything that would build and we we get I can say I'm not a super expert here But fortunately we've got plenty of experts in the community that are currently helping us to get the proper signatures And all the proper standards set in place and speaking about learning something new We've got here our latest addition to the maintainer the core maintainer team that is Melissa that already Introduced herself. She has not been there for long, but she I'm sure she already learned a lot So Melissa would you like to share with us something cool that you learned since being a maintainer Yes, so as the news maintainer having come from being an adopter I'm still a power and user but now I'm also exposed to the challenges other companies are facing and Bong these challenges is to need to scale up a lot more amidst remarkable Variation and workloads across companies the upcoming falco release introduces new mechanisms to select the system calls that you're interested in Specifically falco now only traces the system calls from each falco rule Along with a set of additional system calls that you need in the background to correctly monitor spawn processes or Network connections and create falco state Notably the state engine is used to retrieve historical parent process lineages in real time and this is a big deal to write powerful detections as A result of these optimizations across multiple areas we as end user benefit from a more robust Resourceful and customized production deployment Okay, what is the most impactful thing I have learned I just talked about it in the next question if we change the slide Jason, how do you spend your time and balance priorities? I Will spend my time. I think things changed a lot since I joined first the community about two years ago We now have many many more contributors and a bunch of more mountainers from different organizations as well Working full-time on the project I usually split equally my time between writing code code reviews and taking part of the decision process And I personally maintain most of the code repositories of the falco project But other mountainers just prefer to just commit to my repository depending on how many time how much time they have To dedicate for the project question is how our decisions being made we usually go in a synchronous fashion So we either go between private communications between mountainers Most of the stuff happens on GitHub either public GitHub issues and GitHub PRs talking with contributors We usually gather some feedback on the community channels such as the one we have on slack When thing gets more serious because there is a you know a more relevant work to do we will go for a written proposal and eventually for a voting when you know it's necessary and We also have opening initiatives and working groups in case of all those working streams that require more than one week to be completed examples of success of this kind of initiatives where the secret rotation that we went through over this winter after a SQL CI incident and The supply security chain Sorry supplies it supply Security supply chain. Sorry working group that we are having right now How can we make sure that everyone's opinion is respected? So we do have an open governance and that we went through like a big review of those documents back in August 2022 We tried basically to publish the documents learning from past points of ambiguities trying to gather feedback from everyone in the community And respect everyone's feelings and opinions. We also got a lot of help from the top contributor strategies so huge thanks to them for this and Usually we go for lazy consensus whenever applyable whenever there are cases of disputes or you know, because the decision to be made We go for a solid scheme of voting About the road map of the project. So most of it happens on get up So we use get up milestones on Falco and all the core Repositors of the project to decide and track down, you know among mountaineers What are the issues and the PRs and the features that are going to fit in the next few Falk releases? We're also about to establish Monthly mountaineer calls so that the biggest stakeholders are always aware of the you know, the current problems and decisions and Yeah, that's pretty much it Carlos. Can you tell us more about how can others be involved in the project? Yeah, sure as Falco is a CNCF project and we have like a community calls and other stuff But how will you if you are interested to work on the Falco project how you can join? I would say like my experience when I start contributing to Falco was like joining doing the first place the community calls community meetings every week and Then start like checking the repositories and see which one like Falco have like several code repositories It's not only that Falco itself is written in C++ But I don't know C++ but that is all the projects inside the Falco ecosystem that is written in other languages that you can help Then you come to me say, but I don't know how to code like we don't you don't need to how to C++ or go or any other language you can help in triage issues in the Falco Issues we have like in bug like the reproduce the bugs they have if you are more Expertizing in testing area you can help us to test Falco itself and the other tools You also can help in writing like checking the documentation and see if that is like up to date or not if it's Make it sense or not and then you can help on those parts as well There is several areas that you can help and it's not only by doing code You can do in other parts as well and also you can join this this like channel and help the community also Answering questions and all this stuff Next one What's the benefits to your organization for the perspective of a maintainer Henry? So thanks Carlos. I started mid last year opening the issue for the Falco lips community in order to get the BBBF drivers enabled for Architecture code as 390 as 390 X or chord these systems links one just a quick question to the audience Please raise your hands. We have heard about as 390 X. Oh Excellent So based on this is was a really great point in time Because also the community started to work on the modern PVP F So this modern PVP F driver actually is using the compile once run everywhere Paradigm so we Improving this make it easier when it comes to deployment so that we do not read to need to recompile the eddy corner during updates, so it took me about half a year to Enable the PPF driver the odd one for our architecture because our texture has a really strict isolation between kernel and user space So I have to review each PPF probe read Function and to divide it is it a corner data to be read or is it data from users based to be read So I think that was for my side and for my company side as part of a product enablement I really create benefit the other thing. I think my Benefit from our company for my company to you are is about the gift bag from a technical point of view to have lots of reviews writing test cases for a modern PPF and Also driving infusing the S390 architecture in this project So Melissa, I have my FICO Drivers running what happens on day two first of all congratulations on making today, too Because if something goes wrong it goes wrong right after deploying so always remember to never deploy on a Friday Going back to day one setting yourself up for success requires clear thinking about what matters most to your organization And a close collaboration with your system admins or side real ability engineers It is very crucial to have meaningful resource utilization metrics in place such as CPU and memory usage in order to derive a shared understanding of the constraints in your environment number one priority is definitely to not break production and perform constant regression tests to improve stability and Catch potential issues earlier before rolling out to production as a maintainer Maintaining our deployment is now more simple because we upstream features that Support stability and performance such as the newly introduced specialized metrics around to work Falco needs to do to keep up with the currently event pipe and that way we ensure that we can use Falco switching gears improved threat detection capabilities are also of utmost importance to us maintainers and essential partnerships with offensive security teams and incident responders Provide valuable feedback helping and pushing us to go deeper and be more adventurous in detecting cyber threats in a more robust manner if you do not have access to such resources The internet can be an excellent guide to simulate the top cyber threats and that way you can perform and to end tests Another substantial amount of work is to operationalize Falco alerts with high quality incident response runbooks For the security analyst what matters most is to string key information down to what what's on top of their mind And this includes Knowing who owns the workload and in what context the alert triggered Falco simplifies these steps because especially for a Kubernetes Falco knows about Containers parts and namespaces and also the context around each system call For example that the event happened over an interactive shell or was a new binary involved and there is a long list of other examples So Falco allows you to augment each Falco lock already on host and therefore you can speed up the whole incident response Process which I personally think is pretty cool The next question we have is Luca. I think it's for you So what are recent wins to catch more bad stuff and to make it easier to use Falco? Thanks again, Melissa and well, of course I'm kind of an enthusiast when it comes to bad stuff So if I had the shell on one of your clusters purely hypothetically I Might want to you know drop my binary and do bad stuff with it I know I know if there's security guys around and security people They you will know that of course That's not the only type of attack but as a defender I think that's one of the first things that I want to look for if someone drops a binary and then runs it immediately sometimes it's even Regular is even regular workload But most of the times it can be an attack and I'm very happy to share that We have two more ways to detect this situation because if you try to do it You will notice that it's not that trivial to detect but we have several ways in Falco and out of the new ways one is Involves the overlay file system of a container where you take a look at that writable layer That's on top and the other takes a look at even the time where executive cutables are spawned On top of the is actually writable flag that we had previously that tells you if Binary is both writable and executable by the users that is spawning it if you know something about memory safety you know that there is a kind of a parallel with files and And yeah in this way you can combine these signals and you can combine these rules To get whatever you wanted to find this little happy dumpster fire situation before it it can escalate and Also, I really like when a Falco can get a tiny bit easier to use so Before we had to pretty much when we wanted to ship our rules to our fleet of Falcos We had to either roll out our custom solution or we had to restart the Falco pods and change the config maps We know that since Falco doesn't really have a back-end we we cannot push rules but we wanted to make it a little bit easier and We decided to support the way that we use that to distribute to distribute container images That is OCI artifacts a lot of science if probably CNCF projects are using those and they're pretty handy They support signatures that they're they're rather cool So now you can have a workflow workflow that pushes from your CICD like we use get abections But you can use whatever you want your rules to your container registry and then have Falco Automatically pull those rules either at startup or even update it if if you want From your container registry being that public being that maybe air-gapped or private It doesn't matter as long as it's a compliant computer register like way or get up container registries or many others It's it's going to it's going to just work So we are adding a lot of cool stuff and a lot of support for Very very cool things and the the thing that made Falco easier to use that I like the most I think is of course the also the new ebpf probe that Henry mentioned before but Now we support a lot of things we support the kernel module for all kernels We support the modern probe that you just press a button and it works But how are we even testing all this stuff and the Hendrik mentioned before that? He is interested in that and is helping a lot the product with that Thanks. Thank you Luca and indeed with my work It really comes to testing so I'm happy that there are a lot of couple of CI changes happening in the last couple of weeks One thing is that we introduce the end-to-end testing frameworks to cover the end-to-end few as well as unit testing and As I mentioned earlier with the beginning work of the modern ebpf probe We implemented this call by Zeus call and always added a couple of test cases for this So this was really the the base for the modern ebpf driver and in the last I guess two months three months This modern ebpf test suite has been changed to cover the old ebpf probe as well as the kernel ebpf probe So we have now a combined test suite for every Zeus call that covers all three FICO lips drivers So with that said we also faced a couple of inconsistencies between those drivers and I would like to give you the chance and opportunity to help us here Looking at those issues and may become a contributor to the project so with that up to Carlos, please Let us know how FICO integrates with other CNCF projects Yeah, FICO itself like it generates a lot of information and we have another project inside the FICO Organization that is called the falcoside kick that collects and then you can output those informations to several Applications or several options that you can you can even output to the the alerts and message from I select channel to A Kafka queue you can do or whatever you like that is we support a lot of CNCF and NATs and all the stuff that you can connect all the information that flows To to the system you want to support and we want to collect metrics For that like let's see what's next in FICO that we are planning to do in the following months Yes, so for the next Farc release which will happen at the end of May So version 0.35 first thing you can expect is for FICO to be more configurable both from a performance And a resource huge usage standpoints So first of all as Melissa mentioned we will have an improved metric system for resource Utilizations, so you're gonna be able to a better understanding your cluster and nodes how good and how bad FICO is doing and tune it up depending on your needs Then the system call adaptive selection feature still by Melissa mentioned before is a huge performance tuning point That will basically allow FICO to just use what it needs in the communication and collection of data from with a kernel And that's tremendously impactful on many systems specifically on bigger machine Talking about the connection between FICO and the kernel. This is true nowadays as well You're also able to configure how big the size of the buffer shared between the two is so many tuning points Also, we put a lot of effort of the on improving quality and testing both on FICO and on the FICO libraries Let's say that both from unit testing and Regression testing as well. We want to make sure that each Falk release from release to release keeps being consistent with the UX and expectations for the user and at the same time the Automatic rules distributions and plug-in distribution system the look I mentioned before was rolled out initially in 0.34 41 Actually 30 34 in the past release, but it's gonna be more frictionless more reliable in a Falk Falk alter all to reload feature Will be, you know exempt from some a couple of bucks that we found during the process Talking about the BPF the modern BPF is gonna be rolled out officially in feature party with like all the other drivers Starting from the next Falk release. It's already available for you know, you guys to experiment It doesn't have all the system cost reported by the framework yet just essential ones starting from 0.35 You'll have basically the full package. So I mean, it's not about the tested But feel free to use it if you have a kernel 5.8 plus because it's very handy just a plug-in place solution and then On the Mmm Planning side we are working on the Gita projects So one thing that we are missing as I said before is that the information gathering is pretty much distribute over get up So we are actively working on improving that by having basically Gita projects as a single solution It is a solution for road mapping useful for maintainers to you know manage what's happening in a game of release and for contributors to Basically better understand what are the expectations for the next Falk releases and what is the expected timeline? That's pretty much it from our side. So Yeah Just fast forward to slides one more One more. Okay here. So this is a call for we need more contributors and people who help us So let's just go through what the five stages of debugging look like in Falko So, you know, like the stage one is denial that works on my machine. It works on my kernel I don't know and this happens really a lot and then you start maybe getting a little irritated But the next phase the bargaining is very interesting in Falko because constantly we have to decide do we need to worry about this now or can we wait until later and You know the Linux kernel in general you kind of like maybe start talking to Linus and your head And you're like why did we need to split system calls and to enter and exit events anyhow and the next phase Can be rough. Should I even be a kernel developer? Am I happy so we go through that phase a lot But I really have to say like once you get to the final stage and you find a solution And it's all working in production like that feeling is actually pretty amazing so I hope that more of you kind of want to join us on that and and just The areas we need more help. So Luca and I were very focused on Offensive security or kind of better threat detection capabilities so I definitely like would like to take a moment to call out that we need help there because as of today Falko can capture You know behavior around to entire classes of vulnerabilities and security threats But there are still more gaps and hackers can get around it and then we need more help for Performance scaling up a lot of servers these days people have have 64 CPUs But the future is gonna migrate to 96 CPUs. So so that's a huge event pipe We have to deal with so performance is another one and then as Carlos mentioned also non-technical contributions are also very welcome and Maybe you guys have a few more areas we can use help up One thing that I wanted to add it which is a super valuable contribution is also sharing user stories So if you let us know how you use Falcon production without your ping points Where are the things that you know put you in doubt or that you struggle with when doing the deployment or when you run it It's super helpful for us even more helpful than sharing like a bug or an issue So just feel free to reach out for questions. Don't be shy It's gonna be very helpful for us as maintainers to better improve the project and better, you know Program what's gonna be the plans for the next few releases? Yeah, I just have Rate rate again like I would like to see you some of you are You all in some community meetings then we can talk if you don't want to join the meeting just a ping us in this Like I'm happy to help you and then they're happy to onboard you all like a fast and the focal community Time for Q&A Any questions? Yes, thank you I think yeah Yeah, thanks for being here and thanks for talking with us here So we're raising a fuck right now. We have Some clusters in Azure and we have some false positives Well, the reasons might be that because we like to say yes to every single rule But I was wondering if you guys have any general advice or I mean kind of community resources Or tips about how we can talk about those kinds of things common false positives So, yeah It very naive my response, but we do provide a default set of rules Which is probably the one you're using plus some custom ones I guess right so yeah the ones we provide of our of course general purpose It cannot be like specific to any very specific use cases So what we usually suggest is if you discover you have a false positive We have plenty of lists and macros that we provide in default rule sets that can just be overridden or you know extended Which usually happen in a form of a loud list So you can basically start excluding values from the fields in the check and allows you basically to define some Behavior that you expect and reduce that degree. There's also exceptions. You can you know customize the rules That's the strategy if the rule is really too noisy for you And that's some legit behavior in your environment that maybe that the rule is not for you You know, I mean that that's the principle in general keep it simple Otherwise focus that's becoming too noisy. There are options if it is really too noisy like a rate limiter or something like that But yeah, it's a there are options. It's a it's a matter of configuration. Basically, and I probably have some more summer tips, too So what I mentioned before that it's really crucial to be very clear about what's important for your organization So very often maybe you only want to alert for production namespaces And since you can add this information to each Falco log you can already include it in your filter Or you do it later in your data like I don't know your setup exactly And then also parent process lineages That's a good candidate to tune your detection and also see if it happens over interactive shell access or You know if other rules were involved But now we're talking about that you have to do post-processing you know data lake and do a few correlations But maybe to also give an outlook to what's gonna happen in the summer So we definitely want to explore options to do more on host anomaly detection Maybe help with these use cases where Falco learns over time and kind of like uses probabilistic data structures and that way reduce the data volume But those capabilities are a bit a little bit far out, but we're thinking about it Thanks Linute I call I have no experience with it But I was just wondering if there is something like a new CVE pops up Is is your team then going to catch it to write rules so that the CVE is now caught Is there such a concept? Yeah, great question. So Actually up until last release it was possible. Of course, it's always possible to write the rule So you can have the rule of course if the vulnerability is something that you can catch via Behavior events remember first of all that since you're new to Falco I want to Remember that Falco is a behavioral tool. So it sees actions that happen. So it's not the vulnerability scanner It's not software composition analysis. It's it's It's something like that some vulnerabilities can be caught In this way, you can always write the rule and since this release it's much easier to Deploy it on your Falco fleet right now the default the Falco rule set that doesn't get updated for every new CVE mostly because It's the it's the rule set maintains by a maintainer and maybe that gets a lot of false positives And we don't want people just running after us because of tons of false positives because of that CV detection But still we have introduced the possibility of updating the rule set really quickly So of course right now you will if something new and important comes out You might find a post in our blog or something that says hey if you deploy this rule you are going to catch at least some instances of this vulnerability and With the with the deployment work flows that we've seen we should be much easier for you to just get it out And detect what what you should you should have? Thank you very much. Thank you We have a question there Is there any updates or anything you can say about the user space drivers? Like are you planning to work on those or are you just leaving that as it is? So regarding so just a second if I want to if I can get to your question, right? So if we can collect the data Cisco data from user space, right? You you had a driver before you archived it. Are you gonna pick that back up? So so yes the so first of all Falco became more modular So now it's got a lot of different sources of the system calls that it can get We had before the and that's an archived project that the Falco Corp maintainer team doesn't contain anymore Something called PDG that was able to Take system calls pretty much with the user space hooking technology using P trace That is cool, and that's a cool example and the ingestion part still works, but it's really slow for For you know real workloads like if she tried to run that She would say that Falco is no good for her, but but yeah We we had another user space application that can do that Efficiently that is divisor that is a system of its own But it has an efficient way of shipping Cisco's without a need Needing to P trace and so that works because it's optimized enough We currently don't have plans as maintainers to create a new, you know, we use a space hooking Engine that is that is performant because right now It's not in the current roadmap But if someone wants and has a need for that we are more than willing to collaborate Like Carlos mentioned this there are so many cool project that spark from the community and from experts Maybe someone knows how to do user space hooking in an efficient way, and we can help to get each other to get that Thank you. Thank you any more questions. I think we're right on time. Thank you all