 So, hi, my name is Chris Nova second time here at FOSDEM presenting I gave a talk last year show hands who have seen it. Yep. It was a good one We explored well y'all ran the track. So thank you, but we explored some anti-patterns and some exciting things in kubernetes Since then kubernetes has grown a lot. I've grown a lot and the entire cloud native ecosystem has also grown tremendously So we're going to be looking at some more concepts tonight something that I've been thinking about and studying for about the past six months And we're going to look at some cloud native computing foundation open source tools Including kubernetes including the open policy agent and I'm going to try to be diligent about calling it the verbose name Open policy agent, but you might hear me refer to it as OPA or OPA as well And some other exciting tools in the ecosystem including the Linux kernel So to start off shout out to my friends over in this section who gave me some delicious cookies and chocolate before I came on stage And also we have the two Falco maintainers here in the front that have some Falco stickers and princess and matcha have some over There so throughout the talk if you see stickers come your way Feel free to grab one and stick it on your laptop and you're going to be learning more about Falco and these other projects tonight So the first thing I'm going to do is I am going to open up my slides Okay, so this is our first slide here So yeah, it's called fixing the kubernetes clusterfuck Which I think is a funny way of basically alluding to kubernetes is complex and it's complex for a good reason And because of this complexity it actually is a very powerful tool Which is why I've been working on it, and that's why I love it so much And why I've been so diligent about being involved with it and using it So in a weird way this complexity can potentially scare folks or cause problems But we're going to be looking at concretely some ways how the complexity particularly around Security is something that a lot of people that I've noticed may not be necessarily an expert on I don't even know if I would call myself an expert But I've been studying it for for quite some time and I'm gonna share with you today everything that I've found So yeah, I wrote a book called cloud native infrastructure Which is how I got into this whole kubernetes thing and one of the things that I noticed in kubernetes that hasn't really been solved Is this concept of security and like what does security mean to me as an infrastructure engineer? It was basically like I don't want anything happening in my system that I feel like should not be happening or that I don't know about or have Visibility into and I would like a convenient way to control that layer of security So recently I've become a maintainer of an open-source project called falco And I've been maintainers of other tools and kubernetes and other projects I've contributed across the ecosystem for the better half of my adult life and All of this kind of alludes to this idea that I fancy myself a hacker in the sense that I see something I don't understand and I sit there and hack away at it until I I finally understand it So the two words I want everyone to think about today the first word is prevention and the second word is Detection and we're gonna really explore these two words and what they mean from a security context and we're gonna actually go through and do a live demo where we take a kubernetes cluster set up with kubernetes cops With the default configuration we're going to exploit the prevention techniques We're in other words we're going to hack into the kubernetes cluster live on stage and then we're gonna look at how falco was able to detect this malicious behavior and We're gonna look at how We can use what's coming out of falco to draft policy using preventative tactics downstream to prevent this from happening again Hopefully when I get done doing this you'll walk away from here saying as an infrastructure engineer as a software engineer as a General kubernetes user. I would fancy a cluster to have both of these for complimentary holistic approach to securing and understanding my kubernetes system Okay, so everybody this is the time where you take your phone out Everything that I'm about to do Including these slides including links that I'm gonna reference including talks that I think you should go see Including links to my github my Twitter. Everybody's getting their phones out now I'll get mine out just so that you don't feel lonely and Everything is there so if you go To github.com slash chris nova slash public speaking. I'm gonna do some remote command injection here by hitting the space bar And of course my internet's not working Hold on no, it's okay. We're gonna need it. I I don't use the phosom Wi-Fi. So give me like two seconds But anyway, if you go to this website at the very top I changed There it goes. Um, I Changed the the title here to go to the actual The link in the repo that has everything that I have checked out locally So if you want to go and follow along all of the notes all of the markdown everything exists here including the Samples that we're going to be going through tonight Okay So Let's go back to my slides here So the first word prevention Words that come to mind when I look at preventing unwanted behavior are locks, right? If you want to keep somebody out you lock the door It's very easy. It's low hanging fruit and most doors and most access to our systems have a concept of a lock on it If you look at Linux fundamentally, right? There's different ways of locking Either users or applications out of what we do not want them doing in the kernel show of hands here Who's created a user in Linux before? Okay, everybody at positive and just put their hand up Who here is written as a Linux policy? Set comp policy Set comp you pf policy one person two three, okay four Okay, so again if you go and you do some research here, you'll understand that We're preventing unwanted behavior or at least we're attempting to and that's kind of the lesson here If you did not want a user to access something on the file system, you could create a user change the permissions There's this whole fundamental paradigm in place that allows you to prevent people from doing things They shouldn't do you can also do this with an application, right? So set comp vpf actually gives you a way to go through and control which system calls an Application could or could not execute if you look at C groups, right in the Linux kernel You could define arbitrary limits for what you want applications that are running within the context of the C group To be bound to and if they violate this limit the kernels going to terminate the process So we have these fundamental paradigms in Linux that we're all familiar with and if you follow along in kubernetes You will see the kubernetes in cloud native ecosystem is following in the footsteps of the Linux operating system Set comp is to kubernetes as OPA open policy agent is to cloud native Or I'm sorry set comp is to Linux as OPA is to cloud native So that's this concept of access control policy enforcement We also have this idea of image and artifact scanning right so in Traditional ecosystems if you wanted to deploy a new application You might want to go through and actually look at the bytecode to see if anything in there looks suspicious There's a well-known set of libraries that are open source on the internet that you can go and you can actually assert your Bionaries against whether that's java bytecode or it's good old-fashioned Machine bytecode and you can actually see if there's anything buried inside of that that you potentially would not want there We have the same concept with images in the cloud native ecosystem the same paradigm applied in a different more distributed way Code reviews right so you and your team going and looking at what the actual code does is there any vulnerabilities? Are you catching your errors? Do you have exposed sockets? What happens if somebody floods the socket just being security-minded throughout your day-to-day life is another big thing that I've been obsessing over So these are all tools that you and your team could use to prevent unwanted behavior But as we all know Bad things can still happen CVE's still happen right there's no such thing as perfectly safe and perfectly secure and perfectly perfect Software right free BSD Linux kubernetes name an open source Projects Jupyter notebooks they've all had CVE's opened up against them They've all been exploited at one point and they've all been fixed, but somebody had to discover this first So this concept of detection is the scientific approach to looking at our systems from the bottom up Instead of from the top down so by taking things that we would otherwise be effectively blind to and Asserting rules against them and using those signals for data processing We're actually able to see things in our system that we otherwise would not be able to see and so detection is this Approach to looking at our system and saying 99% of the time it behaves in this way given these input signals But on Tuesday last week all of a sudden this happened and we have never seen this before and we weren't expecting this to happen And we can programmatically assert that there was we would call this an anomaly that there was an anomaly that happened in Our system and that is where Detection comes into play so some people use tools like observability to do this right so whether we're we're auditing Cloud infrastructure or the application yourself itself or the Linux kernel that you're running on or the Kubernetes audit logs We basically would just want to have visibility into our system with high cardinality across the whole stack We also look at things like intrusion detection Right there's been a couple of exploits over the past year where folks have found out that people scan images People scan PDFs And if you upload an image with thousands of URLs buried inside of it or a PDF with thousands of images buried inside of it You can effectively dock someone Intentionally or intentionally so there are ways of getting things into a secure system and you may not be aware of a certain vector So security is this whole concept of studying these attack patterns and the humanistic approach to how Somebody might think of being intrusive in your systems. And so I've done that with Kubernetes and I think the approach to preventing this from happening to securing this and do Detecting something that is malicious that could be going on is this word that I have been using that I would like to Start advertising pull requests accepted if you don't like it Called runtime security that is a hybrid of both the practice of using something like Kubernetes access control or policy enforcement To prevent unwanted behavior, but also understanding that in some cases that might not be enough So we can begin to use tools like observability tooling like Falco what you're going to see in a moment To actually audit the kernel and understand what's happening in our system And I believe that having both of these Creates a set of checks and balances where an operator or an infrastructure engineer could go in and not only prevent unwanted behavior But detect it and then after they've detected it go through and create new policy to prevent it from happening again And I think this is a complementary approach to understanding our systems and to security in our systems moving forward Okay, so I'll give you the the 30-second pitch on Falco Don't worry. We're going to compile it and actually run it so you'll be able to see Concretely what it does. It's a CNCF incubation project. Who here is ran wire shark before? Okay, I really wish we could have seen that but everybody in this giant auditorium just put their hand up So Laura Stu Johnny my boss the founder of the company. I work for Cystig Was one of the original creators of wire shark he has his PhD in Linux and He his original thesis to solving this problem of understanding our systems was that TCP is the fundamental packet of truth Right, it's the atom. It's the quark if you're into quantum mechanics of how we understand computer science As we moved into cloud native as we moved into computers We realized that the network isn't necessarily the ultimate source of truth anymore So what we did is we started to look at kernel tracing who here is familiar with kernel tracing Okay, so maybe a third of the room just put their hand up they'll put his hand up And there are a couple avenues for how you would potentially trace events in the kernel But the idea here is that if all software ultimately flows through the system calls in the Linux API interface By auditing these system calls at runtime We should be able to understand exactly what's going on in our system and gain otherwise Unavailable information about what potentially is happening. So this is where this whole observability thing comes into play So Falco has taken this enormous onslaught of data from the kernel globally So if you use something like P trace, I mean by definition P is stands for process It's concretely married to a process itself with a pit what Falco and what The cystic CLI tool does is it has some libraries that allow you to go through and globally audit What's happening in your kernel the two ways? We do this is by either running a kernel module or by using a newer technology called ebpf that allows us to implement Kernel tracing in user space so that we can understand what's going on in the kernel What Falco does is it takes this stream of data these signals from Linux and it asserts them against well-known Anomalies right what happens if somebody executes open the open system call against Etsy shadow Do you and your team want to know about that? I probably would And if you're a savvy I got to use the word savvy in my presentation If you're a savvy Linux user you could probably find ways of doing this on a system with the system and user space Not being aware that you did this but the kernel ultimately would have to execute the system call So by going to the kernel level you're able to see things you would otherwise be blind to So again, it's an evolution of wire shark, but for the kernel and this allows us to begin kernel tracy So how does it work? So Falco takes not only information from the kernel But also other bits of information from a containerized system as well And we're just using what's going on in the kernel to tell a broader story about how we would potentially be detecting anomalies in Kubernetes I like how people are taking photos of my very professional ASCII diagram on the screen here I mean come on. I went through and actually centered this with spaces and counted the spaces this took like at least 20 or 30 minutes So on the left side here We have system call events is what I basically just described We also can parse other bits of made it information from our systems as well who here has ever explored the Docker socket Handful meh Yeah, another third or so of the people here If you can actually go and connect to the socket you can actually get all kinds of interesting made it information about the containers That are running on the system Kubernetes also gives us some visibility as well. We have Kubernetes made it information What is the name of the pod? When was it started? How long has it been running? What namespaces is the pod running in? And we also have this new feature in Kubernetes called Kubernetes audit logs that basically give you the who what Why and where of something happening of some mutation in your infrastructure in your system? So if you ever go and you know followed the tutorial online and blindly download a YAML manifest and apply it to a cluster and just Kind of hope it works, which we've all done before I'm sure What's actually happening is you're mutating the data store in Cooper Denny's and then all these little controllers come out And they go and they try to reconcile this new configuration that you've pushed to your cluster and if you're lucky it should work So by having the central database we're able to tell an even broader story about what's happening in our system So all of this data comes into Falco, which is written in C++ and it's highly optimized for efficiency here I mean we're dealing on the order of magnitude of Millions of system calls potentially a second coming up from the kernel and how it comes up from the kernel is over a Ring buffer and Lorenzo Fontana probably the most technical maintainer on the Falco project sitting in the front row here That's an inside joke of ours He gave a wonderful talk earlier today about eBPF. He literally wrote the book on BPF. There's and that The slide asked you to take a picture of you can go watch his talk and he goes into much more detail here But basically we have a 16 megabyte ring buffer per CPU running on our system I'll show you concretely in a moment what that looks like and We're able to pull these system calls up through that Combine that with kubernetes information combine that with the container information and then assert this against well-known security anomalies Once I don't allow anomaly is detected. There's a few things Falco at lots is do Fundamentally Falco is designed to be composable so you can take an output from it and you can plug it into anything you want The first one we see on the screen is gRPC This is relevant because this has allowed us using tools like protobuf to easily build clients and SDKs for you to plug Falco Outputs into other arbitrary parts of your system right now. We have rust go Python and if you would like to generate your your own pull requests are accepted We also have a concept of a web hook of actually going out and trying to send data to a configured web server And in this example that I'm going to be running today good old standard out Which we're just going to look at in the terminal here So again to summarize from the bottom up We have the Linux kernel on top of the kernel We have either a kernel module which will go more into what that looks like in a moment or an eBPF pro Then we go into our ring buffer that basically runs on that thin layer between the kernel and the rest of user space And we move up into user space where we have two libraries that are able to pull information from the ring buffer And then ultimately Falco is built on top of all of these libraries and allow us to interface with kubernetes in docker And actually tell a full holistic security story So to summarize Falco is a static binary you can run it potentially in a container It's written in both C and C plus plus we have rust go in Python clients And this whole thing has been optimized for speed get hub comm slash Falco security if you want to see more So let's talk about the kernel module So what this does is this parses system events So kernel modules were the our first approach at how we would go about configuring custom logic in the kernel There's a fundamental problem with this which is if you're running a potentially unknown kernel configuration Or if something happens on your hardware or something that you didn't plan for happens in your kernel module You can potentially crash a system Furthermore imagine imagine me a security engineer like walking into a company and saying hi Download our kernel from our kernel module from the internet and install it in production. We promise that's going to be a good idea So this is this problem Is why ebpf is so successful ebpf says we're going to take the The bpf Berkeley packet filter and we're going to go a step further and we're going to start to build more logic And more capabilities into this very old otherwise relatively unused part of the linux kernel And what we're going to do is we're going to guarantee a few things and particularly We're going to solve this kernel module problem of if you want to do certain things We're going to prevent you from being able to crash a system So we started to play with ebpf So we wanted this to do the same thing that our kernel module was doing We wanted to parse these system calls because we have found that this is actually a good source of truth For doing things like detecting anomalies And we also wanted to make sure that we couldn't potentially crash a system So because ebpf code is already pre compiled into the kernel you're effectively just telling the kernel to turn it on Right, it's just like javascript running in your browser. It's just as just saying you already have this logic Just please do this one thing for me instead of please run this logic. I wrote myself So bpf or ebpf rather It's unable to crash the kernel. It's effectively read only and it's not turning complete But you're still able to do some pretty powerful things with it and then once you get it from the kernel You can implement that in a turning complete language of your choice So if you want to look more go to the the open source project and check out scap.c and scap bpf dot c Who here remembers wire shot cap cap files? Same concepts, but with bpf and for the Linux kernel So earlier today I met with a guy Gress if you're here. Thank you for helping me out earlier And he helped me get opa or open policy agent set up for my demo And we're going to actually hack into kubernetes and then we're going to go through and use this to prevent my hack from happening again And we're going to run a series of experiments here So uh more on opa in a moment But basically it's a cncf project just like falco and it works with more than just kubernetes So it doesn't have to work for kubernetes although in this example we're using it and it was designed to basically just solve The problem of creating a policy engine that we could implement Anywhere so one policy engine to rule them all is basically what I think of when I think of opa Gatekeeper an open source tool is an implementation of this broader policy enforcement mechanism And gatekeeper specifically coupled with kubernetes and that's what we have running in my cluster so If you want to run something like opa or opa in kubernetes gatekeeper has sort of taken This existing more flexible more modular project and optimized it for the single concrete ease case of kubernetes Okay, so let's talk about my demo looks like we are 25 minutes into my talk So i'll probably do another 10 or 15 minutes here of this demo and i'm going to go pretty fast So i'm going to leave some questions at the end So if something doesn't make sense or if I skip over something Please either you know ask me afterwards so I can document it on the internet I'm sure you're not the only one about who have this question Or even put your hand up at the end and I'm happy to answer quickly at the end of the demo But what we're going to do is we're going to start off by showing you how we're doing some kernel tracing on my local laptop here So i'm running arch Linux. I have a fairly old kernel not too old But also not like brand new to kind of demonstrate what I would think most people are running in production And we're going to create a user local bin fosdem on a couple of different environments The first one on my local laptop and we're going to parse this using the kernel module And you're going to see the devices and you're going to actually watch me load the kernel module on my laptop The next one, uh, we're going to start falco with bpf and i'm going to delete the kernel module And you'll see the devices go away and you'll still you'll see falco still working dynamically Which is exciting because we didn't have to load anything into the kernel Next we're going to do this in kubernetes again, and we're going to do this by we're going to have a Cluster administrator kube config configured, which is basically like root on my kubernetes cluster And then i'm going to use kubernetes access control and prevention techniques are back To create a new Configuration that only gives me access to one namespace in kubernetes I'm then going to create a a shell in kubernetes privilege escalate through that shell Gain access to the underlying node get root access all of which that should have been reasonably prevented giving kubernetes are back After we do this, and i've hopefully sufficiently scared a number of people in the room here We're going to go through and we're going to look at the opa policy and the gatekeeper policy Of preventing this from happening again, and we're going to look at how falco the whole time Had every system call and it was able to tell a story about what happened and basically explain the threat model And the attack factor for what was happening in kubernetes okay, so Done with my slides So the first thing i'm going to do is i am going to Show you my slash dev on my file system here. Can everybody see okay? Cool change directory slash dev you can see here We're looking for it falco down here in these these devices And if you notice you don't see them So next i'm going to go to this directory In in home here, and you're going to see i have two pre-compiled Objects here one is which is a kernel object that we're going to load as a kernel module And the other one is just a regular old elf objects And we're going to use both of these subjectively as we start running falco And so What i want to do is i'm going to just run pseudo falco and let's see what happens Let me enter my Password here and you can see here We got an error unable to open device falco dot zero remember earlier i mentioned a 16 megabyte buffer per core I have eight cores on this machine. So we're looking for zero through seven right zero index Device files that do not exist So what we're going to do is we're going to in's mod falco Probe dot ko And if i list mod and we'll grep for falco You can see it's loaded And if i list slash dev again, you can see here We now have Linux devices for every one of my cpu cores So now we have something that's coming from the kernel and this ring buffer is iterating around and around over itself It's 16 megabyte increments and nothing is pulling from it. So we start from falco and Now we're actually able to gain data. So falco is doing nothing right Nothing's happening on my system. I'm running a pretty primitive system here I have an ide and a couple of Folders open from when I plugged my phone into my laptop But even if I had chrome running right now You would see some set gids and set uids and you would see falco starting to alert us that something was happening So for our first experiment In a different terminal pseudo Or actually we'll do this without pseudo first user local ben Flossed him Permission denied Linux is using preventative action to keep us from doing something that we shouldn't be doing We escalate to a user. We happen to know the password We're able to create the file falco alerts us pretty simple alerting mechanism here and you can see here that because this was a well known directory I'm sure most people in the room here of are familiar with user local ben or user ben As well as maybe maybe some other files on the system such as slash proc slash dev There's a lot of things that you would potentially want to know about if somebody's executing open system calls on some of these Directories are on some of these files. Perhaps pid one would be of interest for some folks Um, so we're able to take this a step further Right, so we're going to keep falco running and I'm going to get some space in here So you can see the alert as it comes and this time I'm going to run a docker container locally and we're going to perform the same experiment And I want you to see how the Linux kernel treats a containerized instance versus a local instance Because this is the fundamental technology that empowers all of the security parsing that we're doing So I'm going to docker run it I have a What I call it my hack container But basically this is a container that I just has like netcat and in-map and Sebastian aliases and a lot of goodies that I use to explore kubernetes And I just push this you push this whenever I make a change to it So I'm going to run this locally You can see here I've got two commands that might be interesting to you that we're going to use in a moment when we run in kubernetes And you can see I'm root here on my My system if we if we uname minus a you can see I'm running manjaro linux kernel version dot or 4.19 And this is the kernel on my system, right? This isn't some newly invented magical virtual kernel or anything This is just the application running in the context of c groups and namespaces Interfacing with my existing kernel. So touch user local bin phosdom You can see here Except for this time if you look at the end you can see we're able to get information from the docker context We're able to get the name of the image that executed this command as well as the the image id So felco starts to pull information from our system as things happen at different layers And if it's running locally we're able to audit it But if it's running in a container we're able to get even more information from the data streams that exist in a containerized environment Okay, so let's do this in kubernetes So i'm going to go back to My public speaking repo here Slides clusterfuck Cool, and i'm going to alias k is equal to cubectl. I'm going to k get pods So Oh gosh phosdom wi-fi. Come on There we go. No resources found. I'll try to keep the uh the internet to a minimum here as we wrap up my talk Um, but you can see there's nothing running in the default namespaces And i'm going to use namespaces as a way to demonstrate that I do have in fact have global privileges on this This kubernetes cluster on this system. So i'm going to list namespaces So I get namespaces and you can see here. I have falco in the falco namespace Gatekeeper and gatekeeper system all installed So if I go to my config directory um dot cube dot config Sorry dot cube And list in here you can see I have config config admin and config default admin is the one i'm using now But if we copy config default over here It's still going to be interacting with the same kubernetes cluster except this time We're going to be using a different service account, which means this user that i'm now running as Should not have access to these namespaces and the the simple trick here is we should not be able to list namespaces So k get namespaces and you're going to notice and see that the kubernetes api server rejected this Request it's preventing us from doing something it doesn't want us to do But as a savvy computer user, we understand that there may or may not be ways around this So let's go now still as my default user without access to the rest of the namespaces in my cluster We can list pods And we can list pods in the default namespace fine, but if we try to list pods in a different namespace We happen to know falco exists. You'll see again that it's going to get rejected So here in my clusterfloc fosdm 2020 directory, I have a small bash function called shell And you go in here and you can see that we have some very interesting configuration bits defined as well as that original kubernetes cluster Container image that I ran moments ago on my local laptop and we're going to run this in kubernetes There's a few bits of configuration here that we're going to prevent from happening again using a tool Like opa, which is this very lovely security context privileged equals true So kubernetes is an abstraction, right? And because of this abstraction, you may not quite understand truly what's going on as you go down to the internal layers of the system That's running kubernetes and basically what's happening here is that we're able to go through and Escalate privileges and exploit this cluster So I have five minutes left and then we have 10 minutes for questions So i'm going to go pretty quick here. So what we're going to do is we're going to run this function shell First we're going to source it Now we're going to run shell And so what this is doing is it's basically creating a tty in my container image running in kubernetes And here I am i'm root at shell as The user in the container I can do a list and you can see that i'm in the root file system of my Linux system But if I cat out at c motd, you're going to see there's two commands here that we're going to use because privilege is equal to true I'm able to go through and i'm able to jump into the pid one namespace as well as the mount the user And the network namespaces and i'm able to basically build a tty Through this such as this so now I'm actually going to do this with bash at the end. Sorry been Bash now you can see i'm ip at 172 20 35 32 Which if you've ever run a ec2 instance before you'll know that this looks like a default vpc amazon instance id and if i list where i am you can see now I am actually on a different file system than i was before because i escalated to the mount namespace that the container had access to Using in s enter to give you an example of where i am and what's going on i'm going to do A docker list Sorry docker ps And you can see all the containers running in kubernetes as the user of the linux system that kubernetes is running on top of In amazon linux. There is a well known ip address That looks like this Thanks, i'm just going to do this so i can copy it Sorry, i'm trying to go fast here Oh Thanks, i know i know cat edsie motd Sorry, fosdom wi-fi is hard We're going to run our In this enter again Run our curl and then we're going to actually build this request. We're going to go to the 2019 api 10 01 you see here we have user space. This is where things are about to get exciting and um Did i spell this wrong? I hear a lot of mumbling, but i can't understand data user. Oh user data. Thank you All right, there we go So if we scroll up This is the configuration file that cops use to bootstrap kubernetes And as i get this from the amazon meta information I come in here and i can actually see that this was hard coded on the system and um We have not only privileged equals true, so we're going to do a grep for minus i priv But we can actually get minus i config And you can see that i was able to get the cube config pass on the system And cap this out here Poof root cluster access from cops running in kubernetes what i would otherwise not have access to There's my cert material there on the screen I would be able to copy this down locally and basically escalate my way to the rest of the cluster And exploit kubernetes while it's running unsecure So if you're not already preventing this from happening, i'm going to show you how to do it So what we want to do Is we're going to go back to this directory here And i have some uh opa policy That's going to get installed for with gatekeeper that if you want to go and actually look at what it's doing It's a lovely set of default policy and we're just going to k apply minus f gatekeeper.yaml And what this is going to do Remember i'm still this default user, but i was able to to escalate my way through to get the root config What opa is going to do now for us is it's going to prevent this from happening again Why did this not work? Oh, yeah, thank you It doesn't work because our back is pervening us from taking action in this Uh default nope admin To cube config Yes, run this again And so now opa is going to prevent us from uh taking action again And if i try to run my shell again, you're going to see here It's effectively denying this request. So what does falco have to say about all of this? So i have an alias here called falco logs and if i can run that and basically all it does is it's going to run k logs minus label app is equal to falco In the falco namespace minus f And this is where the whole lesson comes to life, right? This is where we can actually see from the linux kernel What was happening on those systems that are echoing these alerts out to standard out And we're actually able to create this policy to prevent it from happening again So the story here if we look at our our alerts that we're getting it's it's pretty concerning First my container happened to to swipe our bash history away And we're starting to get information from kubernetes and from docker We're going through and we're creating new shells We started a privileged containers falco was able to alert that To us as well and here at the very end you can see the the big exploit itself Privileged containers stacked and that's where I started to escalate into different parts of the system So the story here the threat model here is there are ways of hacking around things if you're not taking preventative action But in some cases that might not be enough And so being able to detect these types of events and these types of anomalies using tools like ebpf allow you to Do it in a safe way so that you and your team and your infrastructure can begin to have this sort of checks and balances As you go back and forth between security approaches with prevention and security approaches with Detection so if you want to get involved with any of these projects, they're all cncf projects myself And I'm sure many other maintainers here would love to have you involved So feel free to reach out to to any of us and if anybody has any questions I think I have about Seven or eight minutes left and why we have the environment on the screen I'm happy to answer questions or show folks things or or anything for that matter. So thank you all for coming I'm chris nova And then um One thing as people start leaving the room if somebody has a question I'm gonna Save the question back for the recording. So just try to be patient with us as we do the audio relaying here Yeah, yeah, go for it. Yeah, and uh, what is the performance? You what is the performance impact of falco on one word? You have to yell at me What is the performance impact on of falco on systems? Negligible. So the question was and I'll say this for the recording The question was what's the impact of falco on the underlying system? And my response was negligible and the reason for that was because Again 16 megabytes per core and it's written c++. So we've got some documents out there on the internet I'll add one to the the markdown document here where we have folks running, you know upwards of 2000 nodes in kubernetes All running falco. It's still able to maintain their other production loads. It's fine In fact sky scanner a company just released a blog that i'll put a link to They have wonderful metrics where they've been doing load testing and benchmarking with falco and you can see the performance of it Yeah, if you have questions just come right up here and we all answer them for the recording. How does this How does this compare to this? How does this compare to the standard linux audit framework? How does this compare to the standard linux audit framework? So it does similar things But it takes a step further when you start looking at how we're able to enrich that otherwise only available linux information With kubernetes with containers with other bits of information and data streams coming Out of your system. We're right now building a new api for inputs allowing dynamic inputs being loaded into falco So we could potentially start to stream information about ioblock devices xdp The rest of the linux kernel and other things happening on your system and building hybrid objects with all of these input streams That takes a step further than linux audit The question was do we plan to replace the linux audit framework with falco? Absolutely not What we want to do is we want to make falco In the community around falco mature enough to where we could start to use tools like the linux audit framework In conjunction with these other tools and assert rules against all of this information coming into falco Yeah, what's up? Thank you very much a really nice talk My question would be is it possible to turn the shield into a weapon meaning that Somebody using falco and observing those kernel Events and calls discover other vulnerabilities in the system By just playing around So the question here was Would it be possible to discover other vulnerabilities in a system just by playing around with falco or just seeing what falco has to say And I think that was one of the lessons that I was trying to allude to Which is giving an environment where we're taking alerts like this Potentially this would be able to be your first glimpse into building the more mature threat model Um of understanding what actually happened in my example I just kind of did it in the reverse way like I sort of did it backwards where I showed you the threat model And then I showed you what falco has to say about it But the idea here is that by detecting anomalies that you Are well known in linux you would potentially be able to start a journey into discovering a cve remote command injection a root kit whatever Yeah, yeah, does anybody want falco stickers? There's got some more up here come here here Sorry, there's I don't know how many are left. There's one there's some there have fun Any other questions? I'm gonna uh Kind of go stand over here if folks want to come meet me, but that's positive So I got to get out of here and let the next person get ready. Thanks for coming everyone