 Okay, guys, thanks for coming for the last day of the conference. Let's get started. Okay, cool Hi, I don't have my name up there, but I'm John I'm gonna talk about Tetra gone. I got 30 minutes, right? So if you don't know Tetra gone is a There we go. That's a cool picture tetra gone is a Runtime observability and kind of security tool that we wrote We developed this about three years ago, I believe at this point, maybe even a little bit longer. I'm not sure And then we open sourced it last year The basic premise is that if you think about the psyllium project We have the CNI which does all the networking and the routing load balancing all of these functions and What people were also asking for is like an observability platform that pulled in all the Kubernetes all the cloud native information and then Using BPF so they want low latency Low overhead And then the ability to sort of aggregate and collect all these statistics from the system So you can think there's a lot of processes being executed and exited and they want some way to pull all that Data out of the system and then put it in some database or in some sim or some Splunk show a couple of examples of things that we can build on top of this I've got a couple Splunk dashboards. I can show that are actually We're actually running on our clusters, but they're kind of mimicking what our customers are using So I can share a couple of those and they basically use the output and This is just kind of a fun graphic that we share sometimes usually for higher level talks, but You can see there. There's an agent and a BPF piece, which we would kind of expect that then hooks all the different parts of the stack It's just quick agenda. I'll do some demos like I said, and then at the end we can I have kind of things that we're working on All right, so if the first question is maybe is is why do people care? Like what do you want to do with tetra gone and we talked I talked briefly about that But one of the piece you can think of is is I want to know everything that's ever executed in my cluster And so you see every process in the system And over time which is quite useful if you then put that in the database like you're saying you might want to know all the Versions of libraries, so this is a use case some people say is like I have a cluster It's got a few thousand nodes or something in it And then the question is how do I know if all these images? I'm running around what library versions they're running so like open SSL You have a patch for open SSL, and you want to say how can I find anything that's running the old version? That's a common common use case I have some other things over here. There's a lot of networking stuff people like to do They want to collect all the networking statistics latency bytes packets Histograms of all these things because those are useful p99 latencies all this kind of good stuff File access and so you can go through the list. There's a there's a handful of things Usually though when people talk to us about this they also want it to be I put real-time, but near real-time So we have folks with like SLAs of Four milliseconds on the network for example so from end to end so not including from application to application So you know you can't stick a middle box and something like that Like a proxy or something and then they want you know almost no CPU usage And they typically will say things like I want to have a less than 500 megabytes in my point So something small This is just to cut down on the cost right because it's gonna be running on every node You don't want to be too bulky Minimal application impacts so there's usually this You know benchmark people will run they don't want to turn on tetra gone and then be all of a sudden be like okay Now all my applications are 20% slower You want to keep things moving along of course? There's some impact people are okay with but you know we keep it try to keep it as small as possible depending on what we're doing and There's also offline and online models So we have some folks that are for example air-gapped. You don't have an access to the internet or network Sorry, just a quick question. So if you have such low CPU usage And memory usage like why would the application be impacted is it cuz like low latency means that other low latency applications suffer or? So there I mean this is just kind of a high-level slide, but there's really two pieces right? So the one piece is how much CPU your agent is using so like collecting data out of BPF aggregating statistics Presumably pushing them somewhere right so like there's the cost of that and you want that to be low Because that's you know if you're on in a big data center You're taking cycles away from something that's that's probably paying money to be on the system And you know we're not generating revenue and then the the other flip side of that is you don't want to Insert yourself into an application with a BPF program and create latency on the application itself. So think like Like a web server Apache web server or like my SQL stuff, right? If you stick a BPF program in there you can you can Rebring down the latency of like request response on that okay on the application So yeah, don't attach like probes or something that would slow down Yeah, yeah, you imagine putting a K probe in the hot path on Dev QX mit for example Right like maybe you maybe you hook every function that has a SKB in it right with a K probe like it's gonna be noticeable So yeah another question. So is your agent running in the guest, right? The agent is running on the host The the guest of in the VM. I mean, yeah, so it I mean it depends on the environment So we can run we have bare metal environments We have Kubernetes environments and then we have virtual machine environments So in the bare metal will just be on the bare metal system in like a cloud environment Yeah, be up eight of what Damon said basically so you'll be an agent running in a pod which also be on the host typically Okay, in the virtual machine case. They'll run it in the VM Okay, so the the metrics that you collected are specific just as the metrics observed in the For them in the VM or in the pod, right? So you can't get the metric on the host in the pod because it's BPF We can run on the host we have enough we get enough privileges from the kind of pod System that we can even though we are a pod We still are our sort of have access to those as a container. Okay. Yeah cool So this is just some things that we do The basic idea here when we think about this is we want to when we talked about this earlier, right? It Tetra gone is is mostly like a platform is how we've been thinking about it recently It gives you a mechanism to deploy BPF hooks a way to aggregate statistics and then push those into a sim or Security pipeline or gaffana, so we have like a protobuf back ends and gRPC and json logs And usually people will collect those and then feed them into their security pipeline Or their alerting systems as well Just a little bit about the model So if you think about the threat model, we don't trust the users Being a security tool, which is slightly different than perhaps a debugging tool where you might be fine with you probes For example, so in general, although we can do you probes and we will do the you probes in some models We're very clear to distinguish between like security observing and Profiling and sort of debugging The reason is because you know, you can't trust user memory if you're trying to try to make security decisions based on user memory This is going to open you up to any sort of security vulnerabilities This mostly comes to play if you think about syscalls, right? So if you have a syscall, it has a pointer to user memory We won't we can read that but in a general like in a deployment model We wouldn't do that we try to hook something lower in the stack And even if you're lower in the stack, you need to make sure that you're not looking at user memory, right? If you're in the security mindset, which is which is right now kind of the main thrust of some of the work we do You probes are also there they're interesting because we've had a few customers say I want to know if this function in my application is ever run Because I know it probably shouldn't or I know like I know something about the control flow Stack traces will be interesting there, but we haven't quite got through it The other thing from a design site like we're really interested in running on larger systems So we'll try not to do things that won't scale To thousands of nodes Usually this comes into play when you're in sort of a Kubernetes environment There's a Qube Qube API server and Qubelet and you can put watchers on that, right? You can watch for other pods and things like that But you can imagine if you were to say like tell me every pod in a in a cluster for example with a thousand nodes You might get the server to tell you you know ten thousand things So those those types of things come into play We always try to fail closed meaning if you're gonna If you're gonna have a gap for some reason then let's just fail This could come into play for example if you overrun a map So we very rarely use LRU maps for this reason we do for some cases But you could imagine if there's like a socket statistic you're gonna either allow or deny the socket If you can't make the connect like the correlation from that socket to the process to the policy Rather than just say okay like a like a routing system might say okay I'll drive my best and we generally try to just stop that from happening even if we can't make that connection The last one is be kind of the security stack. This is a big one You find this pretty quickly once you try to deploy this the security team doesn't like it if you blast them with you know 100,000 events a second or something right like their whole security stack will fall over And sort of fix that And the other one is if you send out like a tons and tons of events to Splunk where they're paying a per message cost You also very quickly get people upset with you So we do a lot of things to filter and aggregate the events Put some rate limits on them compress the Cardinality of the events so that instead of saying I want to have an event for every connection Just tell me all the endpoints in my note or something This is all about cost and making the data usable Okay, cool. So those are the kind of high-level things and then at the at the sort Of core the very first like kind of core functionality is exec tracing And you can see there's a json output on the left So basically we hook the the exec events and the exit events and what you get then is a Sort of json here that describes the process and some some interesting things are an exec ID Which is a unique ID for the for that execution We have some work in some stuff in the works to do the shot to 56 There's a couple PRs out in branches with this basically. We're taking the the digest of the file if you're running something like FS variety underneath This gives you a strong identity versus just a binary name And then you can see we have a whole bunch of other stuff there I even cut a bunch of stuff out because you know I needed to fit on the slide But we have all of the pod metadata which would be the Kubernetes context the binary name Working directory arguments. There's the namespaces and capabilities and C groups and all that kind of stuff So you get a good a good view of where everything's running and This is important um Out of that information we can build two things the identity Which gives us a unique unique kind of a compressed version of what that thing was and then a location So you want to know what cluster node namespace pod can enter time And once you put these two together You get a globally unique ID That's going to be unique in your database For that thing that ran at that place it ran out and What this lets you do then is you can say a month from now somebody tells you there was an incident You can go back into your database You can find this thing and then you can because all the next events that we'll show next have this information You can say I want to know everything that that that object did And if you this is I did this for fun I pulled out like five minutes of exec data out of our cluster our little test cluster and graphed it you can see it That basically what that is is all of the nodes are the executables in the and the links between them are the The parent-child relationship You graph it and you can see there's like you know thousands and thousands of things and that's that's a pretty small cluster So quick quick question. So So yeah, when you were first describing it it seemed like a Way to basically like document everything that's happening which is obviously incredibly useful But then also in the last slide you said that you rate limit like the events that you send to security team So is it both sort of like? It's a both sort of like a way to kind of document everything for investigations later Or is it also a real-time? Observability tool that like notifies if there's an ongoing incident. Yeah So we actually have folks using it in both cases So there's really users on both sides of that and and users that go both ways actually so One One use case is the like kind of post analysis like you have a security team. They get an incident report They need to go back and dig through the logs and figure out what's going on, right? And so then they can say okay, what executed during this time window. What was it talking to? Oh look at it, you know, it's spewing out to this s3 bucket somewhere, right? It's like, you know, not our s3 bucket So that'd be that kind of the post analysis and then I'll show in a second We have some of the dashboards that the customers use are one of our customers uses and they what they can do on top of this is then Hook up something like Splunk or Gafana these kind of runtime systems that have alerting right so you can alert on top of that And then when I get into the actual events, we can also do enforcement in the kernel too. So that's okay. Cool. Thanks. Oh Okay, let's do it now Let's see so for execution tracing Oh, I gotta jump this is gonna be a little bit awkward, but Can you see that oh, it's it's huge So if you if you look at this is a Splunk dashboard that mimics it's not actual customer data It's our data, but we have a few folks doing this and when you're asking like what do you do with this data? Right this plugs into to a Splunk database And then you can do things like show me all the pods with access to the host network namespace and the entire cluster Right. So like with five nodes. This isn't really matter But if you have ten thousand or twenty thousand nodes, you can see how or even a hundred nodes, right? And you want to know What's running here with? With host access you can see cube proxy Hubbo enterprise, which is some of our stuff. I don't know what PDSI node is maybe somebody else knows nobody Merlin, whatever that is so you can kind of they break these down What else do we have in here? You want to know everything in your cluster that runs with capsis admin so we can see there's some privileged pod I think that's probably some privilege to the pod must be some test thing that we're running Capnet raw cluster wide So here you get a breakdown and I think the slice of the pie is how many times that it was found in the database over some time timeframe If you go farther down then you want to maybe you want to know more details about this data You can see here We have start time the namespace. These are all in default the source pod source image where it came from The caps that it actually had The count so this is just an example of Basically taking all that data that that's coming out of the system putting it in its plunk and then building these dashboards, right? And then if you're if you're really interested in like how you tie this back into Alerting is there's ways in these tools to say like alert if I see a new a new thing start with copnet admin or something, right? Just of that and I better get moving so that I can do a demo So the if the exec is kind of the core piece that tells you the exact tracing the next thing is we have this K probe yaml file What this does is lets you put K probes or trace points or trampolines Basically on any function that you know we can attach to in the kernel Dynamically without recompiling We're restarting the pod through just pushing CRDs around a CRDs or YAML file configurations inside Kubernetes and so if you look over here, it's just a spec you have the call Which is the function that you want to hook You can tell if it's a syscall or not we can discover that too But it's there if you want to be explicit and then you give it the args The args are interesting you we could auto discover the args of course, but usually We don't want to spend the time to copy all the args, right? Like if it's a if it's a Like a write function. Maybe you don't want the 4k buffer included in your message, right? So this lets you tell you what to do and then the selectors or the filters And so you can say match PIDs will let you match certain PIDs This is perhaps could be better, but basically what that's trying to do is say I don't care about things in the host I'm just trying to match PIDs in the pod space Which is pretty common thing to be doing forks basically just means we're going to follow the the tree of of Processes until we find one of these Then we have a bunch of operators on the args And then an action what to do. So basically what this is saying We just summarize it. It's basically saying if I see anything open Etsy password I want to follow it which which will be a trigger to start doing more actions on it So like start watching more events People you then use this these are just some tools people have written on top of a top of tetra gone There's this which is just a CLI. It's fun. You can connect to it I'll show it in a second. This is just a GUI that somebody wrote that shows like the execution trace and then pretty print some things About what you're connecting to Going the wrong way again Another thing we can do is file integrity monitoring I'll just be quick. Here's just another thing if we hook FD install almost all kernel file descriptors go through there so you can start to build a Tree or a bunch of events on what files are being open and closed There's a dashboard for that I'll show it in a second here. I'm gonna skip ahead Very similar to the last one we looked at just like I want to see every file that everything that opened Etsy password in my cluster or something like this Here's another example of doing enforcement if you look at the actions down there at the bottom We have a couple different actions. We have a sig kill and an override There's a few more. We can also send alerts over DNS or alerts over HTTP What they do the sig kill will send a signal to the process to kill it. It does this in the kernel Directly from the kernel the advantage of doing that is it's in line with the hook So you're gonna kill the user space process. It's gonna bring down all the threads can't be masked because it's a kill and It's gonna set the signal so that the next time the kernel checks for the signal It's gonna bring down stop that execution thread. So if you use these correctly, I would say in this environment You can ensure things like writes never happen And I'll show a demo here in a second. Oh, and this is a picture showing the same thing Why do you need override if you do sig kill? So There are two models right like one model is you want to kill the application you wanted to just be gone The other model would be like a little bit friendlier just to return An error but in this example you're able to pose. Okay Good catch So there are some things that do not some pass in the kernel that do not check for signals So like if you want to stop a sys call And you want to kill the process and you don't want to worry about if the path in the kernel has a signal check Before it does the operation and you don't want the operation to complete if all that makes sense Then you can kill the process override the thing which will stop the kernel from running that flow as well So it's more to like prevent kernel side changes. Yes, exactly. Yeah getting back to user space. Yeah I return error. Okay, and if you know, I should say a lot of times we've hooked just sig kills inside the kernel Like deep deep in the stack. It's like you don't need to be at the sys call level You can be down wherever in a like a socket create or a socket close or something and and that case The curtain was actually pretty good about checking signals before it does operations Which makes sense, right? Like if if somebody's killed the process you probably don't want to open a file descriptor for it, right? So you check so as long as you put those hooks in the right spot You can kill the process and the operation won't happen in the criminal Which is both the wind from from that side arguably if depends on the operation doesn't really matter if the process is gone It doesn't matter if it has a file descriptor not really it's coming Well, if it managed to mount something probably it matters. There are things that have effects like rights like you know system like file rights mounts Things like this sand packet sin packet. Yeah This is just a benchmark So you can see Basically what I'm what we're trying to show it here is like the base overhead and when you're kind of watching execs and library loads He's quite small on a on a kernel test. So this is running perf doing J16 on a kernel If we're monitoring all the syscalls, it's a little higher but not terrible With a filter basically that's a filter to not monitor everything in the system, but just monitor very specific things So you're filtering out tons of data And then this shows basically what happens when you don't do any filtering and you run You just run that command and try to monitor every system call in the system, right? I think I got that right So basically what we're what we're illustrating here is you really need to start filtering in the kernel Or you're gonna start hitting that buffer ring over and over again, and it's gonna be expensive All right, let's do how much time I got I got five minutes. Let's do it. Let's do a demo. I Can I can do it I think? How big is it over there the biggest problem is the resolution on my screen is small Okay, good. So let me pull this up just a slightly bit here So this is our That's CLI tool. It's just a If you don't give it any filters it will attach but without any filters and you can see There's my system is doing stuff, but it's not terribly interesting like it's hard to see exactly what's going on So let's let's kill that then Let's do Let's monitor a specific process so we can monitor curl the top here is Is the is the policy that I loaded for this guy? Kind of hard to see there. Let's do this There we go So I have a policy to do in monitor install What it's doing is is see checking to see if anybody opens this temp my secret file I created it'll then monitor for close events if anyone writes to it. It's gonna I think kill them Yep, sick kill And then I added some TCP connect stuff just for for fun So that's the policy The top bar is just the Tetragram running. I'm just curious like your customers write those configs themselves or like they give you Requirements and you actually translate that into a function. Yeah, so so it's internal thing Yeah, yeah, yeah, so like this I mean any tetra gone exposes This is the interface to tell you what to hook right and so you can hook any function in the kernel with this Right like I by just putting it in that call I mean anything that can be hooked right like you get an error if it's you know an in-line or whatever And then you give it the arcs And you give it the actions right so you can imagine you it's up to the user to build those policies Right the user of tetra grown It's unlikely that a kubernetes diva operator is gonna know like I need to hook TCP send message And oh, I know it's safe to do a sick kill here because there's a signal check in the kernel Right, you can imagine that that's probably not on the top list of their concerns And so they're probably not all of them some of them actually do write these But you can imagine that they would want somebody else to build them prepackage things to build that they can then use To build those swunks that dashboards like that splunk dashboard was like something we created Gave it to them. They created this dashboard, and you know now they have all the data All right, so this thing is doing Running you can see I was playing with it before here. So if we like do a curl We should get something down there. Hopefully Yeah, there we go. So Basically because of the prop whatever I have set up in the file there You can see the process starts it does a curl out to these IP addresses on these ports does a sin message Does an exit? No, just a basic flow if you if you really wanted to get more carried away You would be tracing UDP you'd catch the DNS and all that all that good stuff Next thing because because you saw we had that we had This is my secret thing here Let's open that. Oh Let's hold on one second. It's monitor VAM. There we go So you see VAM opens up my secret closes it opens it again closes it opens it closes it I'm sure what oh, I think those those closes are for different things But anyways when VAM comes up it it opens and closes a bunch of stuff It because of how it does it's buffering Anyways, let's try to do all right. That's that was what I wanted to show Let's do Hello Let's save it And you can see it was killed because we saw the right we did a sick kill to the process There is a signal check in that path So if you Can you now cut the file you can we can cut the file we have permissions to read it We just don't have permissions to write to it It's curious if it made it today. I know there you go No tricks works it works So that's the demo and then As promised you can see we have some other dashboards here that I just populated up You imagine you can take that data that we just saw right and then you export it out and you can do bites by pod Aggregation you can do egress traffic by pod You know and you can see these are the things connecting to various things you can see the sidecar Just different discussion, right? And then here you have like more of the data just kind of in chart format not graphed If you want to look at sensitive files, here's another one you can say here are all the files that were opened By counts you can see these are the top whatever five things that what that happens Or actually it's for a binary is for the core API. I guess you want to see what core API is doing Here's the file names of things that core API opens Etsy password a lot. You can see Etsy shadow I don't know what it's doing with the shadow, but something And again, then you have all the raw data down there or just a chart of it so my time is up so Just a quick thing We've maybe did this backwards, but you'll notice like that's why we care about these things like t digest and q digest We're looking for ways to build summaries inside BPF on these things Why we use multi attached because you I did a demo with two or three things there But you can imagine we have hundreds of things that we're actually attaching to SROV is interesting because it escapes the some of our met our low-level metrics on interfaces Applications signatures are I didn't talk about it. I only mentioned it briefly there But there's a shaw right so we care about this in signing of applications BPF signatures are interesting. We haven't had a anyone really asking for them yet. So but I put it up there anyways I know it's a topic Stack traces we haven't done yet, but we have some code that does it but we haven't really deployed it That's an interesting one. What about frame pointers? What's that? What about frame pointers in user space? Do your applications generally compile with frame pointers or was that? How much do you care about this frame and stuff like this? I don't know. I don't have a good I don't have any data point to say yes or no like what they do. I've not gone around and sampled them, right? We could figure it out probably Because if they don't build with frame pointers like users tech traces are meaningless Yeah, we could figure it out though because they're Mostly they're public like a lot of the major things are published on like images for Kubernetes So we could go and look and see if things like engine X and all these things have have the frame pointers, not sure And then we have a lot of networking stuff. I put that there because you know, I'm working on it So it's it was top of mind Contribution slide. It's on github. If you care psyllium tetra gone There's all these all the examples I had today are in the CRD's examples Directory so you can load them if you want to try it and Thank you. This is our nice slide Yeah, any questions? No, thank you very much