 Okay, let's get started. I'd like to welcome everyone to today's CNCF webinars. K-8, Auditing in Depth. My name is Jerry Fallon, and I will be hosting today's webinar. We would like to welcome our presenter today, Randy Apernephy, managing partner at RXM. We just have a few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee. There is a Q&A box at the bottom of your screen, so please feel free to drop your questions in there, and we'll get to as many as we can at the end. This is an official webinar of the CNCF, and as such is subject to the CNCF Code of Conduct. So please do not add anything to the chat or questions that would be in violation of the Code of Conduct. Please be respectful of your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF webinar page at cncf.io slash webinars. And with that, I will hand it over to Randy for today's presentation. Randy, your mic is on mute. Hey, thanks a lot for that. Good morning, afternoon, evening, and welcome. This is the Cates auditing and depth session, a bunch of interesting stuff to take a look at over the next half hour or so. So why don't we go ahead and just jump right in. All right. So my name is Randy Apernephy. I'm a cloud native geek of the first order. I'm a big fan of microservices and Apache thrift and things in that area. And I work for ArcSan and we're, you know, diving a little cloud native folks over here. It's just a quick note on me. The session today is going to cover auditing. And we're going to start from the start and go through some of the basics, but we're going to quickly move into more advanced concepts. And we're going to talk about some of the challenges and issues with multiple API servers and mutating admission controller web books and how to deal with different audit log back ends and scenarios where you have, you know, kind of massive throughput requirements and, you know, what exactly you can expect from audit logging from a throughput and capacity standpoint. So let's, let's start off. You know, audit, what is audit logging? We just started the very start. Well, the definition of audit is an official inspection of an individual's or organization's accounts, typically by an independent body. And this is a, you know, kind of an interesting parallel to what Kubernetes auditing is. And you can see the, the, the context sentence I think is actually even more telling audits can be expected to detect, can't be expected to detect every fraud. So this is exactly the spirit of auditing in Kubernetes. Logging happens in, in services that you run in Kubernetes, including the control plane services like your, you know, controller managers, scheduler, the API server, kubits and so on. And they all log to standard out standard error. And if you're a system D service, that's going to be manageable through journal, kuddle and all that stuff. If you're actually running in a pod as you may, if you're running, you know, kind of a cube ADM style setup for your, your control plane, then kubectl logs would be able to show you the log output of these different services and it can be managed with, you know, plugins that forward the logs off the back ends, low key elastic search, what have you. And then you've also got events taking place inside the cluster control plane events. And those events are going to be visible through kubectl get events, for example, or if you describe an object, you'll be able to see information about it. But this is a different beast auditing right auditing is designed to give you the ability to inspect an individual organization's accounts. Right. And to be able to detect activity that might be fraudulent, for example, some of the other verb uses here, companies must have their accounts audited. They may use some knowledge gleaned from economics class we audited. So being able to watch and oversee something right is sort of the idea of the audit log. And it's a, you know, it's different in kind because what it's designed to do is capture the who what and why of activity going on in the cluster and it's usually at a far more granular level than these other types of logs right application level logging that you get in standard out and standard error is going to be things like, you know, I created this, I did that, I did this other thing. And some of the details may be obscured for security reasons or something like that. But an audit log is designed to capture all of the details. It is designed for no holds barred inspection of what's going on in the cluster. And so, you know, generally only privileged individuals should be looking at the audit log because it can expose a lot of stuff. You know, you can, you can look at the exact manifest posted by every user for all of the resources that they're creating, you can see the responses in detail from the cluster, and so on and so forth. So it's really a, you know, a function like you would have a security audit. That's really what the audit log is for it's for it facilitating those types of activities. So let's start off with just some of the basics. This is the definition straight out of the Kate's IO docs. Kubernetes auditing provides a security relevant chronological set of records documenting the sequence of activities by individual users, administrators or other components affecting the system. So an interesting thing about the audit log is that not only are you looking at the actions of users, administrators or otherwise, but you're also seeing the interaction from the, the system principles within the cluster. So you're going to see activity from cubelets and from controller managers and schedulers. And in this way, you can use the audit log as a great way to get it in depth understanding of how your cluster is actually operated. And in fact, you know, how frequently, you know, certain types of activities are taking place. And so, you know, you can always go and dig around and config files and things like that to sort of figure out how things are set up and what they're doing. But going to the horse's mouth is always, you know, the authoritative answer because you might see a configuration file that says this thing's supposed to happen every five minutes. And you look at the audit log and it's happening every three seconds. Well, that's putting a lot of load on the control plane. Maybe you want to look into why that is. Is it because the config file is mistyped? Is it because there's a default that is, is at play in some scenarios where there isn't a config file? Yeah. Yeah. So you can do all sorts of interesting things. You can glean from digging through the audit log. So it can be used for, you know, for security professionals and forensics and things like that. But it can also be used for cluster debugging and performance tuning. And it's a, you know, just a really all around powerful facility. So if we were to look at the architecture picture of the audit log, it probably looks something like this. All roads lead to the API server. The state manager for the cluster at the end of the day. It's the microservice that owns all of the metadata describing what the cluster is doing. And the API servers are stateless themselves, but they have the logic that is there to handle authentication, authorization, admission control and all of these types of things that decide whether something that a end user would like to create as a specification is going to be accepted or not. Now, if that specification is accepted, it's going to be dropped into Etsy D. So Etsy D is a highly consistent key value store sitting behind the API server. And, you know, this is a simplified model. You would generally have multiple API servers and a cluster of Etsy D nodes. But the communications channels are the same. Right. Everybody talks to the API server and only the API server talks to Etsy D. And so if you want to know the status of something, if you want to create something, delete something, update something, modify something, you do it through the API server. So essentially the API server is the gatekeeper of all, you know, configuration and status for the cluster. When we kubelets, which are the node agents running out there in the cluster nodes, report in the activity that they're seeing, the status of their memory and CPU consumption, the specifications that they're maintaining for pods and containers, all of that stuff's being dumped into the API server and dropped into Etsy D. So if we enable the API server to log all of this activity at the API level, and since the API servers API is the gateway to all state in the Kubernetes cluster, then, you know, we're really creating a place where we can see everything happening. Now it's not completely true. It is a distributed system, you know, kubelets keep a cache of the pods that they're supposed to be running and there are, you know, little pockets of information throughout the system. But at the end of the day, if the API server is right and good, most things in your cluster are going to be right and good, and if there's something wrong, the API server is going to be able to see that in most cases. So it's the perfect place to be capturing this kind of detail blogging. Now, when you run an API server by default, there is no audit log. And so we don't have this facility. It's therefore very good to know that the audit log exists and to start thinking ahead and saying, hey, today we don't need the audit log, but after something catastrophic happens, it's too late, right? You want to see what caused this big problem and you have no record of the activity. So there's a lot of different ways we might want to configure audit logging because audit logging can range from non-existence, so no load at all, to capturing every single thing that comes and goes from the API server, which is going to be a lot heavier. When you think about the cluster picture that you see here, you can quickly rationalize kind of the amount of activity that's going to be coming into the API server. If you imagine that you've got a maximal-sized cluster currently upstream Kubernetes is 5,000 nodes, right? So if you've got a 5,000 node cluster, you've got 5,000 kubits reporting status to the API server on a regular basis. You've got reconciliation loops and the configuration managers looking at all of those deployments and services and as pods come and go, they get evicted because one node gets a little bit tight on memory and they pop up somewhere else. You've got the scheduler moving things around. Imagine you maybe have 10 pods per node in a 5,000 node cluster. That's 50,000 pods. That might equate to something like 5,000 deployments and 5,000 services and 50,000 endpoints for all of those services. That's a lot of just quiescent activity. Even though nothing might be happening in the cluster driven by users, the reporting and the constant reconciliation, the self-healing behaviors that Kubernetes is performing all the time can create massive flows of messages into the audit log depending on how you've got it tuned. So the API server is really the centerpiece. All the requests to view or modify state go through that API server and so it's the central place where we want to do auditing. So what's in the audit log? Well, it tells you what happened. So if I, for example, create a pod or create a deployment or create a service, that's going to be recorded. It's going to record when it happened. So the API server will timestamp the audit log event they're called. It's going to specify who initiated it so it'll capture my identity. And this is very detailed. So if I am an administrator and I'm impersonating another user when I create this pod, it'll capture both of those identities, my identity and the party I'm impersonating. So what object did this happen on? So the pod, the deployment, whatever, you can get the exact identity of that object. And where was it observed? So as information is being processed by the API server, we have different stages and so on. And then from where was it initiated? Where did this request come in from and where is it going? So if there's any destination for this thing, that can also be identified in the audit log. So an example of an audit event might look something like this. And as you can see here, we're just tailing the audit log file and just grabbing one line. And the modern audit log is JSON based. So the old audit logging, which is deprecated at this point, legacy audit logging was text-based format. And so audit events were a single line, but the current approach is JSON, which is a lot easier to parse and process and store and search and index and all that kind of stuff. So if we just clean up the formatting and space it and indent it a little bit there with JQ, our JSON query tool, we get a nice dump like this. And so you can see this looks a lot like a Kubernetes resource, a Kubernetes manifest or a Kubernetes spec. And so it follows the same exact principles as everything else in Kubernetes, where it is kind of a sort of a declarative approach with key value pairs and support for nesting and collections and things like that. And so any Kubernetes object that you would try to create would have a kind. And so that's the type of thing it is. This is an event. However, the kind of a thing doesn't create a unique definition. And this is because parties can create custom resources, for example. And so if my company, let's say RXM creates an event type of resource and then let's say, another company creates an event type of resource, how would you disambiguate? Well, there's an API group that you would organize those kinds under. And so as you can see here, this is an audit Kates IO based event. And it also has a version. So it takes three pieces to put together a complete, kind actually. You need the group, you need the kind itself, which is subordinate to the group, and then you need a version. And so there are other types of events. And this can be confusing to people first getting into audit logging, the typical Kubernetes event that you deal with as a control plane event that is not an audit event. So there are different kinds of events. Keep an eye on that API group to know which type of event you're dealing with. And then these guys have, they've got a number of other bits of information, which makes them a little bit different from a typical Kubernetes resource. Generally, Kubernetes resources would support metadata and the object in question would have a name. So these events have identity per se, but they're not named, right? They're just events in a stream. And so it's not like a pod that would have a name. And also you'll note that these events aren't labeled, right? They're omitted only. We don't create them. They're an artifact of activity in the cluster. And so they're created by the API server in a stream as things happen. And so they're a little bit different from your traditional resource, but the format is kind of similar. Now you will note down at the bottom here that we have annotations. And these are just exactly like annotations in a typical Kubernetes resource. They give us the ability to expand on the functionality of the audit log event without damaging the overall spec. So for example, if you create a pod and you want to tell, you know, some, let's say some, some C and I plug in something special, like maybe the C and I plug in has some tricky dual networking functionality. And you want to tell it to put you on the B network. You can use an annotation. Kubernetes doesn't know anything about multiple networks, but by plugging in an annotation in there, you're creating a key value pair that Kubernetes basically passes around everywhere, but just ignores. And so all of the plugin components and extension points in Kubernetes are often going to use these annotations to, to, to augment the functionality of a particular thing. And so in the case of audit logging, that's exactly true. So if we have, for example, at an admission controller that we've added to our cluster through a web hook, well, Kubernetes can create audit events around, Hey, this thing got denied because the pod security policy denied it. But if the, if the web hook that is not part of Kubernetes denies this, we need to, you know, maybe have some, some reasons why, or if it mutates the request, we might want to know what the mutation was and all those types of things can be represented in annotations. So annotations are really, really give us a lot of flexibility here. Some other things that you'll note that we're going to talk about in a second are that there are stages of processing. And so you can record events at a given stage of processing or multiple stages of processing, if you like. We have a, you know, the user as we described and who was involved here. So this is this particular node. So that's the host name of the node that, that made this API get request. And you can also see the URL. So this was the request API. So this, this particular node was getting the API v one nodes information on itself, which it is allowed to do. And it is going to do on a, on a regular basis, right? To get, get any, you know, kind of updated information about itself is an interesting thing about Kubernetes, right? You have to remember that when you submit a specification to Kubernetes, the API server basically verifies it from a security standpoint and then dumps it into that CD. There is no guarantee that that means it's going to be okay or work, right? And so things asynchronously then kick off after that, like the scheduler assigns pods to nodes. And if there's no node available, your pod might be pending. If the pod does end up on node, but there, you know, the image that you've specified in the pod is no good. That's going to cause the Kubla to not be able to pull the image and it's not going to run again. But in all cases, as far as the API server was concerned, the spec was good and it saved it that CD. So you have to also have, you know, a fair amount of understanding of Kubernetes to be able to follow through with some of these events, because the ways that you would find out these other things would be after the fact, right? The user posted that pod spec. Sure. And there was no errors, but that doesn't mean it's okay. The scheduler might attempt to do something and report that it couldn't be scheduled. The Kubla might report a status of image pull, you know, fall back and, you know, be failing to pull the image and continually retrying and reporting that. So you can find lots of different pieces of the puzzle and wiring that all together is definitely a skill that you develop through practice. So, you know, one would suggest then that if you're going to, if you find that audit logging is going to be an important part of your, you know, operational environment, working with audit logs and starting to craft, you know, some experience and dashboards and things like that through your back end, you know, log management systems, whatever they may be, Splunker, Elastic Search, Kibana, you know, Grafana on Lowkey or whatever it is, you know, getting prepared and developing some skills ahead of time can really pay dividends when you're in a scenario where there's a failure or some security event that you need to deal with. So what is the definition of the fields and the audit event and how are they all organized? Well, that's a good question. And I'm just going to pull up something that you're probably familiar with here, go to kates.io slash docs and pull up the reference here. So if you want to know what's, you know, how to specify resources, the API for Kubernetes is essentially these JSON documents, right? I mean, you get post-put and delete these things, but all of the activity that's taking place is in response to these documents. And so as I mentioned, if I go ahead and search for event and we look down the left-hand side here, you can see that there's a metadata API and there's an event defined there, but this is part of the core group, right? Anytime you don't have a group name, so for example, let me just go over here to a cluster and do a cube CTL API resources, right? So these are all the API resources known to this particular API server. And so these are things you can post, you know, and put and delete through the API server API, but you'll see that there's a bunch of these resource types, pods, you know, the early guys that were there with version one that don't have an API group. That's the core group. So those guys are always in the core group. And then you've got a bunch of these guys that have different resources depending on, you know, which working group is managing those particular resources. And you'll note that if we look for events, you can see that we've got events and event, right? Events.kates.io. So this is a completely different group. It's not the same resource as the audit log resource. It's not part of the API, right? These are just a format for audit, you know, information being emitted. So, you know, we're kind of stranded here because if we, you know, you can search around and there's, you know, you're going to find these are legacy, right? Old API versions. So there's just, there's no, there's no information here. There's limited information about the format of the API events and things like that. And so whenever you're in doubt, you know, go to the sources because Kubernetes being open source is a huge benefit. The quality of the code in Kubernetes is pretty being high because there's all of this governance involved in, you know, how changes are made and reviews and minimum requirements for documentation and the code. So as you can see here, all we really need to do is go to Kubernetes in GitHub and, you know, move down to the API server package and then look at any audit B1 types go. And you're going to find definitions of all of the types of things that the audit subsystem uses. So you'll find information, for example, here on the event struck and, you know, every single field is described and you can, of course, even see the data types that are being used here, which if you can read go, which isn't that hard to ingest if you know any programming language really, you've got a leg up. So getting detailed definitions for events and all the different fields you can find here running into something that you're not familiar with. But let's cover a few of the key things. API servers, you know, process requests and stages. And so they authenticate users, they authorize users, they admit resources as a final stage in the security processing is other things that happen as well. And so from an audit standpoint, receiving request is something that we can log if we want to. This is the first stage. This is generated as soon as the audit handler receives the request. And so if you're interested in, you know, every single request that's made, that's a stage. Next, the response started. So this is after the response headers are sent, but before the response body is sent. So this would apply to like long running requests, like a watch request or something like that. And then response complete. Again, this is the response body complete. So after there's no bytes left to send. And then there are also, you know, in go programming, a panic means something pretty catastrophic has happened. And while you might be able to recover from that, most of the times that means that, you know, the API server would crash. So I'm pretty serious. So those are some stages that you can see identified in the events. And you can also use these stages for filtering events too, as we'll see. So audit levels control the level of data admitted for an event. None means you're not going to log anything. This is the default. So if there's no policy specifying what to do, nothing's going to happen. Then you've got metadata. So this is basically going to log all the, you know, the high level stuff, like the header type of information that you'd have in an HTTP site sort of exchange. So it's going to log the user, the timestamp, resource information, the verb used. So you can basically see what's going on, but you won't be able to see the details. So you wouldn't be able to see, for example, you'd see that somebody's creating a particular pod, but you wouldn't see what the pod spec is. Now that's going to do two things. By sticking with metadata, you're going to be able to see broad activity in your cluster. You're going to be able to know what kinds of things are happening and which objects they're happening to, but you won't have the details. You won't know specifically, you know, when this thing happened or when that thing happened or when the next thing happened in so far as mutation of a pod, for example, where if you go with requests, that's going to give you the request body, but you won't know what the response is going to be. If you're going to capture the request, which is often, you know, a big piece of the puzzle, then you might want to think about capturing the response as well so that you can see the response body coming back. Though, again, you know, if there's a lot of activity on your cluster that's looking things up constantly, then the responses could be large and I could, you know, not just incrementally, but, you know, potentially multiplicatively increase the amount of logging. So each of these gives you progressively more log output. And that means you're going to have to have more capacity and throughput in your log. You know, a function. So however you've got your logging managed. So here's the audit policy. So an audit policy is, again, a lot like a, you know, a typical Kubernetes resource and it is saved in a file and provided to the, to the API server in order to allow it to, you know, decide what kind of auditing you'd like to do. The audit policy file is incredibly powerful. It allows you to really get very, very specific about the types of things that you want to capture. So for example, you've got high level specifications that you can add like omit request received stage. So that's, you know, that's sort of a global policy. Then you've got individual rules and these rules specify the level. So in this case, request response down here, metadata down here, none. And then you've got the resources. And so you've got the group. So this is the core group, right? Empty string is core group. If you wanted to, you know, reference apps, you know, you'd put apps in there. If it was networking.case.io, you'd put that in there. And then you've got the resource types. And this is a, of course, a list. So you could have as many resource types from the core group as you'd like to specify here. As we move down, you can see that you can even choose specific resource names. So if you want to, for example, make sure that we're not logging config maps, then we could specify this specific config map by name is not to be logged. And then we've got scenarios where we're picking specific users from the list. So, you know, logging activity from the cube proxy in this case and specifying the verbs that we're interested in, well, in this case, not logging. And so maybe there's a lot of activity in your, in your log that you've identified as not being pertinent in the scenarios that you want to be able to do forensics around, you can, you know, you can sometimes carve out 10, 20, 30, 40% of the, you know, of the IO by just carefully blocking off certain bits of logging using level nine. So really useful, useful tool and, you know, very, very powerful and gives you lots of granular control. So again, where do we get the details? This is not a Kubernetes API server resource. It's an audit subsystem configuration file. So again, if we go to types, you can see all of the different settings for a policy rule, which is, you know, the main thing that you're going to construct. And if you look through here in the types, you'd see the, you know, the policy list and the audit policy types and all that, but the rules are kind of the interesting one. And so you've got the user's component of the rule, for example, user groups. And if you're familiar with RBAC, and if you've done any Kubernetes security, and I would say that that's almost a prerequisite to, you know, working with the audit log. If you're in this space and you're doing this kind of stuff, you may be a security professional or that's a hat you wear. And so audit logging involves similar types of constructs, right? You are, you're in RBAC going to give a particular principle, a user, a group or a service account, some capability. So some verb on some object. So some resource type. And those resource types, again, can be scoped by the group. And then they can even be scoped down to a specific named resource and then a kind. And so that just kind of carries on here. So if you're familiar with RBAC, the audit rules are very similar. So as we mentioned, auditing is not cheap. It increases the memory consumption of the API server. Remember, from the model that we saw only the API server is involved in auditing. So, you know, you don't really have to worry about the activity from, you know, the controller managers or the other guys. It's really the API server. And so, you know, topping your system, getting some baselines of your server without auditing. And then, you know, maybe progressively ratcheting up the policy to increase the amount of auditing and watching your resource consumption on the API server side. You know, it's not a bad thing to do that way. You can sort of get a sense for, you know, where the diminishing returns are. If you, you know, if you basically log everything, it's going to be, you know, a crazy, you know, every bite in is going to be magnified by two, because you're going to be writing it to be audit log with a bunch of extra metadata. So, you know, kind of getting a sense for the, the throughput capabilities of your system and the memory capabilities is going to be important. Memory is a key piece of the puzzle because the, the API server's audit subsystem is going to capture, you know, various context and other types of information about all this logging, audit logging output. And in many cases you'll want to be buffering your output as we'll talk about in a little bit. So watching memory utilization of your server and also the IO consumption of your server to whatever kind of backend you're using for capturing the audit information. So how do we set up the API server? Well, the API server has 30 audit logging options. And the most important one is perhaps audit policy file. And so that's the file to actually use to, to, to define your audit policy. And a lot of times when people, you know, start thinking about, you know, this, this file, you, you, you can, you can put it in a bunch of different places, but at the end of the day in a cube ADM scenario, you would probably put it in a protected directory. And then you would host path mounted into the API server container. That would be a typical scenario. So the next thing that we've got, and let me just, maybe I'll just show you a quick example of that. So let's come back over here. And let me just dump out the, the manifest that's running our API servers. So, and so you can see this guy, this guy's got a host path for bar log audit. That's where the log output is going. The, the, the, the policy file also, as it turns out as there, a lot of times the policy file will be an Etsy Kubernetes or something like that. But there's the mount path inside the container. So it's the same as the host. And then in the configuration of the API server, as we run the cube API server and our configuration, we specify the audit log path and then we have the policy. So those are two key configurations. And you can run the Kubernetes API server with a minus, minus help. And that'll dump out all the switches and there are a lot. And you can also, you know, use the documentation for reference if you want to, but this is a complete list of the current with Kubernetes 119.3 audit log options. Now, if you want to, you have two possible back ends for the, the API servers audit logging, one of them is a local file or doesn't have to be local a file. And then the other one would be a web hook. And so there's a posting protocol for the web hook to receive all the events. And in either scenario, there are lots of settings. Right. So these are all the log settings. These apply to a file based log output where regular file. It would be what the API server was doing. And these are the web hooks where it would be an HTTP, you know, post style output. So that's the, you know, the basic configuration stuff. So now let's, now that we kind of got an idea of audit logging, we've seen some events. We know how to configure servers. We understand this policy thing. So what are some of the concerns that you run into in practice with this? Well, one of the first things that you run into is having multiple API servers because nobody in a production environment is going to have one API server because the API server goes down, your cluster is dead. Two is probably fine. For most clusters, you get, you know, that way, if one of them goes down, you still have the other one. And, you know, it's, it's pretty unlikely you're going to lose two. And, and when you add API servers, you, you get some scaling ability, right? Because the logic being processed by the API servers is, you know, now distributed. But really at the end of the day, you know, maybe I'll just go back here to this previous model. At the end of the day, the bottleneck is at CD. So at CD is an in memory key value store. This is why the audit log is not going here. Right. It would totally at CDs already a bottleneck just keeping up with the, the, the, the configuration of the cluster. Right. And the events that are happening because those, you know, the eight, you know, control plane level events, those events are actually being stored in at CD for a period of time. But the audit log data, that's massive. So we have to have a completely different channel, totally independent for the, you know, the audit log. And when you think about this at CD is, is often in a production system running on a different cluster. So you would have a, you know, a three, you know, well, in production, you probably have a five or a seven node at CD cluster. And so, you know, the API server, a cardinality is independent of that CD. If you're, if you're not running them on the same box, right. So a collapsed at CD API server where the at CD nodes are running on the same machines as the API servers is, is an okay way to do things. But if you're doing things in that way, then what defines the number of API servers you have is the at CD cluster size, not the API, you know, not the API servers. Two API servers is fine for high availability for most clusters. But here's a problem. When you have multiple API servers and let's say you have three, you're going to need a load balancer. And so you might set up, you know, if you're using Google Cloud, you might use a network load balancer on Amazon or something. Google Cloud use their load balancer, Azure use their load balancer to basically front end your API servers. And all your, your, your cube kittle configs are going to refer to the, the TCP load balancer. And so they hit this guy. He just forwards the traffic on to one of these API servers. And you don't know the difference, but there's a health check so that when one of these guys crashes, they only send you to the guys that are alive. Well, that's great. But the downside is you don't know which one of these guys is going to be dumping out the audit log information that you're interested in. So you could do something like run a pod, as you can see down here, and then, you know, tail the audit log looking for, you know, some sort of pod activity and not see it, because you're looking at the wrong audit log, right? And you came in here and hit this guy and this guy logged the activity and these two logs don't have anything about it. So the API servers are shared nothing, right? They're microservices. They don't know, you know, what's going on in the other server. They're really focused on, you know, being as independent as possible. So the downside is your audit log is now sharded basically. And so to unshard it, you're going to do something like run a fluent D or a log stash or a beat or a fluent bit or something like that to splunk forward or something to move that log data into a back end where you can get a complete picture of what's going on in the cluster. And so that's important. Another thing about these distributed, you know, API servers is that on the upside, you get scaling, right? So if you've got, you know, huge throughput going on in your audit log, you just divided it by three by having three API servers. And so, you know, as long as your network can handle it and you've got, you know, the bandwidth on the actual wire, you're using three nicks, you're using three sets of memories, you're using three sets of disk, whatever the case may be, you've really got scale there. So this is one way in which actually having multiple API servers can in fact have a dramatic impact on your scaling challenges because we're not using audit logging, you know, I mean, one API server can handle a pretty honking big cluster. It's the Etsy D cluster. That's always the bottleneck. And so, you know, two API services, good for HA three is even better. But, you know, if you add audit log challenges, you might want to go to four or five and, you know, get your audit log scaled out, you know, using more API servers. And remember, the cardinality could be completely different from the Etsy D cluster because, you know, usually that's a separate cluster of servers. Another thing people often, you know, stub their toe on for a day or two is using config maps. Config maps are awesome for configuring things. And you might say, hey, I'm running the API server in a pod. Why don't I set up the policy as a config map? Well, you could, but what happens when you start the cluster? There's no API servers running and you fire up the first API server. And for him to configure himself, he needs a config map. Well, how are you going to get that config map? You need to make a request to an API server that's going to hit Etsy D and give you the config map, but there's no API servers. There's a chicken and egg problem. So most people skip that and do some other technique to standardize their policies. But this is another pitfall, right? What if API server one has one policy and server two has a different policy and server three has a different policy? I mean, there could be excuses for doing that. It is totally possible, but it's going to give you weird asymmetric log output, right? From the different servers. So in most cases is that I've run across. You probably want them to be the same. And you might want to have some, you know, sort of immutable infrastructure, Ansible, Sol, you know, whatever type of thing that's, you know, keeping those files in sync or, or have them from a shared disk, you know, or something. So next thing to talk about, we've got a few more things here. I think we have to wrap up, but I'll, I'll try to hit a couple more things here and then we'll see if we can get some time for questions. So mutating admission controller webhooks. It can be useful to know which mutating webhook mutated an object in an API request. And if you've got a, you know, if you've got a bunch of plugins into the API server, that are potentially changing the nature of a resource that somebody created by default, the API server won't know anything about it, right? It's going to call these guys and it's, it's they do what they do. And then the API server just, you know, moves on to the next unit in the chain. And so what we want to be able to do is see where in the chain, you know, change A happened and where in the chain, change B happened. So a popular example would be Istio, for example, and the Istio proxy injector. So I create a pod, I'm oblivious. I just, you know, wrote my app and I put it in a pod and I go to deploy it. And now the, the API server says, oh, we have a, we have a, you know, a mutating admission controller here that wants to mess with this pod. And what is he going to do? He's going to add to the pod of an init container that's going to rewire all of the traffic in the pod. And then he's going to add a sidecar for the proxy, which is going to intercept all the outbound traffic from the main container. And he's going to then, you know, TLS encrypt it and do MTLS and init tracing data and whatever else he's going to do. But that mutation is fairly complex and it could interact in weird ways with other mutations. And it becomes hard to sort of figure out what's going on unless you have some way to introspect. And this is exactly what mutating admission controller webhooks can do by using annotations. And so you can see in the example here, mutation webhook admission controller, Kate's IO round one index two. So if you're familiar with admission controllers, we have, you know, different phases. So round one is, is the first pass but then we want, once everything's mutated, you might also have, you know, an admission controller that's going to allow or disallow only. And so that, that admission controller, you know, would come up in another round. And so we have these different rounds and then we have the indexes. So in this case, we're the second of the, of the mutating controllers. And so we have the configuration and we specify some configuration data. We have the webhook information and then we have the status of whether this guy mutated or not. So in this case, this, this controller did not mutate the resource. And so that's a nice, you know, piece of documentation that you can now get from your system. You can, you can see, you know, if you're mucking around with admission controllers, you can really look in and get a deep dive into what's going on. Another thing that we can do is we can specify the actual mutation. And so if you have a request level, you know, audit or hire, this is all you'll get. By the way, if you are just at metadata level. So just yes or no, I mutated this object. But if you're at request level or hire, which is more detailed, then as you can see, we get the patch, right? And so you actually have the, you know, the information about how things were changed, you know, really, really can be useful again in debugging scenarios. Okay, so I think a couple more things and then we'll wrap up here. So auto log monitoring, the, the API server, it has two open metrics style metrics endpoints or metrics metrics in its slash metrics endpoint. And one of them is API server audit event total. So that's the cumulative total of audit events. And then there's API server audit error total. So that's the total number of events that were dropped due to an error and exporting. So I haven't talked about this just yet, but if you're, if you're, you know, dumping huge amounts of information to a backend, like an elastic search or a fluently aggregator or something. And you're overwhelming it with IO's. One way to fix that problem is to batch a bunch of events together and do a single IO with a collection of events. And so you can reduce the number of IO's by a factor of 10 by just collecting every 10 events and submitting them as a unit. And that often solves problems. Another thing that you can have is you can have, you know, sort of up and down, you know, performance in these aggregators because they might be servicing lots of other, you know, long streams. And so you might need to buffer your output. You might send them a batch and a whole mother batch and another batch and another batch, you might have five or six batches waiting. And then as soon as they process that first one, then they might catch up. So you need to sort of look at what the lag is and figure out a buffer size that also works for them. And so if you end up running out of buffer, you're going to drop events and this will tell you if you're doing that. So those are both really important because the first one event total is going to give you an ability to sort of estimate and discover spikes. So if you have a Prometheus or something system monitoring the metrics from your API server, you can plot that and look for anomalies or trends. You know, if you're increasing continually day after day, you might want to make sure you've got the head room to get to where you're going to need to be in a month. And then errors, of course, are always nice to know about. So this is just an example dump running a cube cuddle proxy on a machine because, you know, to avoid all the TLS stuff that you need to get the metrics and then just curling the proxy to get through to the API server on the metrics endpoint. And you can see that we've got the audit event in this case is what we're repping for the total. And you could pull up the error in the same way. So that's some of the metrics. Other things, handling massive throughput. So we talked about batching, blocking, and strict blocking. Batching is where you're going to buffer events asynchronously, blocking. You're going to actually block the API server responses until the event is processed. So that's, you know, that's pretty draconian and will impact users. And then blocking strict is the same as blocking, but when there's a failure during the audit logging, the whole request is rejected to the user. So that's even more strict. So batches, you know, typically what people would set to for their, the buffering strategy. And then there's a, you know, buffer sizes, wait times, throttling, all sorts of things that you can set up here to help control the throughput. Other considerations. Remember that each API server is independent, shared nothing. And so scaling them can give you some scale. So if you're back in webhook is the bottleneck, you're going to have to think about that as well, but you can scale the API servers to scale the stuff out. Okay. I think we're getting pretty close to the end of time here. So I'll wrap up. Thanks a bunch. Really appreciate your time. And maybe we'll see about some questions. Okay. Well, thank you, Randy, for that excellent presentation. We have about five minutes for questions. So if you have anything you'd like to ask, please drop it into the Q&A box. The first question here is the JSON output formatted according to CADF specifications. That is a good question. So the types go source is the, the, you know, defines the structure. So I'm not positive. I'll, I'll, I'll see if I can find out though. And maybe I can post an answer with a follow-up in the, when we, when we upload things. Not a problem. Does anybody else have a question they would like to ask? We have a few moments. So please feel free to ask away. Would you be able to elaborate a bit more on the relation between an API server and FD? Sure. Okay, let me, let me back up here to this picture. So the, the FDD cluster. I was going to see if I could draw something, but I don't think. Oh yeah, I can't. Here we go. I used so many different darn presentation tools these days. I have to keep track. So if you have an FDD cluster, let's say, let's do it simple. Let's just keep an example of three. That FDD cluster with three nodes is going to use the raft. Consensus protocol to elect a leader. And so let's say this guy's the leader. So if you've got, let's say three, let's say you've got five of these guys just to make it a little bit more interesting. So say you've got five FDD nodes and you've got three API servers. So the API servers are all going to write to the leader. And in general, they're going to read from the leader too. And you might say, oh my God, that's terrible. Because the more API servers you have, the more load you're creating on that leader. And while that's true, at the end of the day, the FDD clusters highly consistent. When you write to the leader, it has to write that data to all of the other nodes in the cluster. And furthermore, because it's a highly consistent key value store, it has to know that a quorum of the nodes have committed the data. And so FDD becomes the bottleneck in most cases when you're, when you're, you know, experiencing control plane challenges. And so the, you know, the API servers are, they're anonymous, faceless, identity-less microservices. You put a load balancer in front of them, you hit the load balancer, you don't care which one you get. Because no matter which one you talk to, it's always going to give you the same picture of the world because you have this highly consistent key value store that stores all the state. The API servers have some, you know, caching and things like that. But at the end of the day, everything comes from FDD. So if you ask number one, two or three of those masters, what pods are out there, they're all going to give you the same answer. And so the state in FDD is the real, you know, bottleneck, the management of that. And unfortunately, adding more nodes to FDD slows it down. The fastest FDD cluster is a single node, because he doesn't have to copy the data to anybody. And so the reason that you need to have multiple nodes and that production systems are usually like five or seven, is that if you, for example, want, you know, diversity and failure tolerance, resilience, what you want, you know, in most cases, you can have three availability zones, for example, in the cloud. And you can run your FDD cluster across all three. And if you lose any AZ, you still have a quorum, right? Quorum is an over two greater than. So in this case, five over two, that's two and a half. So three would be the next higher integer. So any three of these guys and we're good. So we could lose a whole AZ or if a node crashes, let's say you take a node down for maintenance, you can still have another node crash and be okay. Seven is a little bit safer than that, but it's a little bit slower. So that's sort of the relationship, right? Stateless microservices, the API server, usually behind a load balancer, like this would be like a Kubernetes service sort of, right? But in the case of a load of API servers, you'd usually use, not always, but usually use something else because you want people to be able to access the cluster who are not in it. And so services, a load balancer service, could make sense, but again, you got to worry about chicken and egg problems when you're creating resources in the cluster to access the API server. The API servers, the thing that gives you access to those resources. So usually some sort of external load balancer in front of the API servers and then the API servers communicating with the leader of the Etsy cluster and the Etsy cluster leadership is dynamic. So all the API servers are typically going to know about all the Etsy servers. So hopefully that answers the question. Thank you very much, Randy. And I want to thank you again for a wonderful presentation. Unfortunately, we are out of time. I would like to thank everybody for joining us today. And as I said before, today's webinar and slides from today's presentation will be available on the CNCF webinar page at cncf.io slash webinars. Thank you everyone for attending. Thank you again, Randy, for a wonderful presentation. Everybody take care, stay safe, and we will see you at the next CNCF webinar. Thanks, everybody.