 So yeah, thanks for coming. Thanks for having me here. For me, it's a strange feeling because last time I was here, it was 10 years ago. I was checking that students are not cheating. We're writing there, I don't know, some tests or something. Because I tried to do PhD here, but I failed badly. Yeah, today's topic will be log to our bug, which is in my toy project. Yeah, we'll learn something about it. Before I dive into the topic, let me introduce myself. My name is Irka Kremzer. I used to work for Red Hat for seven and a half years, actually. I still do recognize some familiar faces in the audience. I also work for Oracle, Apsa Group. And actually starting yesterday, I started working in jazz from the IO. This is what I like. I like 3D printing. On the top right segment, this is actually graph on the board for my printer where I can monitor the temperatures and whatnot. These are the 3D printed chess. What I also like is doing road trips with my family. These are some of the road trips we've done. Last one is two months old from Seattle to New York. It was real fun. And I also like to hack on Kubernetes operators. Actually, our previous talk was about such an operator, so it was a great segue for my talk. These are some of the operators I've created. Fun fact is that half of them is written in something that's not GoLang. This is Java-based ones. And the most famous one is probably the KAGB, which was accepted to CNCF sandbox. But I'm only maintainer. I haven't found it. So today, I'm going to talk something about this look to our bug operator. So first, let's talk about Kubernetes in general. To give you some motivation for the tool, try to imagine you are a developer. I think it's going to be easy for you to do. That you are working on operator and Kubernetes. But basically, on any cloud-native application that has a Kubernetes client in it and calls the Kubernetes API. If you do that, you end up in a situation that when you add features, you also need to tweak your air bug rules. But the problem with it is it's different language, different paradigm. You're coding your operator in some GoLang or Java or some imperative paradigm style. But then you have to switch your mind to YAMLs and do decoratively to tweak your airbag to allow your application to do its thing. So you have to switch all the time. And what people end up to do, or what I end up to do, is basically say, I don't care about the airbags. I just give my application cluster admin and do the security eventually. But this is a bad thing to do because it may, you basically never do it in the future. And it opens a huge potential risk for your cluster because if there is a cluster admin role, these days you've got all these various attacks for supply and chase, to software, where if somebody attacks your application and says, I don't know, it will have full cluster admins basically and do whatever we do cluster. And another use case for the application is given you have a Blackbox container, you found an internet and you want to make it run, but you don't have any YAML manifest for that. I did that many times. So you basically end up in some errors where it's saying, hey, you can't start your application because it wants to list a post, but you don't have such a right. So that's another use case I'm trying to solve. And before diving into the topic really, I will have some like one slide about what RBAC actually is for Kubernetes. So we have basically this model that you've got service account which is tied to a web pod through something and the service account has a role. And this relationship is done through object called role binding in case of role or cluster role binding in case of cluster role. What the role is, it contains a set of rights, so right or rules and rule is a verb and object. So for instance, there are examples like you can list posts or watch deployments or star secrets. The star represents all the verbs and it can be also on the object place. So this is basically the same situation in the diagram. There is also like more complicated concepts like aggregated roles and you can tie the role to user or group, but let's abstract it from it. What's important that these rules and these objects are persisted in Kubernetes in YAMLs and as a story at CD Database. And this is, there are cumbersome to write. So I made an observation that most of the current Kubernetes clients have relatively stable error messages. It's structured, so for instance, if you are a Zingolin client or JavaScript client or Python client or Java client, these messages when it comes to RBAC will be relatively similar. So wouldn't it be nice to have a tool that would basically scan your application during runtime and give you these RBACs? It would, right? So that's what the log to RBAC does. And it's inspired by the audit to allow logic. You might know it because there are a lot of registers. It's a tool for SE Linux where it basically was able to turn a breakage of a rule to a policy, to a rule. So I'm doing the same, but doing it in runtime. I'm basically running this operator next to your application and it scans the logs from the application. And if there is a breakage of RBAC, it will capture this event and turn it to a rule. I also got a kubectl plugin, but talk is cheap. I've got a demo prepared, but I'm not sure if it's visible, so let me. Yeah, well, I have to zoom a lot. So I will start the cluster because nothing is running currently. Okay, cluster has been started. Faster, so I will deploy an operator for Prometheus, which is an upstream famous operator for Prometheus, but without any RBAC rules. So it's gonna not work. So let me read. I can show the manifests. Yeah, so it's just a deployment, one deployment. No RBAC rules. I've got nothing in my sleeves. So if I deploy this thing, all right, it assumes a namespace called monitoring. So let me create namespace monitoring, create and do the same again. So it has created a deployment, but no puts are actually running. So let me check what's happening. Right, I don't have a service account created because it was only a deployment. So I also have to create a service account. And this is actually, operator can create the service account for you, but I want to demonstrate that this application is not running correctly without the RBAC. So if I list all the puts, it should make the font bigger a bit. Yeah, this is the, so it's currently starting. If I show you the logs from it. Yeah, there's a lot of error messages. And basically the common topic is something like, hey, there's a service account with this name. We just created a service account called Prometheus operator. Cannot list, Prometheus is in API group, blah, blah, blah. And this is the stable log messages coming from RBAC breakage. So now I'm going to install the operator that's going to scan these log messages and turn these errors into a rules. To do that, I've got a kubectl plugin. So it's called kubectl log to operator. Sorry, log to RBAC. And you can install it actually from using crew. So if you do something like do pre-install this guy, it will install this sub command to kubectl so that you can run it and it will create, it will open this interactive UI for working with operator. One of the options is to ability to deploy it. So let me first deploy it. So it deploys the operator itself and the custom resources. If I list the pods, I can see it running, almost. They're almost, yeah, it's running. So let me now demonstrate how it works. In this section, I will be watching for our ends. These are the custom resources for our operator. It's a short version. The long one is called RBAC negotiation. It's what I call it, the process of negotiation with RBAC. So currently there are no such resources in our cluster. And here I will be watching, by watching, I mean like periodically running the commands that's describing a cluster called new Prometheus operator role. And there is no such role at the moment, but if I run it, it says, yeah, nothing has been found. But if I request the negotiation process, it will create the role class and it will be trying to populate the role with the rules. To do that, I can use the plugin again. And there is a, not the deploy, I'm sorry, it should be negotiate. Here it asks for a namespace. So our Prometheus operator was deployed in a monitoring namespace. Now it asks for deployment or type of a resource because it can work for deployments, replicas, sd, monsens, and yadda, yadda. So it was a deployment and name of the deployment. And here we can see actually the shape or how the custom resource looks like. So this is a very, very simple one, but you can also specify in case of multiple, if you have multiple containers in your port, you can specify which container you are trying to scan, but also other things. So if I run it, it's great. Yeah, now you can see that it has created the resource, Arbeck negotiation. I will take it. And it has added one entry at the moment, and this is the role. And it says like, hey, now the operator can list and it's got other measure configs. And what the operator does, it actually in each iteration, it restarts the port and starts the application again. And at the end of one time, it takes only one rule and edits to a role. By default, it's waiting 20 seconds because in general, it's hard to guess how application takes to start. But so I choose number 20, but it's configurable and but it's low. And I know that application starts much faster because it's written in Go. So I can tweak this number to something shorter and increase this, speed up this process. So let me actually stop it here and do it right now. So I've prepared make file task or target, reset and speed up. If I provide dash end to make file targets, it will just print it and not run it. So this is what it will do when I run it. It will delete our original Arbegg negotiation resource that's responsible for the process, or delete the cluster role, cluster role binding and change the deployment of our operator to give it different environment variable that says, hey, sync the, sync after two seconds, by default it was 20. So if I do it without the end, it will just run it. And now operator should be, I've got to sleep five seconds there because it takes some time to deploy. So operator should be running. And now if we're lucky, the process should be relatively fast. Sometimes it can miss the event and if it misses the event, it goes to a different iteration and then should be fast. But yeah, so now we should, yeah, you can see that it's like two or three seconds for each rule, which is kind of good because if you imagine that you can plug into your CI CD systems, you can basically run it all over again with your, together with your end to end tests. So that's some of the demo. And yeah, so implications, I've just already mentioned it. So yeah, it assumes that you basically, you are using the application together with the operator because what I've just shown was just a startup, right? I just started the operator, but I haven't actually done anything useful with the operator. So if I switch back, it says like, there are like 14, there should be I think 14 rules or something. But at the end, it would just verify and edit the rules just for the startup. But what I want to achieve is, if you run your application together with end to end tests, basically export all the possible code paths in your application together with our electric operator, you should be sure that the role that's end up from this process is tailored specifically for your application. It's not like the rights are not bigger or smaller than the application, but it's perfectly for you. Also, it can notify you that you, like imagine you're developing an operator and you remove some feature, and I'm pretty sure that people tend to forgot to remove also the RB Google. But if you have such a tool that syncs basically the declarative rules in YAML with the imperative style in GoLang, it can be helpful. I have also an idea of static dynamic analysis or sorry, static versus dynamic analysis because what I do is basically dynamic analysis. I run the application and do something with it. But it can be also done by just exploring the code, building the ASTs, and basically trying to figure out what the application does. But this way it would be language dependent. You have to do it for GoLang, for Java, for Python. But this look to our back thing works for all of them, for free. Yeah, that's it. And thank you for going and this is the links for the deck and repo. QR code doesn't work. Hey, so I have several questions. First of all, as you mentioned, this behaves like OD2LO, even though I see that this acts immediately. Is there an option to do it just like with OD2LO that like to generate something which we decide afterwards to apply or not? No, but there is a tool that can do that. It takes different approach. It's basically, it's very similar to my tool. It takes the logs from the, I think it's not admission and control, but there is a way to run your API server Kubernetes in an option to basically audit all the requests to your APIs and it will create an audit log. So there is an exit tool that takes an audit log and creates those RBG rules based on this audit log. The problem with that is not like continuous process. It's just one past thing. Yeah, both can work in different environments. Yeah, also the other thing is that, like do I understand well that like it always acts how the application acts? So like, I mean, in case the developers, the QA whoever did not think about some action when applying this thing, then it would just allow some things in the middle of the run. Yeah, yeah, you're right. With Grimm's power comes great responsibility. It's like giving to the application and you are right. It does work when, during the time when the custom resource is present in the cluster. When you remove it, it stops working for the application and I introduced also a feature when you can pause the process. So this, I actually, sorry, I don't have the custom resource example on my slides but there is a in spec, under spec, you have a post flag and you can say post through and in that case it will be switched. Those events from this custom resource will be switched and the RBG negotiation will be paused in quotes. Okay, thank you. Thank you.