 Hi everyone. Welcome to our talk. It's about project updates on the network policy API work that we've been doing in the Kubernetes 6 community. I'm Surya and I'm an engineer working on the OpenShift networking team at Red Hat. I'm also a contributor to the network policy API project and very, very happy to be here at KipCon with all of you. Hi everybody. I'm Hunter. I work at Microsoft on network policy and observability and today I'll be diving into user tooling. I'm very excited to show you guys a demo later and I'll pass it back. Yeah, so let's get started with who are we, right? So how many of you are hearing network policies for the first time? Can we have a raise of hands? That's perfect because everybody knows why you're here for, right? So we are actually a sub working group within the SIG network. So a special interest group network, but we are a sub working group within that group and we call ourselves the network policy API working group. And we are mainly focused on developing new policy APIs and maintaining the existing ones that we already have. And these are basically the network policy API, which is stable and V1 and has been around for a really long time, five, six years, if I'm not wrong. And it lives in the core Kubernetes slash Kubernetes repo. And recently we've been working a lot on the next two APIs, which is what we'll be talking a lot about in the session, which is the admin network policy API and the baseline admin network policy APIs. So we'll dig a little more deeper into what those are. And if you are interested in our code based repo or our websites, please take the privilege to go and click on these links when you have the PDF from the sketch available. So let's look a little bit into what network policy API is, but I can see that a lot of you at least know what that is. So that's good. And I want to give a huge shout out to numerous contributors in the community who have made this possible. So for the past so many years, I don't even know the extensive list of contributors, but huge shout out to all of them for making this possible. Some of the use cases for network policies and the reason behind why we have this API in the first place is the fact that it was designed keeping application developers in mind, right. It was meant for namespace owners. So as an application developer, when you have your namespace and you put your apps on it, security is an important perspective, especially the networking security aspects of it. So you might want to control what's talking to your pods, what can talk, what should not talk and stuff like that. So that's the basic use case for why we have network policy. The second thing is having a tiered architecture. So you might have front ends, back ends, databases and bunch of other stuff around it. And you might want to be able to say that your back ends can only receive traffic from your front ends, databases can only receive traffic from your back ends and have multi tiered architecture. So these are the two of the most common use cases we had in mind in the cap for why we designed network policies. And this was really, really for the developers, right, like for the tenant owners who deploy apps. And like I mentioned, this is like a very basic example of how you can use the network policy API. So you might have two namespaces, the front end and the back end. You have a bunch of pods in each of these namespaces. And you might want to be able to express things like, hey, I want my back end to only receive traffic from my front end, right? So ingress traffic, let's say, in which case you just add a network policy to your back end namespace, which creates this implicit deny, which is the red block that you see. So right now all ingress traffic is blocked. And then you can apply and allow rule on top, which is the green line that you see right there on the slide, which is saying that any traffic coming from my front end, I want to allow. But nothing else is allowed, right? So your namespace is basically very secure at the moment. So like I mentioned, network policy API is namespace scoped. So which is why you see in the metadata the namespace field. This is a sample YAML for network policy API, which a lot of you have seen before. And it's simple because it lets you express simple ingress and egress rules on that CRD, right? But one thing to keep in mind is the fact that it is implicit in nature after all. Because the first network policy that you create in your namespace gets you that implicit default deny to that namespace, right? And then any allow rules that you are on top is what defines who can talk to you. And some of the peers that you can express today with the network policy API are the namespaces, the pods, and IP blocks. Stable API has been around for a really long time. We have a lot of implementations. We don't have the extensive lists, but these are the commonly known ones that are still active in the community. And let's talk about admin network policy API, right? So this is the new API. And remember how I mentioned the network policy API was designed for developers? This one was designed for admins, right, to cater to their needs of the cluster. And we have two objects, two CRDs as part of this feature, which is the admin network policy API and the baseline admin policy API. Both of these are out of three. So they're not in the core Kubernetes slash Kubernetes repo. So when you install your cluster, you do not get these by default. You need to actually opt into them. And they are currently in the Webern alpha, alpha one actually. So we've, it's been around for a year and a half, right? They kept merged two years ago and the API has been drawn for a year and a half. We are still actively developing a lot of features and adding stuff into it. We are preparing to move towards beta, not there yet, but hopefully in a couple of months. We'll see. And again, thanks to numerous contributions here. I want to particularly call out our maintainers Dan Yang and Andrew who do numerous code reviews because of which this project is still sustainable. So some of the use cases for the admin network policy API, which is designed for admins. Again, as a cluster administrator, you might want to have or enforce security for your entire cluster, not a specific namespace, but actually have rules that span cross namespace, multi tenants, isolate tenants, have non-overrideable rules, which actually cannot, well, non-overrideable rules as in the network policies cannot override them, which are created by the developers. So, and all tenants in your cluster have to adhere to this, right? So that's our number one use case. And then tenant isolation. A tenant here could be one namespace, multiple namespaces, and you don't want to really replicate your network policies across all the tenants to express the same stuff. So you can just have a global cluster scoped policy, which just does this work for you. Again, we can look at some of the use cases which was mentioned to the original cap of the admin network policy. The first one being I already covered this, which is isolated tenants, right? So you have the full tenant, the bar tenant, you don't want them to talk to each other. Explicit allow, which is you might want as an admin to tell that your tenants or your applications should always receive traffic from the monitoring namespace. So you might want to monitor, gather metrics, check what the application owners are doing in your cluster as an admin. And so you might want to do explicit allow. So which means if there is a network policy in that tenant's namespace, it still doesn't matter, right? Like the implicit denial of the network policy will not kick in. Your allow is what holds precedence. And same for the ingress case, which is the fact that the Qube DNS, which is the internal DNS in your cluster, you want to always allow all the tenants to talk to it. Another interesting use case is the delegation, or commonly known as the pass action. So we wanted to make sure that the admin network policies coexist with the network policies and they interact with each other in some manner. And the admins should be able to choose to delegate specific set of traffic. So you might have an implicit deny rule as an admin saying, oh, nobody should be able to talk to anybody. But then you might have these explicit rules, which is if the traffic matches this side or this network or this node, you want to pass, which means you want to delegate the decision of whether the traffic should be allowed or not to the namespace scoped network policies in your cluster. So that's the yellow diagram that you see here. And then the last use case is default deny to sensitive namespace, right, which is the explicit deny. How can you do this, right? So I mentioned there are two new APIs as part of this new feature, one of them being the admin network policy API. And this is how a sample YAML looks like. It's kind of like network policies in terms of, I'm sure you recognize the ingress and egress and the selectors, namespace selectors, pod selectors, but this is cluster scoped. The first thing to note and then note that you have a priority field, which is explicit. So you can have multiple admin network policies and actually order them based on priority. So that gives you a clear way of saying which A and P will be evaluated first, right? So zero being the highest priority and then as the number increases, lower the precedence. Within the admin network policy, it's the same. So you can have multiple ingress and egress rules, which explicitly allow and deny traffic. In our case, we have taken the example of the monitoring namespace and the DNS namespace, which we spoke about in the previous slide. So here we have an ingress rule that says any traffic coming from the monitoring namespace should be allowed. And we have an egress rule that says any traffic towards the DNS namespace should be allowed always. So even if you have network policies, doesn't matter, right? And your subject here is all namespaces. You see the empty selector for the subject field. And then this is the baseline admin network policy, which from the name of it might sound a little bit confusing because I've had multiple people reach out to me saying, hey, is this the baseline that I put in the cluster, which will always be adhered to? The answer is no, actually, because the precedence of baseline sits after the network policies. So this is evaluated after the network policies. So you want to be careful about it. So this is not something that you put on your cluster and say, oh, yeah, I have this, so I'm secure. Not really. If you have something that you really want to do as an admin, please create an admin network policy. Baseline is meant to be used when you have past traffic and then you have network policies or you might not have network policies to catch them. It's a catch thing that sits at the lowest precedence in your cluster. It's explicit in design. It's like network firewall admins telling what they want to allow and deny. So it's not implicit like network policies. So that might catch you by surprise if you have been using network policies for a really long time and then you switch to using this. Because if you want to deny something, you have to explicitly state what you want to deny. And we will only deny that. We are not going to deny everything for you just because you created the object in your cluster. And then we have some, the commonly used peers or the allowed peers are the pods and namespaces, but we also have nodes and networks. Networks are like the IP blocks. So the nodes and networks are only supported for the egress at the moment. Only been around for a year, right? We are on the journey towards beta. Only two implementations exist today. And this is where we really need help from the community. We want more implementations. We're having some dialogues with Cillium. We want help from Calico. So shout out to all of you here if you have implementations and you're interested, please to add this to your roadmap. Give us feedback or maybe even try to use it and let us know how it goes. And just to sum all that up because maybe that was a little bit lot in the short time. It's, this is how it's evaluated. So this is the admin network policy API which comes first. It has priorities. So it's evaluated based on priority. It's created by the cluster admin. And that's what gets evaluated first. And then you fall down to network policies if any in your cluster, or maybe you pass the traffic in which case it lands to the network policy. And then the network policy rules are evaluated. And finally, if nothing matched these two layers, that's when you fall down to the baseline admin network policy, right? And it's a single ton. You can only have one BA and P in your cluster. So you use it wisely. So that's a little bit about the APIs that our community supports, maintains, develops. And we also maintain all the ecosystem around it. So new features, end to end tests, conformance tests, bugs. So feedback, right? Everything. So we're going to dive into some of the new features that our community has been working really hard on for the past few months. And we call them the network policy enhancement proposals. We kind of borrowed this from the gate to API enhancements, the gaps. We call them the NPEPS. So this is basically how the community can just come up and then propose their ideas, right? If you have a cool idea, you want to get involved in the community. Maybe you have a security use case. Maybe you have other use cases for policies, multi-clustering, or whatever that might be, multi-networking, whatever that is. Just open an NPEP with the user stories that you have in mind against our repo. This is really low key as in it's not super complicated as the cap. It's literally what are your goals? What are you trying to do? Just open a PR and then we can get it reviewed. And then if the use cases make sense and are generic enough, then we can start implementing the API, get that in. And then these are the stages that the NPEPS go into. But basically the point TLDR is contributions are welcome. We are a small community, but we love getting feedback in more use cases here. So the first NPEP that I do want to talk about is the egress traffic controls. The original KEP for the admin network policies, the scope were limited to east, west, and the northbound was not supported. And so we did an NPEP to add that feature. And it's an experimental status right now, which means that the use cases were accepted. We also have the API merged, but we will wait for six months before it goes into standard channel so that we can get some feedback from the implementations. And we are sure that the API semantics are correct. So this NPEP basically introduces two new peers, which is the nodes and the networks. And the networks are actually in line ciders. So you have to put your ciders as a peer within the object. Some of the use cases are what you see on that side of the slide, which is you might want to block egress traffic controls to your control plane pods at a specific port. You might want to see the tenant who can talk to the DNS, but not talk to some other external servers. You might want to deny everything to your 0000 slash zero. The next one is the CIDR group peer NPEP. This is kind of a continuation of the previous NPEP. So when we did the previous NPEP, like I mentioned, the networks was actually an inline peer, right? Like so you have to put your CIDR blocks as within the ANP and the BANP. And if you're referring to multiple CIDR blocks, so if you have a really dynamic set of ciders which keep changing, you might have use cases for it. So you might want to actually have a separate object called the CIDR group. This is not much. This is just like a sample proposal for the NPEP. And then you might want to refer to the CIDR group from your ANP or BANP. This NPEP is currently in the provisional state. And a shout out to Michael and Joe, who were the ones who proposed the user stories here. So we are trying to discuss and have more discussions around if this makes sense. In addition to the inline site appears, how do they work with each other? Should we do it or not? So if you have use cases, please comment on the NPEP if you would like to see something like this, right? So the next one is the egress FQD and traffic controls. Again, shout out to Rahul, who's been working really hard on getting this in. User stories are merged. So it's kind of, again, egress peer, but proposing domains or FQDNs as your peer, being able to express that. So we are adding support for this. We have had lots of people come up to us and say, we want to see this in network policy, we want stable, but we're not accepting features there, right? So we're at least doing the ANP. So hoping that this is going to be useful for many of you here. If you have, and this is currently in a stage where the API design is happening. So again, we are looking for some feedback on this. The next one's the ingress traffic controls. This is being a pioneer by Nadia. She shout out to her for all the amazing work that she's been doing. And this is the inverse, right? So we want to, maybe you have southbound use cases. We are a little bit struggling to figure out how this use case would look like, because for the southbound traffic that's trying to talk to your pod, you always have an SNAP or an entity that sits in the middle where we are still figuring out how the policy interaction would look like here. So this is in the provisional state. Please reach out to us and comment on the issue if you have user stories where you want to have security posture controls for inbound, right? So external peer to pods who are your subject in the cluster. Tenancy expression. So remember that we mentioned that one of the main use cases is a multi-tenancy model where you want to isolate tenants, but tenants might not mean single namespaces, right? So you might want to express it in a better way than using selectors, like saying anything that's me or anything that's not me and me here being the subject tenant. We are trying to express that, add a feature which will allow you to dynamically express it in a much better manner. And this has been an ongoing NPEP for a long time and we are trying to converge on how that API would look like. So again, appreciated. If you are interested in this NPEP, please do leave us feedback on the API design. Conformance profiles. I gave a talk earlier today, so I don't know how many of you saw it, but this is in collaboration with the gateway API community. As we are developing... Sorry. As we are developing new APIs out of tree, it's also important that we think about conformance tests and how the implementations are going to help give us feedback and how we can track them better. This effort is actually aimed towards that, where we want to have... where we defined clear profiles and test suits that the implementers can then just take and run from our library, run it in their suit, give us the report back, right? So this is kind of meant for implementers, but for end users, the benefit that you get is there might be multiple implementations doing A&P or B&P. So as an end user, you might be confused on which CNI should I use for the policy. This will help you because this shows you what CNI supports, which features, and then you can choose which one you want to use. So that's what the end users get from here. So that's all the new stuff that we've been doing in the recent past and now coming to the future plans for us. Network Policy V2. Should we? Maybe. Let's look at some of the problem statements that we have with the existing Network Policy framework. I think I covered some of them in the beginning, which is the implicit deny, right? The first policy that you create in the namespace just blocks everything for you. This is not written anywhere. It just works that way. The other syntactic works might be, let's say I just mentioned implicit deny, but if you have an IP block peer, and I don't know if you've used the accept in that, so you might have a block, but you want to accept some of the IPs within that block. So that lets you kind of explicitly deny stuff or not allow stuff. So it kind of is weird in the sense of how the API is designed. So because of all those reasons, we wanted to start fresh, right? So also it wasn't covering the admin use cases, but we fixed it with the Admin Network Policy API. So I'm just going to quote Dan here, right? Which is, we did A&P, which is a better network policies for administrators, and today administrators are actually using just network policies, and you can easily switch the A&P. Should we also be doing a better network policy for developers, right? So which follows the same explicit list and easiness that A&P and B&P API is offered today? This is still just an idea. We actually need somebody to just own and drive this in the community. So if you're interested in working on network policy V2, please reach out to us. We are looking again for feedback, use cases, ideas that you might have here. With that, I want to hand it over to Hunter, who will talk about all the exciting work that he's been doing with the user tooling and show us a cool demo of how all of what I said works under the hood. Yeah, thanks, Surya. So I'm going to shift gears a little bit. I'm going to be talking about how you can get your hands dirty with these policies today and how this will be helpful when you're using them in production. So I think we're all familiar with the Monday morning issue. Imagine, imagine for a moment that your application is down and what would be the typical troubleshooting process today? First, we would probably access the nodes, use a utility like TCP dump, and we might find out that there are drop packets. So now we need to find out why when we proposed the admin network policy, we thought let's try to prevent these headaches and we wanted this tooling from the start. One example is a stacked policy. Like Surya was saying, we have priority-based policies, so what happens when the priority is the same? There are also some new features, so let's just prevent headaches with all this stuff. So today, we are introducing policy assistant. This right now is a command line interface. You can do things like summarize all of your policies. It has an analysis engine, so you can simulate what policies would do without actually needing a cluster. I'll get into that. And finally, there's some conformance testing capability, which again, I'll talk about at the end. So this is an extension of a pre-existing project called Cyclonus for network policy v1, and it's written in Go and it's included in our repo, so please do check it out. I'll have links at the end. But yeah, why would we even want a policy assistant? I think of it as two things. One, we want to prevent issues and we want to troubleshoot issues. So preventing issues, you can break that down into a pre-validation. So you have a policy and you're about to apply it to your prod cluster. Let's hope that it doesn't break anything. But the aim with this is you can actually validate that it won't break anything using policy assistant to do the heavy lifting, so you can say like, how would this rule affect pre-existing traffic? I think of this kind of synonymous with experimentation. You can learn about the API, ask it to summarize things for you. Just learning about the API. And the other leg of it is troubleshooting issues, like with the application that was broken earlier. Are my policies causing this drop traffic? And which policy is doing it now that we have three APIs? So just a final recap before we get into things. We have the admin network policy coming first and that has priorities that will come into play. And then we have the old network policy v1. And then finally, you have the baseline admin network policy. So please join me in praying to the demo gods because we will be doing a live demo. Before we get into it, I actually do want to just describe the topology here. So we have a demo namespace with just two pods, very simple setup. We have pod A and B. B is connecting to A on port 80 and 81. And similarly, A is sending traffic to B on port 80 and 81. And like I said earlier, you don't need a cluster. I will not be using a cluster in my demo. So these are just theoretical connections. There are no pods. Everything is being simulated. Okay, let's do it. All my policies are going to be in the demo namespace. To start, we're going to have a rule that is ingress for incoming traffic and it's talking about traffic on port 81. And it's going to be talking about all pods in all namespaces. And just to be explicit, this is a deny rule. So without further ado, let's look at this CLI. We have this command right here. Since I'm not using a cluster, I'm going to be using this thing called a policy, or I'm just going to feed the policies, the directory that they're in. And now we have this thing called a simulated connectivity table. So like I was saying, we're talking about pods in the demo namespace, both A and B. On the left, we have source pods. And on the top, we have destination pods. And so again, port 80 and 81 are in play. This dot is saying that this admin network policy would actually allow traffic from B to A on port 81. Or sorry, port 80. But then the X on the right is saying that B to A is denied on port 81. So we have a similar thing here. For A to B, it's allowed on port 80, denied on port 81. So now I'm quickly going to add a couple more admin network policies. The first one's a carbon copy of the previous, except it's doing this pass rule. I like how he was saying pass means stop evaluating admin network policy, hand it down to the network policy V1. And this is actually a higher priority than the previous one. And we also have an admin network policy talking about port 80 this time. And it's allowing traffic. So if you run the same commands, now we see that traffic from B to A is allowed on port 80. So there was that difference where, sorry, I mean port 81. It was denied before, but now all ports are allowed. So let's try to visualize what these policies are. We have this summary table that will group all policies by their subject. So here we're saying everything in the demo namespace. So we have all three of our admin network policies right now, targeting all pods, all namespaces, and then we group it by port. So we have that allow rule for port 80 and for port 81 we see the pass with a higher priority. Okay, finally, I'm going to add a network policy V1 what we know and love or don't love. And so this is targeting pod A. Again, demo namespace and it's saying don't allow any incoming traffic. So we can see one change. Traffic from B to A on port 81 is now blocked because of that deny all network policy. I can go back into Y in a second and if things start to get confusing, don't worry, we will be walking through things at the end. So finally we have a baseline admin network policy and this is going to be doing it's going to be talking about the demo namespace again, talking about all pods from all namespaces and saying deny the ingress. Just whatever comes past the network policy we should be denying it. So we can see that there's been one more change. A to B on port 81 is blocked. I'm going to show one more thing here. Let's figure out what's going on. This summary table also includes our network policy V1 and our baseline admin network policy. And so for instance with NPV1 we can see it's targeting pod A. We can see it allows any peers but it doesn't have any peers that are allowed hence the deny all there. And finally we have the baseline admin network policy denying all ports and all protocols. So let me bring up the table once more. So I'm going to walk through all these things. I'm going to do it the hard way first and then I'm going to show you one last cool feature that shows I guess an easier way to do things. So let's walk through our four connections. We have B to A on port 80 that is allowed. Why is that allowed? Because we have this admin network policy allowing on port 80. Similar thing for A to B that's allowed because it's on port 80. Then port 81 traffic is passed down. So now we're looking at this one. B to A that's going to be denied because there are no allowed peers for this network policy targeting pod A. Finally for A to B on port 81 the baseline admin network policy kicks in. So the more policies you have the more confusing this could get the harder it is to walk through. So let me show you this final thing. We have this mode called walk through so it can do all the heavy lifting for you. So we have all our four connections again. A to B on port 80 that's allowed by the admin network policy same with B to A on port 80. And then why is the traffic on port 81 getting denied? We can see that for A to B that's denied by the baseline admin network policy. But for B to A it comes down to the network policy V1. So we're really excited about this feature. Hopefully it can help you debug issues in your cluster or prevent them with that pre-validation. Just going to summarize real quick so we have the simulated capabilities looking at allowed and denied traffic. We can summarize all our policies grouped together and we can walk through all of our policies and why and which policy is causing an allow or drop. And just wanted to give a shout out to Future Roadmap. So this is just the beginning. We just have three modes in a command line interface. We welcome all ideas when it comes to new features, new designs. So please check out the parent issue. These are just a couple of things we have in mind right now. One being conformance testing which with the cyclinus framework we can add another conformance badge that just makes sure we don't have to see an eyes cover all of the edge cases when it comes to these new APIs. So you don't have to worry about that. That's just something on our roadmap. And I do want to give a shout out to Nikola who wrote that summary table you saw today. That was a big help. And big shout out to Matt Fenwick who wrote the whole cyclinus framework basically that we built off of. And so here's the parent issue if you want to check it out. And also I encourage you to check out the demo code. Mess around with it. Try different admin network policies. Just learn about the new APIs. Yeah, just have fun with it. So I'll pass it off. Yeah, that was the cool demo. We're at the end of the presentation. Just want to say if you want to get involved in the project please reach out to us. We are here at QubeCon and also on Slack. We live on Slack, the state network policy API Slack on Kubernetes. So please check it out. Check out our website and code base. We need help with lots and lots of issues. The conformance profiles, testing, all the endpeps if you have new features. Add them. The quality assistance needs help. So we're always in shortage of developers and contributors and even non-code. Just giving us feedback on how these APIs are working. So thank you everyone for coming to our talk. And if you have any questions, Hunter and I will be here.