 All right, let's get started. My name is Ahmed Balkan. I work at Google Cloud as a software engineer. And I'm here to talk about Kubernetes network policies. So one of the things that keeps me interested about this feature is it's simple. So I hope many of the beginners here, if there are any, can also understand this without knowing too much about Kubernetes. But I hope you know some amount of Kubernetes. After the talk, you can come pick up some Google Cloud stickers. We actually have a pretty cute logo I want to show it, much better than the one over there. And we have some GK stickers if you're interested in that. So as I said, I work at Google on the Google Kubernetes Engine team. And we're going to be talking about network policies today. And mainly my agenda is, what are the network policies? How do you write them? How do they work? And hopefully the four items on you go home and actually start using them. So network policies control traffic coming out of the pods and coming to the pods. So this is a construct that manages traffic at the pod layer. Like you basically say that I want to control what comes to this pod and what goes out of this pod. Network policies let you do that. Yeah, so who's using network policies already? Really curious. So for the record, there are less than 10 people out of a full room. That's amazing. All right, I hope you're going to learn something today. So without the Kubernetes network policies, your clusters are like, actually, you don't know they're on fire. But I tried to do the, this is fine dog meme. But it didn't work because it's copyrighted. Anyway, your clusters are actually not fine, just so you know. And so the network policy API is started around, the workaround policy has started around late 2015 from mainly the folks interested in the networking space and bringing closer network integration to Kubernetes. And as you can see, it took quite a long time for the API to actually evolve. We have some couple of seats up here, if you want. So the API went through some stabilization period. And that period wasn't necessary because if you actually look at the initial proposal of the API, it's completely different than what we have today. I actually want to say thanks to the people who contributed the feature a lot. Namely, Casey Devonport from Tigera and Christopher Luciano from IBM and Dan Winship from Red Hat. These people made the feature happen. By the way, I did not work on the feature. I actually like the feature so much. That's why I'm here to tell you about it. But yeah, these people actually did all the hard work. So I guess the thanks go to them. So the feature has been stable since 1.7. It means that this API is in your Korean's clusters if you're using 1.7 or above. So let's talk about writing network policies. Like, how do I write one, right? So first, I need to talk to you about labels. If you don't know about labels, every pod in Kubernetes better have labels. It's not mandatory, but this is how we do things in Kubernetes, and this is how pretty much everything in Kubernetes works. So labels basically associate some metadata with the pods. And then we have label selectors, which can select a set of pods using these attributes. And in this case, on the right, you can see I have a label selector that says match labels equals shopping. And in this case, I selected these two pods. So in this case, I said, choose the database tier. And I selected those two pods on the right. Pretty simple, right? So the next thing is what the empty selector is a thing. And I want to just ask this question. What do you think the empty label selector is going to select? Who says it's going to select none of the pods? Two people. OK. Who selects? It selects everything. Perfect. This is amazing. That was not necessary, I guess. So let's talk about what's in the network policy. Like, how do I write a network policy? So first, you've got to specify which pods does this thing apply to. Like, what is it controlled, right? And we do that by using label selectors, or pod selectors, which can select using labels. So the second thing is, where does this policy apply to? Is this for which direction? Is this controlling the incoming traffic, or is this controlling the outgoing traffic? So in this case, we call the outgoing traffic egress, and the other way is ingress. And what are the rules? Like, for, let's say, an ingress rule, allowing traffic to the pod, who can connect to this pod? So we've got to specify some sources that are white listed to connect to this pod. So this is roughly how the object looks like. It's in the networking.kates.iov1 API version. It's called network policy. It's got all the metadata as usual. It's got the spec. And in the spec, we definitely need to have a pod selector. And we definitely need to specify a direction. You can either specify ingress, or egress, or both. And those can have multiple rules. So the first rule of network policy is, the traffic is allowed by default unless there is a policy in place controlling that traffic. So if you are selecting the pod from a network policy, that is actually restricted in the traffic. So if you don't do anything, all the traffic is allowed. Right now, if you use Kubernetes, and if you don't use network policies, which is pretty much all of you, network policies are not restricting anything. So the rule two is, traffic is denied if there are pods selecting that traffic, those pods. Sorry, if there are network policies selecting that traffic, but none of those policies are actually allowing that particular source or destination. So I'll come back to this in a moment. But I think this is important to understand. From rule one and two, I can say that you can only write rules that allow traffic. If you watched the keynote today, they were showing a, was it the keynote I'm trying to remember? I guess it was the keynote. They showed a network policy which was specific to Tiger's implementation. That's not the Kubernetes Core Network Policy. But in their APIs, they had a deny rule. You cannot have deny rules. Like you can't say, I want to deny this explicitly in Kubernetes network policies. So what you do is you only allow things. So as an example, so this is denying everything. This is a policy object. It will deny all the incoming traffic to all your pods. And the way it does is, first you see a pod selector. It selects all the pods. It's an empty selector. And then it specifies ingress, means that this policy is enforcing the incoming traffic. And then it's not specifying any rules. It has an empty array in YAML. And this means nothing is whitelisted. So once you put this policy in place, all the incoming traffic to all your pods from outside the cluster, from within the cluster, everything is blocked. So this is how it works. So the rule three is traffic is allowed if there's at least one policy allowing it. So let's say you deploy a bunch of policies. You deny a bunch of traffic here and there. But if you had one policy that allows that traffic, it will go through. So let's give more concrete example. In this case, in the middle, I have a database application. On the left, I have a back end, which is you can't talk to the database just fine. But the front end probably should not be directly talking to the database. It should probably do something else. So how do I write a traffic network policy that enforces these traffic rules? The way to do that is I have a pod selector that matches to the database first. It's enforcing the rule on the database pods. And then in the ingress section, I have a single rule. Like if you just look below the ingress, you're going to see that there's only a dash indicating this is a rule. And in that rule, I see a from statement. It means allow the traffic from these pods, which is, in this case, it's selecting the back end pods. And it's allowing the traffic only from those pods. So this is, if you deploy this, you're going to get basically a system where back end just can't talk to the database, but not to the front end cannot do that. And in this case, no other pods cannot talk to the database either. So you're basically blacklisting everything except the back end. Let's take this example further. And let's say the database is exposing two ports. One is port 80. The other one is like MySQL port 3306. So you don't want the port 80 to be accessible. So you just take the same rule and say, I want to add these ports explicitly. If you say ports 3308 TCP, the back end will be only able to use that port. If you don't specify any ports, all ports are open by default. So this is like further adding that little bit more security in terms of whatever minimal access is needed. In this case, the back end only needs to access the port 3308. Then you just give access only to that port. A little bit, a small note about the port numbers. If you can't currently use the service ports, you have to specify the actual port numbers. I don't know why this is the case. Currently, all the network policy implementations are just going this way, except other than like the SPIG is not saying that. But the SPIG is not blocking that either. So we'll come back to how the network policy is enforced. So I'll have a chance to talk about it later. So we said that all the policies are, if there's one policy allowing the traffic, the traffic will definitely go through it, right? So let's talk about how the policies are evaluated. So if you have a bunch of, let's say, ingress rules in your network policy, basically, if any of those match the traffic, then the traffic will be allowed. So it's not like it should match this rule one, and then it should match this rule two. It's not an end statement. It's an or statement. So to give, I guess, a little bit more concrete example, in this case, I have a single ingress rule, which has two pod selectors. So if any of the pods are matching one of these pod selectors, then they will be able to get in. So these are combined with an or statement, not an end statement. Similarly, I have two rules here, right? I have two from rules. And this has the same effect as the previous one, by the way, just because everything is or. In this case, the rule one is or with the rule two. So at the end, you're getting the same effect. So let's look a little bit about empty selectors, because you can do cool things with them. So I want to say, allow all the monitoring pods to connect port 5,000 of all the pods. So I come up with a very simple network policy spec like this. So this spec says, select all the pods, and then allow traffic to port 5,000 of the role equals monitoring label. So does anyone see any problem with this at all? Yes, please go ahead. Yes, anyone can, considering with the role equals monitoring, that's true. But that's not what I was trying to get at. So this works only if the monitoring pods and the pods you're selecting are in the current namespace. If your monitoring pod is somewhere else, this is not going to work. So the reason is that the pod selector is selecting only pods from the current namespace. And the network policy object itself is a namespace sculpt object. If you have a network policy, you have to deploy that to every single namespace. You have to enforce the same kind of thing. To illustrate this better, let's say I have the same setup in both namespaces above and below. The one below does not have any network policy enforcement because you did not deploy your policy there. So you have to deploy your policies to every single namespace you have. So then in this case, how do we allow traffic from other namespaces? Because the pod selector is matching only the local namespace it is deployed to. So we need a new kind of selector in this case, which is the namespace selector. It's like the pod selector. It still uses labels, but it selects namespaces. By the way, who is actually labeling their namespaces? I'm really curious. Some people, very few. OK, so the gentleman back there said, we only label our namespaces because never policy needs them. That's cool. So yeah, they came up with this thing. So if you want to use them with namespaces, you totally got to label your namespaces. I guess let's give an example. In this case, I have two namespaces, let's say foo prod and foo storage. And I want to give access to my apps in prod storage. So I basically specify that. I want this namespace selector. I specify label purpose equals prod. And then I basically allow traffic from all the pods in that namespace. So there is a problem with that. If you use namespace selector, it does not let you specify traffic selectively from another namespace. You either allow all traffic from that particular namespace or those namespaces, or you don't allow any traffic from those namespaces. So you can't do what I just illustrated in the picture. You can't enforce traffic to selectively different pods because the rules are applied with OR. Once you allow traffic to all of the namespace, you can't reduce it back. So so far we learned about two ways to specify where the traffic can come from. There is a third way, which is called IP block. So if you don't know what a CDIR is, it's a way to specify IP ranges. So I put a little table on the left. It shows you if you do 10.0008, that basically means all 10.something IP addresses. So that's what CDIR is. In the example on the right, you can see that I have a front rule, and in this case, I specify a CIDR range. And I basically allow traffic only from there. And in the example below, I say, OK, allow the traffic from that IP block, but except this particular smaller block. So that's a feature. Let's talk about the egress rules. So egress rules are pretty much the same thing as the ingress rules that allow traffic to your pods. And egress rules enforce the traffic to whatever is going from the pod to outside. So you can send traffic to some places, but not everywhere. This is actually pretty helpful. And it's as helpful as the ingress rules. These are recently added in Kubernetes 1.7, if I'm not mistaken. So here's the same example for the one we have with ingress. How do I deny all the egress traffic? So in this case, we're looking at selecting all the pods again. Different than the ingress one, I specify a policy types. And one of the elements in there is called egress. You have to explicitly specify this unless I'm mistaken. It's just the thing if you do with the egress. You can also explicitly specify the ingress, but I think it's inferred because of the API compatibility reasons. So in the egress section, I specify, again, an empty set of rules. This means I'm not allowing any traffic to go out. So that's what it is. So this is a more concrete example. You don't want the front end instance to call anywhere except the back end. So I mesh to the front end. And then in the egress section, I say, go to the only tier equals back end pods. So this is enforcing that policy, pretty simple. So if I actually go into the front end pod, if I do like QCTL exec, and I do like cure HTTP dash dash back end, it says, cannot resolve host back end. Didn't I just allow access to back end ways of giving you that error? Does anyone have any ideas? DNS, yes. So this thing, turns out, actually blocks QDNS too, or DNS resolution in general. So you have to either allow all the DNS traffic, or you have to figure out the IP address of your QDNS service and whitelist that particular IP address. In this simple example, I basically just added ports here. It's a new rule. It's like the rule one is the one that gives access to the back end. The rule two is giving access to ports. In this case, I don't specify a destination. So all access on port 53 can go out, can go to internal IPs, can go to external IPs. But in this case, I'm giving access to both UDPN, TCP of port 53, because this is the DNS traffic that we want. So more fun with egress, you can actually block the external traffic and limit your cluster traffic to internal stuff. Like let's say you have an app. You definitely don't want that app to establish connections to outside your cluster. Like you don't want it to go download packages. You don't want it to go, I don't know, send your passwords to somewhere else or something. So this is how you do it. In this case, I'm, again, much matching to all pods. But in the two section, I'm selecting namespaces, but I have an empty selector. So this is selecting all the namespaces and all the pods in them. And it's not selecting anything else. So this is only restricting traffic only to other pods in your cluster. So that's useful. Let's talk about how the narrative policies are actually enforced. So I have a surprise. If you don't have a networking policy installed in your cluster, you can go home, deploy these rules, and nothing is going to happen. They're silently ignored. Like that was my little surprise. I thought it was just kind of magically gory. So it actually does not work out of the box. You have to install something like Calico, VNet, or Romana. I don't personally endorse anything. But I'll come to that later. You need a network plugin to enforce your network policies. So in Google Kubernetes Engine, we use Calico. And it's pretty simple to create a cluster with network policy enforcement. We basically just say gcloud beta container clusters create, dash, dash, enable network policy. So this is installing the Calico networking plugin to your cluster. This is something that Calico and Google manages together. Like if there's any problems with your network policy pods, we know about it. The on-call comes and fixes it. So the Google SREs are on this. This is like you don't have to install and manage this yourself. We can install and manage that for you. And if you go to cloud.google.com slash Kubernetes Engine, you can find more documentation about how this works. So more on network policy enforcement. So you've got to know the connections are duplex, meaning they're two-way. Once A can connect to B, obviously B can send data back to A on the same connection. That's how HTTP and everything works. But that does not mean B can connect to A. So don't think of it like once you have a policy restricting the outbound traffic, you cannot send any packages ever. That's not the case at all. This is pretty much how you would expect I think it works. So the second thing is network policy is a connection level filter. It's not applied to packages. It applies to connections that are established. So after you deploy a network policy that would block some connections, if you already have some connections in flight that are established already, network policy is not going to kill those. This is not in the specification. Some providers out there might be killing that. But the spec does not say anything about it. So don't expect them to be killed. But any new connections established after the network policy takes effect, those are going to be blocked. And they even look like they're timing out most of the time. I guess that also depends on the implementation. So I want to talk a little bit about what are the performance implications of using network policies. Because when I was using that, I'm like, this definitely changes the performance. Because I was thinking they definitely create an overlay network, and they add a bunch of rules in there. And they re-evaluate the rules any time there's a new pod comes up, or there's a new network policy comes up. So I thought it probably adds a lot of latency. So good thing is that the Romano folks actually published a blog post about it. And the blog post basically says that we deployed so many thousands of rules, and it only affects only a little. And that's like during the first time to bite or something. So it's not a big deal. And I guess this also depends on the implementation. Some providers use IP tables, some just establish an overlay network on your network. So also depends on how they actually implement the algorithms, like how they do caching, and how they react to the pods coming up, and so on. Because basically any time a new pod comes up, you have to update your entire cluster with the new set of IP tables rules. So there's a little bit of work there. So let's talk a little bit about best practices. I showed the default deny rules for ingress and egress. I think those are really important. The reason they're important is they, I think, establish a baseline, because you're blocking everything, and then you're whitelisting one by one, based on the need. If you're developing a new microservice, and you know where it's definitely going to talk to, you're establishing that network policy that's allowing that traffic one by one. So you're basically starting from zero and building up, as opposed to having everything able to access each other. That's not great. So I think this is one of the best practices. The second thing is you have to understand how the rules are actually evaluated. It's always good to be explicit about what you mean. One of the problems is that right now I'm up here. So I'm giving this talk, but I cannot confidently say the difference between those four things. I can tell some of them, but some of them are blocking all traffic, some of them are allowing all traffic. And it depends on where the empty object is specified. It can allow or deny all traffic. So you should be careful about what you're doing, which brings me to my next topic. You should test your policies. Once you deploy a policy, make sure it's actually doing what you expect it to do, and test it with different conditions, like test the external networking. Test it, I don't know, can it access DNS? Can it go outside the cluster? Can it access another namespace? Can other namespaces access this? So if you try all these things, you'll have a better understanding of how the feature works as well. Also, I think if I'm not mistaken, Kubernetes 1.9 is going to ship with a better Kube CTL described printer, which is going to explain the network policies much better than today. So that should actually help. So I've been working on some network policy tutorial repository. It says tutorial, but it's not really a tutorial. It's like a recipes repository. So if I figure out a cool thing to do with network policies, I just put it up there. And you can contribute to it as well. I have some drawing, some diagrams explaining how everything works, and the policies you can directly download from there. So there is that. Network policy is not the end game. There are already several network policy implementations in different parts of the stack. Istio claims to be doing some part of it. Some network policy providers are claiming to be doing what Istio does. So everybody is attacking one layer above and below. I think that is fine. But in this case, your applications have larger concerns, such as, how do you do authorization? How do you do pod identity? How do you make sure you're enforcing the right set of policies? I think these can be used editably. This is more defense in depth. Like you can use Istio, but you can also add the network policy on top of it. It would just work fine. So this is not the end game. This is not going to give you the silver bullet. This is pretty much my talk. If you're actually interested in the network policy recipes repo, definitely check out that repository. You can find my slides also on the talk page. So if you want to give a try to the Kubernetes engine, definitely go try it out. I think we have a pretty good free trial. And if you want some stickers, they're here. If you have some questions, I think we got actually some time. Wow, I'm impressed. Yeah, please go to the microphone. Thank you. I have to repeat it if you don't say it to the microphone. So I have a question about egress policies, actually. Could you repeat that I couldn't hear? About egress policies. So if a pod accesses a service IP address, basically the question is what is supposed to happen, especially if the egress policies allow a subset of the pods that could be accessed by that service IP? If you're trying to access another service, I would just copy the labels of that service and add it to the egress group. So just from a spec perspective, if you're doing an implementation, hypothetically, of network policy, what is actually supposed to happen? That's a good question. I'm not sure, actually. I mean, would you whitelist the pod IPs? Would you whitelist the service IPs? I mean, it's actually pretty awkward anyway because at the level where you're applying the network policy, you're seeing just the service IP. So it's not really clear if you're supposed to be pulling information about the service endpoints into the network policy and applying it at that level, or it's problematic in a way that egress policy is not. You do need that because at the end, it's just into dnet. So if you don't have the actual endpoint IP, you only have the cluster IP, it's just not going to work. Yeah, it definitely won't work. But the question is, what is it supposed to do? For example, you can imagine I intersect the labels on the service with the labels of the network policy and apply only to the subset of the network policies, and then it would have to be integrated with the load balancer so that the load balancer would only load balance to the right set of pods. I'm not sure why it wouldn't work with the cluster IP directly. Because the cluster IP is not an IP address of a pod that is selected by one of the network policies. Because the network policy applies only to pods. It does not apply to services. That's true. Yeah. Honestly, I don't know how it's enforced on the services. Yeah, I would say sticking to labels will probably solve it. Oh, the labels are what's causing the problem. It's because I guess that I'll have to maybe bring it up in SIG networkers. Probably, yeah. SIG network is definitely a good place to ask these questions. Please go ahead. So I think if I heard it properly, when you install Kubernetes, it doesn't have a network policy implementer by default, right? It does not have a default implementation now. So what's the reason for that? That's a good question. I think the reason is because everybody has a different way of doing networking. And pretty much every platform you're using is using a different networking model. And there is no single implementation to rule them all. There are ways, like the way you do bare metal versus cloud is wildly different. So I'm guessing that's why. Also, the current implementations are offered by commercial companies. So I think that's one of the reasons why there is no default. Yeah, but that doesn't quite resonate if it's a part of the spec, and then you write it in your YAML file. I find it interesting, too. I agree. It's a core object. And there should be probably a preference implementation that works on, say, Minicube, right? Yeah. I agree with you. Yeah, thank you. Any other questions? Yes, I will repeat your question. Yeah, so the question was, what if I have a metrics system like Prometheus and deploy it in a restricted namespace, and I have a multi-tenant cluster, and everybody's got their own namespace, and the customer wants to access their own metrics, I will deploy, I don't know, my own custom solution, or multiple Prometheus instances. I don't know. Probably never a policy will help you. Yes, that question. What is that? So the question was, did we have any discussions about cluster level network policy? What does that mean? Like, applies to every pod or something? Yeah, so that is not part of the core API. I think the private providers like Calico, they're enforcing that in their separate implementations and APIs. So I think that's possible with their APIs, but I don't think there's anything going on with the Kubernetes core API right now. Yep, good. OK, so the slides are actually on the event page. You can go to the event page, click the event, and download all the slides. I'll be also tweeting about them. Yes, question there. What tool do you think you can use to paint this? I use iPad, just with a pencil. All right, thank you all for coming. Appreciate it.