 Okay, good morning, and thanks for joining us today for this session on network security at scale I Am Bernard van der Waal. I'm a principal software engineer at Splunk And I work on operationalizing at scale Kubernetes, Istio and Envoy. I Used to do this at cruise previously and my whole career was in between networking and software engineering and And my name is Mitch Connors. I'm a principal software engineer at avatrix where I am working on Istio project I've been a member of the Istio project since 2018 currently serving on the technical oversight committee I'm also a local here to Seattle, which means that I can give you Istio swag I can answer any questions you have about the project about avatrix where I work or I can give you really good restaurant recommendations for the area so come chat with me after Very nice And so today we want to talk to you about Splunk cloud and the story behind it So Splunk cloud is basically a redesign of Splunk to be cloud native So we wanted to standardize it on the Kubernetes API We wanted to be cloud agnostic as much as possible and today we run both in AWS and GCP mainly in AWS We run in about 35 clusters all over the world in all regions of AWS pretty much and The tech stack we are going to talk about today is Kubernetes, Istio and Envoy mainly and so if we look at the agenda I think our goal today is to tell you a generic story and go in detail into each of the layers That you deal with when you manage the network of such a cloud deployment Nothing, it's especially confusing because to some level you can configure networking policies on the cloud provider on your VPC You can also configure network policies on Kubernetes and you can do that on Istio as well Right, so I think what we want to do today is to go into each of those layers and explore what you can do Our goal today is to make it as generic as possible. So it's not really tied to Splunk even though it's what we use Okay, so let's start with the cloud provider level This is for AWS and GCP So we deploy your clusters into standard VPCs. So we use a cookie cutter VPC One cluster per VPC typically and the goal is it gives us some level of networking sandbox for clusters We deploy your VPCs with three different set of subnets So we got typically a bunch of private subnets which will be used for the Kubernetes workloads That's where we want to have our nodes running and at the end of the day the pods as well We do this because we want the private subnet as the name says to be fully isolated from anything else And then we got public subnets for internet connectivity And that's where we want anything ingress or egress coming from internet to go through and we got internal subnets for connectivity inside Splunk or to other clusters and This brings me to the first knob that you can use for security which is network ACLs So pretty much every cloud provider supports this Those are very low level crude ACLs. They are stateless. So they don't really take into account already open connections The issue is they get applied per subnet, which means you pretty much need to Allow everything that any of your instances will eventually require because you can only apply on the subnet level You cannot specify specific instances They gave you very basic layer 2 and layer 4 capabilities Meaning you can match on IP and ports both on the source and destination and I pretty much see them as a kind of catch-all last resort set of rules The way we use them for example is on the internal subnet. You never want to see anything else than the IP IP private range like 10 slash 8 the RFC 1918 IP range so we default we by default we deny everything Except traffic with those source IPs We do the same thing for the private subnets and we also explicitly allow some traffic coming from internet from specific ports for anything ingress like 443 Okay, so on the private subnets we will deploy node groups Those are the Kubernetes nodes typically and we will group them into different functions into those node groups For example, we got ingress node groups. That's where we want to deploy or ingress gateways Where we will learn and void eventually we'll talk more about this later. We also got some generic node groups and more cloud specific node groups All of those get deployed into a private IP space because again, we never want anything Outside which which you know directly to those node groups And so doing this gives you the ability to use something else that most cloud provider support Which is security groups? So security groups are kind of network ACLs plus plus first of all, they are stateful. So they support connection based You can also apply them for instance, which means you can do more fine-grained things like this We want for example to fully isolate the generic node groups and only allow connection in Through the ingress node group. And so what we do is we apply a very locked down security group to the generic node groups and a slightly relaxed Security group to the ingress node groups. And so this basically forces anything coming into the cluster to transit through those ingress node groups The other good thing we said security groups is They allow you to use a slightly higher level syntax. You can use other security groups as source or destinations For ingress connectivity, we want everything to go to our NLB and ingress gateways and ingress node groups So what we do is we deploy basically a bunch of NLBs into those public and internal subnets The public NLBs are obviously made for public internet connectivity and we use Internal internal NLBs for anything that needs to be reached from inside Splunk Those will be IPed on the 10 slash 8 IP space obviously And what they do is they basically target the ingress node group and the ingress node group will run the Envoy gateways that will terminate connections and proxy traffic upstream inside the cluster So for internal traffic once you've hit the NLB Inside the internal subnet you still need a way to get to another cluster other cluster lives in another VPC so for that they're using AWS transit gateways those are advertising only the internal subnet source IPs So those private subnet public subnet IPs are not getting advertised across that transit gateway And of course this transits through the Splunk firewall where security rules can be applied In addition, there's a number of shared services that every cluster needs to be able to access And they need to be able to access it without transiting an NLB. So for for these cases Splunk is using avatrix avatrix Allows a flat network at layer 3 where all of the clusters directly from their private subnets can access those shared services across the avatrix transit gateway and Of course once you've established Peering so that your pods can access those shared services Aviatrix by default would allow those pods to communicate with one another Which of course would bypass all of the rules that we've set up in our internal subnet So in order to prevent that Splunk has used avatrix's Network domains feature which simply allows you to say this VPC can talk to that VPC that VPC can talk to the other But certain VPCs can't so in this case you see a diagram where like the blue Cannot talk to the green and the green cannot talk to the blue But we have a number of shared VPCs that are both blue and green that are accessible from all and that's the setup That Splunk is using there Okay So let's look at the next layer, which is Kubernetes So first of all our Kubernetes deployment that Splunk we treated as a full self service platform Which means that teams own a namespace again They can do whatever they want with the Kubernetes API. They can provision objects CRDs for STO, etc Our goal as a platform team is to provide some level of base capabilities like networking security observability, etc One of the things we try to do obviously is for the service teams not shoot themselves into the foots I'm going to show you a couple of examples on how we do that first of all one of the things we mandate is Any ingress connectivity both internal or external needs to go to our NLBs Ingress nodes and ingress gateway. That's a way for us to apply security into a central point Another note as well is once you get to the Kubernetes layer you do pot-to-pot communication as well This is mainly invisible to the cloud provider, right? Most of the time use tunneling services like Calico or Cilium and Yeah, that's add some level of complexity So one thing we do we use quite heavily some validating webhooks So the goal of a validating webhook is it's basically a callback from Kubernetes API Whenever a code a create update or delete of a specific object is being performed And so we use this in order to validate that users use those objects correctly a typical example is For a service of type load balancer you by default Kubernetes will Create an NLB if you use a service of type load balancer We absolutely want to block this right since again We want to force traffic to go to our NLB and ingress nodes And so we use those validating webhooks. That's just one example in order to enforce those things And there are plenty of open implementations of those validating webhooks OPA is one example I used to use K-Wail at cruise which is a good open source alternative in this case We we create our own ones, but they're pretty easy to implement Okay, so next thing is network policies on Kubernetes So those network policies on Kubernetes are very Kubernetes centric. You it's basically a CRD defined inside Kubernetes They allow you to use the Kubernetes syntax, which means you can use labels in order to select pods So that you don't have to explicitly Define the IPs of each pod you want to match on Those are by default implemented by your CNI plugin something like Celium or Calico and a typical example is something like this right in which You have a 2-tier application and you want the back end to be able to be reached from inside It's our namespace and you maybe want as well connectivity from the front end Right and so you you are able to select namespaces and pods based on labels and you can end up with something like this All right, so far we've talked about from VPC level at layer 3 up to Kubernetes policy How security is applied at each of these layers to scale up to the level that Splunk requires now? We're going to look at layer 7 and for that is Splunk is using the Istio project if you're unfamiliar with Istio It's a service mesh and lots of people have attempted to explain what a service mesh is it can all seem very confusing and hype At a fundamental layer this allows application layer Application layer networking abstractions over your network So if you want to talk about your network not in terms of this IP import can talk to that IP import But instead in terms of this service identity can talk to that service identity you want a service mesh If you want something that can globally collect telemetry and say something along the lines of HTTP requests with this path and this verb are succeeding this percent of the time you want Application layer networking abstractions you want a service mesh You can also play with routing a little bit and say perhaps our premium tier users are routed to a subset of this service Which we know to run on a little bit better hardware and be a little bit faster That's what Istio does We're going to talk a little bit about how Istio does this and then we're going to look at how Splunk is leveraging Istio Well, all of the magic of Istio comes from Envoy We insert an Envoy sidecar into every single application pod Which means that this diagram isn't quite correct if we were to correct it it would probably look a little bit more like this There's a lot of application pods in your average service mesh which means there's a lot of Envoy sidecars We're running Envoy everywhere, but the advantage is that means that we have control over all of the network traffic We can collect telemetry over all of the networking traffic and so the trade-offs in Splunk's case have been worthwhile One of the key things that Istio offers for Splunk is in transit encryption without any need to modify applications So the applications are sending plain text HTTP as far as they know The Envoy sidecar will intercept and apply mutual TLS to that traffic so that it may not be intercepted or modified in transit You can run Istio in strict mode which requires that MTLS apply to all traffic Occasionally you'll run into some hiccups there so Splunk runs in permissive mode Which means it will attempt to use MTLS and fail back to plain text if necessary And then they have this fancy alert setup that lets them know when there's traffic that's happening in plain text So that they can alert their security team rather than blocking it proactively They'd rather have developers respond to that plain text and correct the problem Yeah So I think one interesting thing to talk about is the way we provision gateways In our Ingress story because this is going to be the the Ingress point for anything into the cluster, right? So as we said before we want anything that comes in to go through NLBs because that allows us to NLBs and Ingress gateways because then we can apply a new uniform security policy to it What we do is we deploy basically NLBs and gateways together So the gateways will be Envoy running on Ingress nodes Based on the source of the traffic you want to ingest for example if you want to public Traffic you obviously will deploy your NLB on the public subnet for the internal traffic your NLB needs to be on the internal subnet What we also do obviously for security reasons and blast radius reasons is sometimes we deploy NLB and gateways for very critical applications We also shard tenants sometimes across multiple NLBs when we reach a critical mass And we do this so that in case gateways go down hopefully it doesn't happen too often It doesn't take down the whole cluster Those gateways are managed by us as a platform team We run them into the Istio gateway namespace and it's basically a capability that we offer to everyone into the cluster They can basically plug into it and the way we do this is what we use Istio We set up gateways so Istio is going to program Envoy as a gateway and we set up Those gateways through the Istio CRDs like gateways CRD So gateways CRD will basically define which hostname and which port you expect traffic on We also provision those gateways CRDs with certificates so that we enforce TLS by default right and then we allow our users into the Into the cluster to plug into those gateways Again, this is gated to a validating webhook so that not any namespace can connect on any hostname They need to be pre-validated in some ways Again, we want everything to be self-service except the critical parts like the Those hostname and ports open to internet And so we quickly realized something which is that users had to to create and provision a lot of CRDs and objects on Kubernetes and This was leading us to see a lot of Mini-secret incidents I would say in which people will configure things wrongly So we've moved forward by creating a service abstraction layer, which is essentially a controller Which will attempt to configure everything for you all those complex objects The way we did it is we allowed teams to provision an open API spec and This controller is going to script that spec and create the virtual services destination rules gateway All the Kubernetes objects you require to get end-to-end networking connectivity It will also manage certificates So generate a certificate for your hostname if it's allowed and set up DNS that goes directly to the right NLB So this is pretty much for us a one-click deploy solution for the network end-to-end if you use those specs that we defined and we see it as a way to Make our clusters more secure since it would use configuration errors Okay, something else I would like to talk about is some Level of layer seven authentication So we wanted to provide in our cluster. So standard way to do Oten so authentication we do this by leveraging a Spec from envoy called X to Z. So X to Z is defined by the envoy project. So we do this outside this tier It's basically a way to get a sync call whenever you hit and void I'm where you reach out to next to the server with the headers of the request and Next to this hour will decide if it's allowed denied or in some case you can even modify the request headers So what we did is we developed a bunch of extra to the proxies that we that act as an extra to the server And those will allow a bunch of plugins to plug in and so we developed those plugins based on what our tenants need For example, we have an authentication plug-in which will validate in some case your bearer token for HTTP We got some JWT talk of plugins and we got things like got a plug-ins Which will make sure that a specific tenant doesn't issue more than a specific amount of RPS The good thing is this is enforced on the gateway before going into the cluster Our goal is to basically drop any bogus request because before they get upstream further into the cluster I mean, I highly recommend X to Z. It was a super easy protocol to implement The workloads upstream into the pod still need to re-verify the identity though This is just a way for us to stop large-scale D doses in some ways or to block anything that should not get into the cluster Other things we use is to forest. We also use some level of network policies some allow lists Once you reach the Istio level you can basically start mixing and matching layer 4 and layer 7 concepts And we do this quite often. It's not super clean, but it allows us to do things like Defining us a bunch of source IPs that should be able to go to specific paths and host names For a service for example So the higher level you go into the stack the more concepts you can use to provide those network policies other things that some of our tenants do on the cluster is they use some request authentication CRDs to Define some GWT Oats and validate that any request coming into their Workload is got a correct token associated into their headers And so yeah for this at this point I want to take a step back and look at the life of an ingress request So this is basically an end-to-end request coming to our gateways going to the envoy x.z and then being Proxy to the upstream workload So if we if we take a step back Any policies applied on the VPC and Kubernetes level are applied for connections Any policies on Istio are applied per request pretty much So your clients when they send us a request They first open a connection to our NLB the NLB will layer three proxies to one of our ingress nodes This will end up into our envoy gateway that will terminate TLS and then we'll inspect the request on a layer seven They will use x.to.z to validate that it's a correct request and it should get into the cluster If it's allowed the envoy gateway will proxy it upstream at that point The network policies from Kubernetes will kick in again to make sure that this proxy to the upstream port is allowed and then the upstream sidecar on the workload will again look at the layers of an request and if allowed it will be Decapsulated and sent to the workload in clear text When we do this we use a public certificate on the front end That's what your clients will see typically to let's encrypt or a bunch of other providers we use And we use fully MTLS in transit encryption inside our clusters. We use a splunk internal CA for that All right, so that was a lot of layers of security And as you can imagine not all of it is as wonderful as one might expect Let's talk about a couple of particular pain points in the security stack and what we're doing to resolve them That slide that we had earlier with like a million envoy instances on it It looked bad and that's on purpose Running one proxy per instance of your application you're gonna run into scalability problems For one compute cost that is very substantial for that many proxies For two in some cases you're gonna want some envoy instances to have more resources than others That's very difficult to pull off with Istio today and to get the tuning done right Obviously we need to capture all traffic But there may be a more efficient ways and we'll look at that in a little bit that we can capture all of the track traffic across on boys There's also a lot of magic involved in Istio today I've pulled this quote from the Istio documentation site and this last sentence here any new pods that are created in that namespace Will automatically have a sidecar added to them. It's magic It's really cool. What about the old pods? Okay, well, we don't do anything to those those pods are immutable we can't do anything to those pods So you've got to go through every time you change something about what's gonna get injected in a pod or a namespace You've got to go through and restart that workload don't miss a workload You might be rolling out a critical vulnerability fix that you're gonna rely on and if you forget to restart your workload Your Istio upgrade will have succeeded. Everything looks good and you're still 100% vulnerable on the wire So we we think we could do better also just answering the question which envoy am I running where? Can be challenging So the Istio project is aware of these challenges and we believe that we have a solution for them We are developing ambient mode for Istio where all of our layer 4 technology all of that MTLS magic that we talked about Will happen in a daemon set that runs on every node and serves every pod on the node So no more sidecar all of the traffic capture happens outside of the application pod That Z tunnel should do very little only layer 4 functionality Which means it should be much less susceptible to CVEs much less susceptible to upgrade breakages, etc Should be safer to run Then if you want layer 7 magic as well, you can still run an envoy But it won't run as a sidecar It'll run as its own deployment in the namespace the Z tunnel will take care of Redirecting traffic from whatever application needs it into the envoy sidecar where we can apply all of the l7 magic that you've come to know And love from Istio, but not one per instance. You can scale that envoy installation Independently of the application scale. You can adjust its resource utilization or allocation Our pods will match our deployment spec Finally, which doesn't really happen with injection today If you'd like more information on ambient or you're interested in getting involved you can check out the blog post below All right, so let's let's look in the rear view mirror. This was a lot of information to cover in a pretty short time We're doing good on time Every single layer of the network from the cloud provider and VPC level in layer 3 to the Kubernetes policy to Istio at layer 7 We have to solve for identity policy and observability Now This looks redundant, right? We've already done identity at layer 3. Why are we doing it again at layer 4 and then again at layer 7? That's computationally expensive The reason for that is that security is About layers every layer is always going to have holes and this is your slide. So I'll let you take it there Yeah, I mean, I think one of the key things is that we learned over time is defense in depth So what we want to do is there is redundant things you can cover in each layers The best way is to close them down each the most you can and to make sense so that you have that Swiss cheese model right in which If by mistake one secret incidents happen at one layer often the next layer is going to catch it The things we learned as well is that self-service platforms are really hard So the best thing you can do is to put safeguards in place to avoid your users doing stupid things most of the secret incidents we've seen is Basically users misconfiguring things. So over time what we try to do is to block those things one by one The one way to do this is to provide the golden path that will avoid those configurations That's what we did with a self-service Abstraction layer so that we can kind of ensure that people do the right thing by default Finally, I will close by saying that for us observability was key. That's how we detect secret incidents With a bunch of alerts to get paged if any public traffic is discovered in some VPCs for example It's also very useful for helping our users debug at the end of the day because there are so many layers and nobody understands all of them I think that's pretty much it. I think this goes to a feedback form if you want and we'll take some questions Also give a brief plug if you're an Istio user We're going to have our first-ever Istio day at cubecon Europe as a co-located event and the request call for papers Is currently open for another week and a half. So submit your talks Yeah, Rob Yeah, I mean a typical example is I it's not really a mini-secret incident But it's so if you still you can for example look at your past through cluster if you don't use a fully strict Encryption in transit and you will see anything that doesn't use MTLS go to that cluster, right? And so that's how we identify those as soon as we see some traffic on this we alert I mean alerts us it alerts the team as well It's it's a weird trade-off between productivity. You don't want to block people doing what they need to do but also being secure right so this We are looking at moving to strict But at this time we are like still in permissive That's a really good question I have as well to be honest and I Think it's a mix of Historical reasons and validation web hook being really easy to to deploy yourself. I mean it's the code is really simple But yes, we are maybe looking at consolidating. That's why I mentioned it to appear I think I used K rail in the past as well, which is also a similar project, which is very Kubernetes centric Any other questions on the stack? Yeah Yeah, that's that's a good point. I mean we try to To keep our cloud deployments on layer three layer four So we don't use things like layer seven gateways on the cloud for example everything comes into Or envoy gateway and we do everything at that level Yeah But I guess we're always looking at what could be done. But at this point, yes, we Yeah, we try really strongly to be cloud agnostic. So that's the Kubernetes API is our point of Referencing and that actually is how avatrix achieves a lot of its features within the networking stack is by providing One central abstraction over the cloud provider specific implementations of security Well, thank you all for coming to our talk. We'll be around for questions or restaurant recommendations after