 Welcome everyone. Thank you for coming today. My name is Bik Lee. I'm a co-founder and chief architect at Platform 9. And today I want to talk about something we built, and it's a lesson about how to use Kubernetes building blocks to create something useful. So we built a tool to make it really simple for us to deploy large numbers of services running on behalf of many, many customers and services that are exposed on the Internet and how to do this in a secure enough way to isolate each customer. And that's what we mean by multi-tenant. So to motivate today's talk, let me briefly discuss our use case at Platform 9. So we run open-source software as a service for our enterprise customers. And we specialize in infrastructure management software. That is why we offer an integrated product that uses Kubernetes for container management, open-stack for virtual machines, and fission for serverless functions. Now what makes this unique is for these stacks we split the control plane and the data plane so that the customers can run the data plane on the hardware of their choice, which could include bare-metal servers or their public cloud instances. And then we host the control plane and run it as a service, and then the two planes communicate securely over the Internet. So what does a control plane look like? For Kubernetes it would be the master nodes that most of you are familiar with. For open-stack, we would run dozens of services such as Nova API. That's one example. And then on the other side, our customers run the clients. For example, they could run web browsers, CLIs, SDKs, and those would typically communicate over HTTP, over the Internet, to one of the control plane services. But there are also data plane agents which run close to the hardware and those need to use binary protocols over TCP. And examples include RabbitMQ and even MySQL. And those connections tend to be mutually authenticated over TLS. So we are currently in a transition period moving all of our control plane services from VMs to containers. And of course, we want to use Kubernetes and take advantage of it. We also want to use and run everything on a shared Kubernetes cluster to gain high utilization and, of course, all the benefits you get from Kubernetes. So during this transition, we wanted to build a set of tools that would help our ops team automate as much of this as possible, right? Something that lets them very easily provision new customers, new regions of customers, deploy the services, make sure that they're exposed on the Internet with DNS names, and secure everything so that everything can be well isolated and support multiple protocols such as HTTP and also some of the TCP binary protocols. And since our ops team is kind of ramping up on Kubernetes, and Kubernetes can be pretty complex, we also want our tool to automate as much as possible. You know, automate a lot of the repetitive tasks, maybe abstract away some of the complexities of Kubernetes without really hiding its true power. And finally, we wanted to build something that is useful not to us, but also to the community. And that is why we started an open source project. And this, I'm going to talk about kind of the very first generation because it's a work in progress. It's called deployment cluster config and ops, also known as DECO. And this is a program that follows the typical Kubernetes controller or operator pattern. What it means is it declares new custom resources that you as an admin can create in Kubernetes. And once it detects those resources, it'll just kick in and do a lot of automation for you. And we'll talk a lot about this. So, for example, it will manage DNS records and inside of the cluster, it'll provision and tear down all the resources that you need to coordinate Kubernetes like namespaces, deployments, services, ingress resources, all that stuff. Okay. From a network security, DECO is going to use a combination of two things, end-to-end TLS and also a network policy. So having said that, let's kind of dive into some of the concepts exposed by DECO. We're going to be talking about spaces, apps, and projects. So let's start with space. So think of space as a naming and security boundary for a set of internet-facing services. Okay. And in DECO, we implement this as just a Kubernetes namespace with a DNS binding so that it is visible on the internet at a well-known name. To illustrate this, let me show you a little animation here. So when you run DECO, it automatically creates a special namespace called DECO. This is where you're going to upload your space resources. Okay. And a space resource describes several things, but the two most important ones are the name of the space and its domain name. So the first thing that DECO does is create a namespace with that name. So in this example, it's called example space 1. And then it will create the DNS binding. And the way it does that is it first looks up the public IP or endpoint for the main ingress service of the cluster. Okay. And then it's going to communicate with a DNS provider to create a DNS record. And the record is going to create maps, the fully qualified domain names. So in this case, example space 1.platform9.net. Okay. So now things inside of that new namespace can be accessible from the outside. It also populates the new namespace with a number of resources to configure traffic management and security. And we'll talk about that later. So now that you have a new namespace or a new space in DECO terms, you are ready to provision applications into it by creating app resources. Okay. So let's talk about apps. So an app resource really just represents an internet-facing service. And it's just a thin wrapper around a pod spec. When you create an app resource, you tell DECO how you want your service to be visible on the internet. And you have two choices. If it's HTTP service and you want to use a reverse proxy, then you specify a property called HTTP URL path. Okay. What this means is if you set that property, then your service is going to be visible at this endpoint, which is HTTPS slash slash the FQDN of your space slash the URL path that you specify. All right. If you don't specify this property, DECO will assume that it's a TCP service. And it will create a new FQDN and DNS record for your service. And the way it's going to name it is it's simply going to concatenate the application name with the FQDN of the space. Additionally, you can specify a flag that determines whether when you do a TLS handshake, whether to verify the client's certificate using a well-known root CA. So that's the verified TCP client cert flag. So here's an example of an HTTP service, Keystone, which is mapped to the path slash Keystone. And we use this in OpenStack. Here's an example of a TCP binary protocol service. It does not have the HTTP URL path. And additionally, it specifies that yes, we do want to authenticate the client TLS certificate. Next, we have the concept of a project. This is really an optional feature of spaces. And what it allows you to do is restrict internal traffic within the cluster. So normally when you create a new space, the pods in that space will accept traffic from any other space within the cluster. But you can restrict that by declaring this field here called project. So here what I'm saying is only accept incoming traffic from other spaces who also have their project set to dev. All right? Internally, Deco uses a Kubernetes network policy to enforce this. And hopefully I'll have time during the demo to show you what it looks like inside. Here's an example of how you could use projects. Here we have two customers, foo and bar, for which we've provisioned multiple spaces representing different regions. So for example, we have foo east and foo west. And to make sure that within a customer, the spaces can communicate with themselves, but disallow communication across customers, we set all the spaces belonging to foo to have a project equal to also foo. And we do the same thing with bar. So they can talk to each other, but not across customers. We also have a global space here running a service we call its console, and we use it as a global configuration database. So each pod, when it starts up, it needs to talk to console in order to retrieve its configuration. And so all spaces need to be able to access console. So what we did here is for the global space, we set the project to be empty, and what that means is it can accept traffic from any other space. Last but not least, let's talk a little bit about traffic management. So this is probably the most complex topic. This is about network routing, and it's also about TLS termination, which are related. So DECO is built to accept multiple providers of traffic management. The requirements we've already discussed, it needs to support HTTP reverse proxy from the internet. And for TCP services, it needs to support end-to-end TLS authentication from the client all the way to the pod. And this has to be authenticated on a per customer or per space or per app basis. It is our intent to integrate with service meshes, especially Istio. That's currently in progress. We are currently blocked by an issue in Istio. It's about SNI support for multiple server certificates. Looks like they're working on this, so I expect this to be fixed within weeks. But luckily DECO also comes with its out-of-the-box, very simple traffic manager, which I'm going to describe today. So to route traffic, the out-of-the-box manager uses off-the-shelf components. And this is pretty interesting, I think. It's using two Ingress controllers. One of them is called K8SNF. This is a pretty neat controller that was first presented at the coupon Berlin. So that one's going to look at TLS traffic, and if it finds a matching destination that matches the SNI headers, it's just going to direct traffic to that service. If it doesn't find a match, it's just going to punt it over to NGINX, which is going to handle HTTP and reverse proxy. In order to handle TLS termination, DECO will inject external sidecar containers into pods. And those external sidecars need certificates and route CAs to verify identity. So typically what you're going to do is you need to create and sign your certificates outside of the cluster and then supply them to DECO through secrets. And we're going to look at an example. Okay, so here we have the DECO namespace that I referred to earlier. And this is the two Ingress controllers that I talked about. They're configured to be chained one after the other. So I'm first going to upload two secrets containing certs and CAs for HTTP and TCP style services. And then I'm ready to create my space. Okay, in this example, it is a space called foo-west with a domain of platform9.net. So the first thing that DECO does is create a namespace called foo-west and then copy over the secrets so that they can be used within the namespace. And then it'll create an HTTP Ingress resource with a virtual host set to the FQDN of the space, which is foo-westplatform9.net. It also initializes a rules table for the paths. And initially it's empty except for slash, which goes to the default service. Okay, so now we're ready to populate the namespace with new applications. So the first example I'm going to show is MySQL, which is a binary service. What DECO does is it'll create a deployment resource, a service resource for MySQL, and notice that in the pods, it also will inject an S-Tunnel Ingress container in order to process incoming TLS traffic. And that S-Tunnel will use the secret that we configured earlier. But since this is a TCP... Sorry, this is a TCP service, DECO also creates a TCP Ingress resource, which is just an Ingress resource, but with an annotation that says that it is meant for the K8 SNF controller. And I'll explain how that works. So now a client on the outside can connect to MySQL.FooWest.platform9.net. That will get routed through the external load balancer and into K8 SNF, which is then going to look for a matching TCP Ingress resource with the same hostname. In this case, it does find it, and that directs it to the MySQL service, which takes us into the pod, and there the S-Tunnel container will terminate TLS and verify the identity of the client. And if everything is okay, it'll forward to MySQL. All right? For an HTTP app, let's say Keystone, here we specify a path of slash Keystone for the reverse proxy. Same thing, DECO is going to create the deployment, it's going to create the service, it's going to inject the sidecars. But since this is for HTTP, it's also going to add a new rule to the path array of the Ingress resource for HTTP. So now, if a client accesses this URL, HTTPSFooWest.platform9.net slash Keystone, again it's going to go to K8 SNF, but this time it's not going to find a matching TCP Ingress resource. So we've configured K8 SNF to just punt over any traffic that it doesn't recognize over to NGINX. So now NGINX will look for a matching HTTP Ingress resource, it finds one with the required virtual host name, it follows that, it also looks at the rules table, it finds a matching path, and finally it directs the traffic to Keystone. All right. Now, in addition to all of this, we also support internal traffic between applications. When you specify an app resource, you can specify a list of egress entries for other apps that you want to talk to. And for that case, Deco is going to inject additional external containers for each of these egress entries and encrypt the traffic and send it to the appropriate service. So the conclusion of all of this is Deco treats internal traffic just like external traffic, right? Everything is meant or assumed to be insecure, and so you always have end-to-end TLS encryption and customers are isolated between each other because they will typically have different root CAs, different hierarchies. And in addition to this, as I spoke earlier, we use network policy with that concept of a project to add an additional layer of security. All right. So I think that now we're ready for a demo, and I'm doing this over VPN. I hope it doesn't crash. So let me exit out of this. Okay, it looks good. Okay. The Deco control can run inside or outside of the cluster. It doesn't matter. It's an operator. Let me start it in here. I hate typing, so I have alias KC to cube control. I hope you forgive me. So let's look at what's going on here. So we have the standard namespaces and also the special Deco namespace that I told you about. And let's look at a bunch of resources running in the various namespaces. So in Deco, as I said, there are two ingressed controller services, K8SNF and NGINX, and only K8SNF is exposed to the outside world. Okay? You'll remember this IP here. We're going to go back to that. Okay, so we're ready to create our first space. It's going to be called dev1 with a domain of platform9.horse. You might wonder why .horse. It's because we use this in dev and test and .horse is actually one of the cheapest domain names you can buy. And one thing I forgot to show you was I've also uploaded some secrets for my space. So this one is verifying TCP traffic and it has a CA in it in addition to the cert. This one is a wildcard certificate for the HTTP proxy. So I have those two secrets specified here. So let's go ahead and create this dev1. Oh, before I create this, let's make sure that the hostname, the DNS is not set up yet. So right now this results to nothing. Okay, so create dev1. Okay, so we created the space. Let's look at the resources that were created. Here's our space and notice something new here. There's something called dev1 and that is the new namespace corresponding to our deco space. So if I say K, you'll see dev1. Okay, now let's see if deco has created the DNS binding for us. It has, okay. So now our space is accessible to the internet using a well-known name. And next we are ready to create our first application. And I'm going to use something we demo a lot. It's called Time Server. It's kind of stupid, but it's great for demos. It's just a web server that returns the current time in a pretty cute UI. And this is just a regular pod spec with some added metadata. And here we're saying that it will be mapped to the time server slash time server path. Okay, so let's create this guy. And we need to create in the dev1 namespace, oops. Okay, so we just created the app. Let's see if it appears anywhere. Oops, let me resize this a little bit. Okay, so where's our app? It is here, okay. And you'll also notice that we've created a deployment and there should be a service as well. Okay, so if everything worked well, the service should be available through that URL. Okay, there it is. But for a better visualization, let me copy this and paste it into Chrome. Okay, so it's just displaying the current time. And there's one color per pod. And currently we only have one pod. But since this is just a regular deployment, we can scale it to three pods. And let's see what happens. Okay, now we get three colors. Okay, pretty simple, but it's showing that there's three pods backing the service. Now for our next example, we're going to look at a TCP type of app. And I'm going to show you, let's see, MySQL. There it is. Okay, so we're going to spawn a MySQL container with a root password set to dummy. And we are going to require client verification for TLS. And remember that for TCP type of apps, we're going to create a new FQDN, which is the app name followed by the FQDN of the space. And right now the DNS record does not exist. So let's create, okay, we created MySQL. And now we have wrong namespace, dash and dev1. Okay, we have MySQL running. And once again, there's a deployment, there's a service. And let's look, take a close look at the pod that's running. Notice that there's two containers. So if I say kc dev1 describe, let's see if this works. Okay, here's our time server container. But also notice that we've injected a nest tunnel ingress to handle traffic coming in on 443. And it's going to forward that to the MySQL container. Okay. Actually, did I look at the wrong one? I think this is, I'm showing you your time server. Yeah, I showed you the wrong one. Okay, so here we should see MySQL. And with its sTunnel container, it's listening on 443, but then it's redirecting and forwarding to 3306, which is the database container. Okay. So how about DNS? DNS has been created for us. And now we can connect to our database over that well-known name over the internet, except that it's enforcing strict TLS. So we can't just run our standard MySQL client against it. So what I've done is I have a local sTunnel configuration that uses the correct client certificate to authenticate itself. It's also using the correct CA file to verify the server. So I'm going to run sTunnel. And now let's see if we can connect to the database. There we go. So databases. Okay, so we're connected to MySQL. Great. The last thing I want to show you is you'll notice that our space set the project name to dev. Okay. And as explained earlier, we use network policies to enforce this interspace traffic. And so somewhere in this list, we should see a network policy. There it is. Okay. So let's take a look at what's inside. Dev1. Okay. So what you'll see here, I don't know how much you know about network policies, but there's various ways you can specify who can do and who can connect to what. Here we're saying only allow incoming connections from namespaces with a label that has this pattern. Deco project equals dev. Okay. So that's what we expected. But also notice that we've inserted another one here for a mysterious namespace with a label, a Deco project equal system. And the reason we had to do this was to allow traffic from the ingress controllers. Right? Because remember a K8SNF and Nginx. If you didn't have this, then the space would not be able to get any traffic from the internet. Okay. So we're almost done. In terms of road maps, so this is work in progress. And from a security perspective, I discussed how we handle network, but we want to go one level further and isolate customers between each other even deeper. You've probably seen a lot of projects around using virtual machines as a better way to isolate containers. So this is kind of our next step is to investigate integration with many or some of these technologies. In terms of other things we want to integrate with, as I mentioned earlier, service meshes. We want to be able to also take advantage of automatically setting up pod autoscaling and perhaps note autoscaling. And then we've looked at single Kubernetes clusters, but we would like to expand Deco to support multiple availability zones and also Kubernetes Federation. So that's some of the things we're going to investigate next. And to find out more, there's a GitHub repo. We need to add some documentation and a lot of tests. You can visit our company website. We have some pretty interesting talks this week. If you like to hear about serverless and functions as a service, we have a talk about Fishing and a co-speaker and myself. We're also going to be talking about how to run Kubernetes clusters more cheaply. And that's Friday. And that's it. And thank you so much for coming here and thank you for your time. And I'll take some questions. So the question, do you have any customers? The answer is we're the first customer. This is just brand new. We're developing this as we go to first meet our use case. Yes. Yes? It looks like each space is a single external IP address. Was that true? Was I servicing that correctly? No. All the FQDNs all map to the same endpoint or IP address. Which is the IP address? The entire Deco cluster across customer is one. Yes. That's it. Yes. Feel free to grab me on the side or in the hallway or at our booth. Thank you.