 Thank you all for coming here this afternoon. I know there was a choice of sessions to attend, so we appreciate your being here. This is the container networking, CF networking update. Our tagline is all your packets are belong to us because that is indeed true. All your packets do belong to us now. By us, I'm talking about the CF networking team. That's responsible for delivering all these amazing things that we're going to be talking about this afternoon. My name is Usha Ramachandran. I am the project lead for the CF networking project, and I'm joined by Jay Dunkelberger. He's an engineer on our team, and we're going to walk you through some of the stuff this team's done. We have a pretty diverse team. We have folks from IBM, Pivotal. We've had people from Swisscom, so a lot of engineers and PMs have worked on making this product successful. I'm going to be talking today. The agenda that we have for you is basically to go over some of the networking challenges. How many of you were here at the CF networking session last year? A few folks. I'm going to start with a little bit of a history lesson with what were the challenges that we spoke about then that we were trying to address, and then we'll talk about CF Networking 1.0, and then Jay is going to do a deep dive into how it actually works and a demo, and then we'll spend a little bit of time on the roadmap. This is a 30-minute session, so we'll try to get through everything. If we don't have time for questions, we do have office hours later this afternoon, so stop by, ask us your questions after the session or at office hours. Okay, let's jump in. So when we started last year, we were looking at Cloud Foundry Networking and some of the pain points that we were hearing over and over again around the fact that today when two apps want to talk to each other within Cloud Foundry, they have to go all the way out to the router and the external load balancer and come back in. This is not desirable for a couple of reasons. So the first one is of course that it's very obviously inefficient, that even though you have apps that are maybe sitting on the same Diego cells, they still have to go all the way out and come back in. The second part of this is around policy. So what you really want is for apps to be able to talk to each other only if they're allowed to talk to each other. The mechanism that we had for some of this policy enforcement was basically really coarse-grained application security groups, and they're egress only rules. So when the destination app B basically got traffic, it just had to trust that what it was getting was indeed from app A. So the desired state, not talking about implementation or how we did it, is really that I have two apps, let them talk directly to each other, they shouldn't have to go through the go router, and when traffic reaches app B, it should know that it came from app A, and I can configure policies that allow app A to talk to app B, and don't allow app B to talk to app C, and so on. So that's where we were and where we wanted to be. And where we're today is that we just released CF Networking Release 1.0. So you can see we're really excited about it. And I'll go through a little bit of what we did with 1.0. What does it mean for you? What are some features that you can use with CF Networking Release 1.0? So the main features for CF Networking Release are basically concentrated around connectivity. So we wanted every single container to have its own IP, and to be able to be identified on the container network. And in order to do this, we started off with adopting a standard. We chose the container network interface, or CNI as our standard, so that we could not only provide a batteries-included solution that people could use, but also open up the door to more advanced SDN sort of integrations into Cloud Foundry. And so that was a big decision point for us to choose CNI as the standard to follow. We also implemented a batteries-included solution called Silk. So Silk is a CNI plugin that's built from the ground up, and it's basically built for Cloud Foundry with the operational guarantees that the users of Cloud Foundry have come to expect in mind. So it's backed by a SQL backend. It takes into consideration a lot of the Bosch lifecycle events that are just going to happen as part of your day-to-day within Cloud Foundry. And what this CNI plugin gives you is a unique IP address for every single container on an overlay network. And as I mentioned before, this also means that if you have a third-party plugin, like if you wanted to integrate NSX or ACI or Calico or Namer or third-party plugin, you now have a way to do it. So connectivity is really great, but I think as operators, we really wanna control what traffic is allowed within a network, and this is where the policy comes in. And you start off with all containers on an overlay and it's a default deny rule. And an operator or a user can go in and configure app-level policies to start container communication between two applications. So that's one big thing we did, expressing app intent to drive policy, and then having it being dynamic and configured through a CLI or an API or a few things in terms of how this is implemented. We have a couple of different models for configuration. So if you're in a very secure environment, there's a network admin who can configure policies. If you're in a more self-service kind of environment, then you can grant space developers the ability to configure policies. So we think it's pretty flexible and a cool thing to have. So there are a few things that we haven't changed. We haven't changed the way application security groups work yet. So we're working towards it, we'll get there. And we haven't changed how traffic is sent out from a cell using that or how traffic gets into a cell through the go router also using that. So those are some things that remain unchanged with this release. So at a high level, this is what it looks like. You have all your apps that are running. You configure your policy at the app level. The important thing to note here is as IP addresses keep changing for the app, you do not have to reconfigure your policy. Your policy stays the same. All your containers are on an overlay network and then all the cells and your infrastructure continue to be on an underlay network. So this is just a very high level. Jay is gonna go into a lot more detail in how does this actually work and how do you get packets across the board. So the key use case that we're trying to enable with this is basically the microservices use case. So if you have a private microservice that you don't wanna have public crowds, then this is a path to go down. Also, if you're looking for improvements in latency so that your apps can talk directly to each other, that's another use case that you get by using CF networking release. There is one caveat which is that because you're using container to container networking directly, you do have to bring your own service discovery so that your containers can now discover the other end points. So we have some examples on our website, on our GitHub page, one with Eureka and another one with Amalgamate. So the net net here is bring your own service discovery and we'll do the container networking with policy for you. So I'm gonna hand it over now to Jay who's going to tell you how it all works. All right, thank you Eusha. Yeah, so I'm gonna just go through some of the details here of the architecture and coordination that we have to do to make this high level picture that Eusha presented a reality. So here is a diagram of the architecture, essentially what our team has built in the last year. So I would pay attention to the color coordination here. So the blue represents the existing pieces of cloud foundry that were there before and are still there today. So cloud controller Diego and then on the Diego cell garden is running and creating containers. We added two new things that are what we consider core pieces of the new networking stack. One is this garden external networker. This is a binary that's on the cell and it's called by garden when it's creating containers. And this is how we hook in the garden to set up networking. And we have a contract with garden about how that will work. On the other end of the external networker, the external networker is calling CNI plugins. So this is where third parties can fit in. Outside of the cell we have a policy server. So this is where the users can create or delete their app tab policies. This is typically co-located with the cloud controller. And it has an external API that the user interacts with as well as an internal API that policy systems can read from. And in our case we have a policy agent that does that. All right, and then in red is the batteries included solution that does all the networking right now by default in or will do all the networking by default in cloud foundry. So it starts with a set of CNI plugins. There's a big chain of them that get called when we do a container create. There's a daemon running on the cell that is coordinating with this cell controller up in the top left. That's how we essentially coordinate subnet and IP management. And that cell controller is typically co-located with the BBS. On the other side of the cell we have our policy agent and that is the thing which is reading the internal API and then creating policies to enforce what the user desires. So we're gonna step through it and just kind of show how all these things fit together. So start with the basics. I'm a developer, I push my apps, I run CF push. That application flows through cloud controller and Diego eventually gets scheduled to a cell. One addition we've made is that now there's extra metadata for the app space in Orguid that is associated with that and that flows all the way down through to our CNI API. And this is important for our policy system so our policy system is able to associate apps with IPs. So, all right, so Garden creates a container. It calls our external networker. The external networker then turns around and calls our CNI plugins. Again, it goes through this chain, there's a few calls. So it calls the Silk CNI plugin. That Silk CNI plugin is going to coordinate with that local Silk daemon to figure out what subnet it should be assigning IPs from and then we assign an IP through a standard IPAM plugin for the container and then it sets up a VET device with one end in the container and the other in the cell. We also have this wrapper plugin which is responsible for setting up all of the legacy networking features, the ASGs and port forwarding for the host. So for access from the go router or from for SSH. And then we also record that IP to app correspondence in a file that's on the system. All right, so now that the app is up and running and on the network, we want to create policies so the user interacts with our external API. They can interact with it directly or they can use this CLI command and then the policy is created and we assign a tag to that application. Then the policy agent is watching that internal API or it's polling not watching and it gets updates for applications that are running on the cell. Then it looks at the net state file to see which containers that corresponds to and then it renders IP tables rules that will enforce the whitelist rules that are necessary to make the network policies function. So that's the flow for kind of like the life cycle of things. So how do we coordinate these subnets when we have many different cells? So that's done with the controller. So the cells are essentially just interacting with the central database. The controller is backed by a SQL database and it's storing a correspondence between a cell IP and a block of IPs on the overlay. So in this example, we have these two cells. They have unique underlay IPs and then they have unique overlay subnets that map onto that. So they're renewing their leases, then they get all the leases and then that entire lease state gets returned to the cell. So it knows where to find the other cells. So let's say a new cell comes online, it acquires a lease. So that lease now appears in the database and when the other cells are updating, they will see the new lease as well. Once that's done, now cell three has a good subnet. It's able to just start handing out IPs to the containers as they get scheduled there. So what happened then? The opposite case is cell goes away. So let's say a Bosch deploy is coming through, the cell needs to shut down. All the containers get evacuated off the cell. That's a normal part of the Diego life cycle and then when the containers are gone, we release the lease and it stops, that lease stops being reported to the other cells. Okay, so with all that done, how does a packet actually flow from one cell to the other? We do this with a VXLAN overlay. So here's just a diagram showing an example. We have app B on the left, which is trying to send the packet to app A. So it just sends a packet with its IP as a source and the unique IP of the container on the other end as the destination. If there's a policy in place, that packet will get marked. It then goes to the VTAP device, the VXLAN device on the cell and that device is programmed with this correspondence from the previous slide of like which underlay IPs correspond to which subnets on the overlay. So then the packet is encapsulated and is sent. So down here at the bottom, we have a packet where it has the initial container to container packet inside of another packet where the source is now the IP of the sending host and the destination is the IP of the receiving host and that's sent over UDP on a special port to the other cell where it's then decapped and then now the original packet is restored with its mark. This is done by some magic with the VXLAN header. And then based on that mark, it can be whitelisted and allowed into the destination container. So all this whitelisting is actually done at the destination. Okay, so that covers the data plane and kind of conceptually what's going on. So let's see it in action. So if I come over here, I have two applications running. So this is an example that's in our GitHub repo. You can check it out and try it yourself. So I have a backend app and a front end app. For my backend, I have four instances. My front end is just one. So if I come over here, my front end looks like this. I'm gonna delete this first. So it basically is letting me make HTTP requests or do like a UDP Echo server with my backend. The backend is over here. And so this is at its public route. It's just reporting its IP. And it's telling me that it's serving pictures of cats on these TCP ports. So if we go and just look at the cell to see what's going on. So we have a deployment. There are four cells. Okay, so I'll just SSH in. All right, so let's just look at the devices on the cell. So now in addition to this host interface, we see we have a VTEP device. So this is the thing that actually makes the overlay network happen. And we also see the host end of these veth pairs. And they're paired with the container end that's in the containers network namespace. So you see on this cell, for instance, we have three containers running. And yeah, we see that the VTEP device has an IP from the overlay network. From the cell, we can also just, we can curl the controller to see what the leases are. So this is what the daemon is seeing is essentially this bit of data. And that tells it where the other cells are, what their IPs are and what the overlay subnets are. And then I'm just gonna show the IP tables really quick. There's a ton of IP tables rules. I won't get into what they all do, but if you're on a cell and you see a bunch of IP tables rules and you wonder who put them there, the answer is we did. All right, so, what is this? All right, let's try. So I have a few different instances here. I can just reload and see different ones that are answering. So let's try sending a packet to this back end from the front end, all right? So I'm gonna send it port 7,007 and I get an error. This is actually an error from the client. It can't reach the back end. This totally makes sense because if we go over here, I can list my network policies and I see that I shouldn't have any and I don't. So, it's being a little slow, unfortunately. So let's change that. So I do cf allow access. This is a CLI plugin that we've made so you can get it from our release repo. So we'll make it from my front end app. So the source app to the destination app port is 7,007, protocol is TCP. Okay, that went through and now if I list, I see this policy, so let's try this again. And now the back end replies and it gives me a picture of a cat. Okay, so that concludes the demo. I'll hand it back over to Usha and she'll talk about some of the future work the team's gonna do. Thank you, Jay. That demo works every single time, I think. Even if everyone's asleep and then you see a picture of a cat typing away furiously and everyone wakes up, okay, awesome. So now that we've seen the demo of how it works and I hope that you see now the value that you're getting for app-to-app communication and how it's enabling a new use case for direct app communication, all policy-driven, all dynamic. Let's talk a little bit about where we wanna go from here and we would really like feedback from the community around problems that are top of mind for you and to help guide us along some of the paths that we're looking to explore. So to start with, we're actually looking at a couple of different things. So we delivered the container-to-container policy part of it and that was our aim to focus on the policy part. But there are some enhancements that we're looking to make just to make that feature itself complete. So there are a few things that we have in flight. So if you noticed, in this demo, you could only give one port in your destination. So if you had an app that needed to open 100 different ports, that's like 100 CLI commands. So we're adding support for port ranges and we're also doing some logging enhancements that are gonna help it be easier as an operator to make sense of some of your traffic logs. So that's stuff that's in flight. The other big thing that we really see is a pain point that we might wanna address is a building service discovery into the platform. And you'll see that I have all different font sizes. That's intentional. It's just to show that service discovery is something that we're looking at pretty actively. And then adding stuff for meta policies like space level policies and doing policy config through a manifest, things like that. So the other theme, the big theme that we have is we have all these app level policies and they're really great because they helped us move away from wide open ASGs to allow app to app communication. But a big pain point that we hear around ASGs is still that it still applies for all the other things that you want to connect to. So as an app, you're not only talking to other apps, you're talking to services, you're talking to a backend database. So the other aspect of policy is just trying to extend it to start including other destinations. And then finally, we're also looking at just enhancements to cloud foundry networking in general. So adding app identity so that you can do your own enforcement outside. And I'll go into a little bit of detail of what this means. And then we keep hearing about IPv6. So we wanna know if there are a lot of use cases and what those use cases are. So if you have thoughts about that, do let us know. So I'm just gonna touch upon a couple of the things that I spoke about in the last slide. So if you think about policy, today we only have the overlay network for containers. And so that makes it difficult to really extend the policy to other entities. And so what this slide is showing you is basically just a vision of extending our policies to support egress destinations using IP addresses to support wash deployed services and then also to support policy for inbound traffic. So as you look at this, it just really enhances the network security posture of your Cloud Foundry environment. And the app identity piece is basically how do I take my app in Cloud Foundry and how do I make it appear like something reasonable that an operator on the other side might make sense of. And we can't really escape it, but IP is still that universal language of being able to identify different endpoints. So that's kind of just talking about how we're looking at these problems. So I'm sure after you saw that demo you wanna go try this out yourself. So we have a GitHub page that you can get a whole lot of documentation around how to deploy this. You can deploy it with CF release. We're an optional flag on the manifest generation. And then you can also deploy it with CF deployment. We have some examples. We are really friendly on Slack. So if you have any questions, stop by, ask us your questions. We're very happy to hear from you. And we have a couple of blogs, just describing the things that we've done and some of the things that we're looking for in terms of the future of a network security. All right, we're gonna stop here for questions. We have time for questions? Okay, and if we don't get through all your questions, stop by our office hours. They start at 3.45 and our whole team will be there to help answer any questions you have. Are there any questions? Thank you. Thank you, Shah, from All Ranch. We are using isolation segment basically with VLANs and external firewall to enforce policies for app-to-app communication. And I was wondering if we push on CF networking on our production environment, basically. How do you infer that and have in the interaction segment, hey, let's say, can't talk to an app or the VLAN can't expand between two different exhaustion segments? Thank you. If I can paraphrase that question a little bit. So the question was that if I have apps in two different isolation segments, how can I not allow an app in one isolation segment to talk to the other one, to an app in the other one? So today, the isolation segments feature and the container-to-container networking feature are actually different features. So the way that would actually happen is that your two isolation segments would be on different networks, I would assume, and you would have firewalls at your network layer that would prevent that from happening. So even if you had a container-to-container policy, it wouldn't work. Any other questions? You're not going to be able to answer. That's a great question. The question was, how are you doing load balancing with container networking? And the answer to that is that we're not. We are relying on you to bring your own service discovery to do that for you. And so that's why it's an important consideration. You would use something like Eureka with client-side load balancing in order to get the load balancing aspects of service discovery. Did that answer your question? Sounds like you want the service discovery feature. That's great feedback. Awesome, thank you. Any other questions? Yeah, the question was, is there a plan to add service discovery in this release or in the future? So we're not going to be adding it right away. We are looking at service discovery as one of the things that we want to add to the platform. So it's something that is a roadmap item. And as we get more responses and flesh out the requirements, that's when it'll get added into a release. Any more questions? Back there. So the question was, is there any plans of doing host name instead of IP for communication between containers? And this basically ties back a little bit to the service discovery question because that's one of the ways we could solve this where a container basically would look up a name for an app and then that would just get resolved as DNS and then translate it to an IP address. So that's definitely something we're thinking about doing one way of implementing the service discovery. Any other questions? Sure. Yeah, and that is something we're thinking about. So it would be possible on the container network, especially if we had something like host names for the applications, but I mean, it's definitely something we would like to do in the future, but it's not an immediate goal, I guess. We want to know why you want it and how much you want us to add it. I don't know if it's your point. Any other questions? Awesome, hope this was helpful. If you have any follow-ups, please do reach out to us. We'll stick around here and if you want to come talk to us one-on-one, we'll be here or at the office hours. Thank you so much.