 Hello, my name is Gabe Rosenhaus and I'm Matt Sykes welcome and we're part of the container networking engineering team So Most of you probably already know the pitch You know why why are people looking to the network the containers? What do they get out of it one of the shortcomings of what we have today with cloud foundry in terms of connectivity between containers And we kind of try to summarize that here in one little bit Essentially what we've got today is we've got a model where Users come in workloads come in and through a front-end router the router then finds the container instance That's hosting your application. It dispatches it inside the container But if that container then needs to talk to some other deployed application some other container in the cloud foundry network The only real way you have today to do that make that work is to come back around through the front door So you basically hit your load balancer you hit your proxies you have to go router You get back to the container that you actually wanted to hit and it's kind of Obtuse it adds a huge expense for the round trip It also forces you to expose Routes to your application in the back end that you may not otherwise want to expose So what we really want is we want to have a model where you come in through the router You can access your front-end apps But those applications can then come directly to the back end that you're trying to keep secret or private or secure Or whatever you want something that you want to just manage so that you don't expose it to the whole wide world And that's kind of what we're trying to do with container networking More explicitly we have a set of goals that we're looking to deliver in this project for users and for operators All right We're gonna try to talk through some of these things and and give examples of how we're doing it And we're kind of hoping that throughout this discussion If there's questions comments concerns, you kind of let us know so we can have that conversation either after the pitch Or during q&a or out in the halls, whatever might be So in terms of the container networking project in general, we have a set of small We have a very small set of goals that we're looking to get to So for users, we want to deliver a solution that allows direct layer 3 Addressability without Nat between all connected peers So what that means explicitly is that if you have two instances to containers right of the same application or other Applications you can uniquely address them and send requests to them without having to go through the entire routing tier All right, so when you receive a request now, you know explicitly who it's from in terms of the address and you have some Notion of the source you have some notion of you know who you're talking to Within the container we're looking to have a single Network interface so today what you have in in the garden containers that exists You have a single network interface that you can receive your application requests on that's great What most container networking models end up doing today if you look at Docker for instance If you have a plug-in or you add another network or even on Some VM machines if you add another network what you end up finding is you have another Network interface sitting inside your content inside of your container your host That adds a fair bit of complexity and complications around determining, you know What addresses you should listen on for instance does everybody do an add or any or do you listen on one side? Which one's private which one's public most people don't want to care So by surfacing a single interface with a single address It's very easy for people to basically come up listen listen on one address and just get going In terms of connectivity between applications We want to be able to allow you to specify a policy a way of describing who can talk to whom right and how So those are the goals that we have for users So you have basic connectivity with direct access single network interface and policy based connectivity rules Now for operators We want to make sure that when you're running cloud foundry since we are multi-cloud and we support many different cloud providers Right, you could bring your own you can go someplace else We want to give you the opportunity to actually choose the networking technology that's going to be in place For for your network, right? So we need some level of extensibility there We want to ensure that when you deploy these network technology that within your infrastructure within your deployment You have the ability to lock things down to explicitly control Where packets are allowed to go right so we want to make sure that that's there and a Stretch goal or something that we hear a lot right something that we really want to have happen Is we want to be able to say that as packets traverse the network right the packets that are coming from containers We want to be able to say you know with certainty the source of that packet So we can tie it back in a multi-tenant world exactly to the application that generated that packet So if you end up getting you know a huge flood of traffic from an application You can track it back or if you find out that there's an attack on a foreign web service Right, and you want to find out who's actually causing that in a multi-tenant public cloud You have the ability to do that. So those are some of the goals So how do we get there? So we have five basic topics that we're going to try to cover today We're going to talk about the technology sensibility. We're going to talk a little bit about address management, right? Policy expression for Cloud Foundry and some ideas that we have there and we're looking for lots of feedback on this because this is a probably the most intensive Portion of this presentation is to get some feedback on the policy pieces well when you want to control who can talk to whom It can become very detailed and very you know Basically, it's kind of obtuse right so we're looking to have a simple solution for that That still meets requirements people have so we're looking to cover that today We're going to talk a little bit about what we have for batteries included solution in the net man release So what's going to happen with open source Cloud Foundry just for connectivity and how the policy enforcement will work with that built-in provider and the last little bit we're going to talk about is some Some ideas around how we could discover Endpoints things like naming so that if you have connectivity from A to B How do you actually find out where B is in terms of addresses and how do you access those things? All right, so that's kind of what we're going to cover today So you just if you're in this room a few minutes ago you heard to talk about garden And you heard about garden run C which is the next version of the back end for the garden API What our team has been working on the last few months has been a Adapter between garden run C and the container networking initiative specification for wiring up containers to networks And what that does what we've got is this this adapter layer that lets us drive via the CNI plug-in interface Any number of different CNI plug-ins which are available if you if you look this up on github You can see that there are a number of Existing reference specification implement like reference and implementations of CNI plug-ins and We have this thing Called netman release. It's a Bosch release that you can go look up today And it uses the final CNI plug-in and provides layer 3 connectivity between containers It doesn't meet all of the goals that we've that Matthew went over But that's what our team is going to be working off of going forward All right, so to dig it a little bit to the address management piece What we have here is we want to demonstrate you know really in picture form the ultimate goal, which is given a graph of containers We want to be able to allow full connectivity between those graphs and unique address ability So what you notice here is that we actually have four boxes. They look like servers But there they could be anything you want that could be containers And we have four unique IP addresses between those things now going back to what I said earlier part of the reason We're looking for this unique IP is it makes a lot of things simpler in terms of What your application has to understand right again with one one address when I you know one interface You'll leave one place to listen What's also interesting about that is if you end up having to roll your own for things like service discovery You know very easily which IP address to expose, right? So long the short of it is across a container network We're looking for a flat address space fully interconnected with unique addresses and that's the ultimate goal there and This is related to CNI in that the CNI plugins the way they actually work today is When garden comes down through the CNI adapter to actually bring up a network interface on the container we're pointed to the network namespace of that container and We're allowed to essentially create all the artifacts that are required to bring that network or bring that container on to a network So the address management pieces of it and the technology that are used are both baked into CNI So this is the first primary extension point that we have for a cloud founder networking Okay, let's talk about policy. It's gonna be a few slides So there's two basic roles that we think about types of people that are sort of users of the policy work There's the developer who just wants to push their app and have it be run for them on the cloud They want this to be connected to the other apps that it needs to talk to maybe microservice applications directly Without going through the router they want that to be easy to use and then there are operators they're administering the cloud finder in the network and they probably have concerns about security and compliance and So what the goal of this effort is to figure out how to make this easy for developers to just get a simple CF push experience the self-service thing that's made CF really easy to use for it for a developer But also not scare away the people who are concerned about security So you can think about you can think about policy This user experience issue on a couple different axes. One of them is the sort of implicit versus explicit access Do you want to default allow? connectivity between applications or do you want there to be a default deny and The user or operator has to explicitly say which application can talk to which other application or which Cloud Foundry space could talk to some other Cloud Foundry space There's also the axis of granularity. Are you talking about apps? Are you talking about spaces or are you talking about orgs in terms of the the units that when you discuss policy? So we can imagine Some simple implicit Default policy, which is that every single application within an organization Cloud Foundry or can talk to it to everything else in that org. There's no restrictions At all except that you can only talk to things within the same org and then sort of going down the granularity scale You could imagine the same kind of thing but only within a space and that spaces themselves You couldn't talk to things outside of the space Except via of course going out through the the router and coming back in the the way it currently works And then going down again in terms of granularity You can imagine that instances of a single application could talk to each other, but there's no in inter app communication And that's all sort of implicit. There's no additional user experience because these rules are just applied automatically and then in terms of going on the more explicit side of things you can imagine that a user might be able to write a rule that says Org A can talk to org B where all the applications in org A can talk to all the applications in org B and Again going down the granularity scale Maybe a space can talk to an org or all the apps within a space can talk to all the other apps in another space and Then you can just imagine an app to app granularity where the default is Don't allow any communication unless I explicitly say app a can talk to app B And then finally in terms of the version 3 cloud controller API There's this smaller unit than an application which is a process or process type And so you can imagine maybe writing rules like a process type can talk to another process type And finally we can add on top of this the concept of ports and protocols So there's this question of what should cloud foundry support like what should the user experience be? What should you be able to do as a user operator? And so it's going to be some Area of this graph. We don't really know some collection of these things covering some Region will be the thing the cloud foundry supports at least out of the box And we don't know what that is yet, and we really want to hear from people if they have opinions about this sort of thing Well, you'll see a slide at the very end about office hours But we're gonna have office hours tomorrow afternoon be great if you came and told us what you thought about this kind of stuff The one thing I did want to bring about this is one of the initial proposals that we had Which was kind of a middle ground was that by default Applications within the same space would have connectivity and then policy controls would be on top of that It's it's kind of a middle of the ground like approach. It's not Agree to it's not kind of where everybody wants to be But that's just kind of an idea of where you could start with something Which is if I push two applications they can talk as long as they're the same space as soon as you cross spaces They're gone. So just that's an idea of like an implicit control point just a concrete way So in addition to those sort of high-level concepts, there's this question of like where does this stuff actually store That does a user put this into the cloud controller API is there gonna be some separate networking API that controls this stuff Maybe they express it via the service model for Cloud Foundry with service bindings We've talked about and thought about these approaches. We haven't made any firm decisions yet But again, we're looking for feedback on this So here's an option or an example of how you might do explicit rules The top line kind of is the most generic case where I could write some CF command line thing That's allows traffic from some source application in a space in order to another destination application Over a space in order using a particular protocol in a particular port. So just to make this a little more concrete I push a web UI app and an inventory app and a checkout app that makes some kind of shopping cart experience And I allow explicitly traffic from the web UI to the inventory app And I allow traffic from the web UI to the checkout and then I allow traffic from the checkout to some credit card sensitive PCI thing that's maybe in a different space and Those three allow rules together allow me to run my complete web e-commerce application There's a question as far as who should be allowed to write these allow rules If you're maybe if you're a space developer in both the space that contains the source and the destination That that's enough of a permission. Maybe you need some kind of a particular network admin scope Those are open questions as far as the the permissions to do this Another iteration on this might be something that's a little more self-service when I push an app Perhaps I can expose a named endpoint and I can say this inventory app is available on port 443 Over TCP and I'm giving it this name and then when I push the web UI app the thing that consumes that Inventory back and I say bind it to that bind it to the inventory and That would be a perhaps a nicer user experience would make it a little simpler to do microservices with CF push so again, we're kind of like Kicking these ideas around and we're interested in people's feedback about what they might want to use And we're certainly not seeing these as the only two options Finally this gets a little more complicated with Cloud Controller version 3 as I mentioned earlier You maybe use specify process types as opposed to applications And there's a question again in terms of like implicit versus explicit whether there should be some kind of default Allow rule all the processes within an application should be a little able to talk or not So to sort of summarize all this stuff in terms of just kind of like commentary on this With a default deny rule you're gonna have to write a lot of explicit Allow rules in order to set up your application and doing stuff Bidirectionally if you have a callback between some front-end and you've got a if this and that application that calls you back Gets a little messy But with the default allow rule if everything's allowed to talk to each other Then maybe the security guys are gonna get a little concerned and they're gonna say what wide these things Why can they talk by default? I want to know explicitly where traffic is going So this question of usability versus security we'd like some some input from people to hear what you think Okay, I'm gonna keep talking about policy enforcement. So There's this overall goal with Cloud Foundry We want to be able to provide some included batteries that enforces policy for you But you should be able to replace them. So the open source thing should just work But if you have some proprietary Network fabric that you want to use That's okay, too So the open source solution that we're gonna we think we're gonna begin working on we've done some work on it already and The flannel release is the the foundation of this is a VXL and overlay network so what this does is it takes the traffic from an application and encapsulates it and Adds a tag to the front of that packet before it goes out on the wire and that tag has something that represents the identity of the application and What this lets us do is it separates the identity of an application from its address on the network This pays dividends with policy enforcement. So to make this a little more concrete for those of you Who are not already networking experts It kind of looks like this you've got Your application data is got its own IP header its own UDP header, but it's encapsulated within the body of An outer packet that has a tag on it and so each cf application would be assigned some unique tag and Then before the traffic leaves the container host before it leaves the Diego cell It would be tagged with its identity and then when traffic arrives at a container It can be filtered based on the identity of the source of the packet Getting a bit more into the details VXL has a few different places where we might put this tag and It has a 24-bit thing called a VNI. It has a 16-bit group policy identifier Which is a very new extension to the specification Depending on how many bits you allow for the tag that puts an upper limit on how many different applications You can deploy to your cloud funder using this approach and you we can imagine a few different ways of encoding the identity of the application into the tag on the packet and Perhaps splitting it based on some concept of a tenant versus an application that's owned by that tenant to support multi-tenant cases We're not Set in stone on this stuff again. We kind of want to hear from people if they have opinions Just a couple of side effects that like people may need to be aware for this kind of thing Is that if you start to actually in cap everything and it's the amount standard end cap All right Then you have some interoperability issues if you end cap everything leaving the container Then you have to basically decap on the way out which means you had another filter for you actually leave your cloud foundry environment If you have existing VX LAN overlays and VNI's are actually reserved depending on how you do routing, etc You may have to you know, segregate some of those things and actually isolate those There's a lot of little things about this proposal that people need to be aware of And but it's something that would be there out of the box. So it's it's a starting point So to give a little bit of a overview of how this might Work with cloud foundry you can imagine the user or perhaps cloud controller itself driving some Policy brain some central controller that that stores the rules the intended rules that the user has about which apps can talk to other apps Or spaces can talk to apps or whatever you want and that there's There's a component which we're starting to call net man that Stays in sync with that with that brain and has the the local copy of those rules that are relevant to the cell That it's running on and and that net man program is going to be actually responsible for doing two different things one of them is driving CNI plugins to set up network connectivity in the first place and it's also going to be responsible for Driving policy controls on the local cell But because we're we've got these three different API boundaries This allows for extensibility if you want to replace these batteries with something else You can do that by making use of the CNI API of the policy API that we specify that net man drives and by the the brain itself and this type of solution should we hope allow for extensible different types of custom network setups for People who are not content with the the batteries that we're building last major thing is a discussion a little about discovery so One of the things that's interesting about when you build Container networks and you have the ability to identify sources and targets when you declaratively when you actually push your applications is you still have the problem actually finding them at runtime and Discovery is a it's kind of a big deal and most of the solutions that exist today Provide some level of support to help application developers find that stuff out either through the environment or through DNS So We've kicked around for a little while the notion of trying to use DNS for some of these things It hasn't gone anywhere. It's not currently on the road map But we're looking for lots of opinions from people right on where we should go One of the options was take those names that we were talking about for explicit policy before where you labeled it back End and you gave it the the port etc make take that bind name and use that something you can reference in DNS Another approach is we do nothing for you since you have a single interface You can get the IP yourself, and if you want to roll your own services cover you certainly can Or if you're using a library in your application that allows you to essentially register You can do that. Those are options But Yeah We're looking for some input here. I personally feel like discovery is something that is kind of important having a network without being able to find Your target is really only part of the solution So if you have feedback come on in Or we were open to it And then yeah, we're at the end. I mean, this is really the call for action Right, please come on and get involved. We've been Around now since the beginning of the year. We have a slack channel. We get a little bit of action in there But we would really really welcome some people kind of expressing your use cases and what you need out of this We have some code that's already out there today We have two releases an old one and a new netman release that allows connectivity so you can play with it deploy it and explore it and You know finally, you know Moving through the five things that we're thinking about right now. That's not an exhaustive list But we're looking at that being enough to get started. So if that's not enough to get started for people What are we missing? Right? That's kind of where we are Yeah, and then More stuff if you're interested in container networking, we're having office hours tomorrow and there's also this recently added thing about C9 OCI Not really sure what it is, but I'm going to attend. So thank you Question front Right. So yeah, so so the the two points were the things that this Yeah, that's what I'm trying to do the two points that we made from the question were one of the things that's really missing is IPv6 support Noted, I think that's a larger effort within Cloud Foundry. I think the overlay them, you know within the container to container I don't see why a network provider the plug-in could not support IPv6 if they chose to But outside of Cloud Foundry like within the fabric Cloud Foundry right now. It's IPv4 So I think it's a larger work effort The the other piece was around multi data center and extending the network and again I think that's where the extensibility comes into play where the network that you have between the two depending on the technology That you use you could federate in bridge you add your own routing rules to do that So if what's battery, you know batteries included I don't think we're ever going to try to support all the enterprise use cases You have multi data center DR and everything else and that's really why those plug points are there right? We really want to be able to get to that now multi Cloud Foundry is kind of an interesting one Right multi Cloud Foundry is where you have you may end up at the point where you have two policy engines And I don't know really how to solve that problem So that'd be an interesting conversation to have during office hours or if you have any ideas that would be I think that'd be really welcome We'd hear those Any other questions? Yeah, yeah, this this is completely separate from router This is that's actually one of the things that we should have probably talked about the way the router works today Is the components within Cloud Foundry actually advertise routes what that means is they constantly broadcast They say this name is at this address at this port and the routers watch for those advertisements And when a request comes in they will forward it to that post import That's true for TCP and for HTTP right with with a couple differences in the flows, but that's largely what happens The container networking it will will allow you to get into the container through the router still by advertising routes But what's more important is now the containers can talk to each other without going through the router It's completely separate from the router So for route services the stuff that coming from the router still applies You have the ability to receive the request modify the request stage the request But you don't have access to what's happening for container to container communication only what's coming from the frontier Right like limits on the routing on the routing tier from the front end only apply to what the router receives and sends out Right, but again if you don't want to use container networking right for the back end You expose a route and you go through the front door, but then what you end up doing is you have You pay the price of coming through that round trip So you now get to choose right you have flexibility that you didn't have before which is a win And if you choose to do your own rate limiting inside of your application or your container you have that ability as well Yeah, so one of the things in the battery So the question is exposing container not just other containers. That was basically the question, right? Yeah, so one of the things in the batteries included model that's being proposed right now is the IP addresses for the container are Actually accessible from outside the container. I don't think that's something that necessarily You can rely on in all environments because as soon as you get a plug it in there those IPs may not be accessible But again, it's going to depend on the on the connectivity the batteries in the batteries included version would do that anything else Thank you