 Welcome everybody to this OpenShift Commons briefing, we're really pleased to have Makaya Masters and Mike Barrett here with us today to talk about external routers and everything under the hood that it takes and took to do our new F5 integration. With that, I'm going to let Makaya and Mike both introduce themselves and kick it off. Great, thanks Diane. As you guys know, the under the hood is a series that we've come back to the Commons periodically and we try to pick a specific piece of the puzzle, the technology and explain it a bit further. The first ones we went over the entire platform just to give everybody an introduction to it. So now moving forward we'll just spend the time on a very specific part of it and answer any questions that you might have on it. As Diane mentioned, I'm Mike Barrett. I'm one of the product managers over here on the OpenShift product and I've invited Makaya Masters with me. Makaya is kind enough to join me at customer sites and around the world at different locations. He's one of our star engineers. Makaya, why don't you say a little bit about yourself? All right, I'm Makaya. I joined Red Hat about three years ago so I was working on OpenShift V1 and V2 and I was among other things responsible for the routing daemon in V2. So with V3 we have a similar approach. I implemented the initial implementation of the F5 integration for V3. Now that is now under the responsibility of the networking team for OpenShift V3 but I wrote the initial implementation so I'm going to talk about the goals and some implementation details of the V3 work. So before we get into that, just a reminder, we are running the OpenShift Road Show. It has been pretty successful with the people that have joined so far. So if one of these hits your area, please visit these slides will be available. It's a pretty long link or just Google OpenShift Road Show and it will be the top two hits that you get. Definitely register and show up. It's a great place to see the community face to face in your local region. So ISV's partners, customers, people who are hobbyists, everybody's coming to these things and pitch free food and beer at the same time. So before we get in just a fresh reminder, this is sort of the marketing architecture, if you will, of the solution. Today we're going to be digging deep into the routing layer and if you think about, if you remember the previous time we had a under the hood, we have a couple of networks involved. You know, when you start dealing with literally thousands and thousands of containers and you want to give them real IP addresses and that you want them to act like first class citizens on that network to be able to go into your firewalls into your reverse proxies into all the layers that you sort of surround applications with. You have to provide some IP addresses and there's a couple of ways to do that. You manually, you disengage the software defined network in OpenShift and you take on that on its yourself, which is not the popular road to go, or you establish an overlay network and OpenShift is using OpenV switch. So these pods are getting a network from that OpenV switch solution, that open flow solution, which means it's on almost that private network that is not routing itself those specific 10 dot IP addresses out to the world. That's flowing through a routing tier and that routing tier is connected to your service declaration and it involves an HA proxy. That is the default implementation, but we understand that you'll want to change that if you are heavily invested in hardware routers or if you have different needs and requirements on getting traffic in and out of your environment. You're going to be changing that layer. You're going to be messing around and changing configuration files and understanding the flow and understanding how we do that really helps you. And the perfect example of this is the F5 router. It shipped last night officially, so it's out in the world under OpenShift 3.0.2, so you guys can totally download it and start using it right now. It's in the official documentation now as well. So Makai is going to take us through that solution and take it over Makai. Alright, to begin, I want to state the general high level goals for the F5 integration work. Primarily, we wanted to enable integration of OpenShift with existing F5 big IP infrastructure. As Mike said, organizations have investment in F5 infrastructure, hardware and licenses. And we understand a lot of people are going to want to integrate OpenShift with that F5 infrastructure. So, within that context, we want to enable clients to reach things, web applications, services. We want to allow unencrypted HTTP traffic using TLS transport layer security as well as non-HTTP traffic using TLS SNI server name identification, which can be used for TLS enabled message queue technologies or databases or any technologies that are TLS enabled. So, another way to put this is that we wanted with F5 integration to have feature parity with what we offer with the template and HAProxy router solution in OpenShift V3. Secondarily, we wanted to provide an alternative to the template HAProxy router that could be used as a reference when implementing integration with other technologies that rely on a CRUD, a create, update, delete-based API. Citrix Net Scaler would be one example. OpenStack Elbas tentatively would be another example of a CRUD-based design where the F5 integration may serve as a good reference to implementing a similar solution, probably a better reference than the template HAProxy router. Next slide, please. So, I need to mention one thing about terminology. Initially, because we had the template router, the HAProxy router, we were calling the F5 solution the F5 router. More recently, we've been calling it the F5 route synchronizer to better reflect what it actually does. If it's not routing connections, it's synchronizing route configuration in OpenShift with an F5 big IT system, which, and F5 does the actual routing. The route synchronizer doesn't route connections, doesn't route packets, it just configures F5 to route connections and packets into OpenShift. The route synchronizer uses the OpenShift master as its source of truth. It gets all its information from there. The route synchronizer itself carries minimal state, just credentials to connect to F5, some basic caching. It doesn't store any configuration. It does not rely on F5 big IP for state, although it will query big IP when it's doing updates during initial syncs. It doesn't rely on big IP or on F5. It's getting the data from the OpenShift master and using that to configure the F5 big IP host. So, the route synchronizer itself is largely stateless. It configures the big IP host using the F5 iControl REST API with a couple exceptions I'll note later on. So where was I? Yeah, you're in the last one of the solution slide. Update external. Right, right. Okay, so I was saying we update F5 big IP using the F5 iControl REST API plus some SSH commands. I'll talk about that later. I think next slide, please. Okay. All right, good. So during development, I used F5 big IP 11.6, virtual edition in Amazon EC2. Because we're using the F5 iControl REST API, we do require F5 big IP 11.4 or later, which 11.4 is when the REST API was introduced in F5. We also rely on SSH access to F5 big IP to upload keys and certificates for routes because unfortunately the REST API does not provide a means to do so. You need to upload the key and then trigger a REST API call to incorporate that key into an SSL profile into F5 configuration. Mostly it's REST. There is that one aspect where we need SSH access to the F5 host. Next slide, please. So the route synchronizer itself runs in the pod on OpenShift, typically on a node labeled as infrastructure. That's not a hard requirement, but usually you would want to have dedicated nodes labeled for infrastructure components such as this. The pod, it's just a regular OpenShift pod managed by a replication controller which monitors health, makes sure the pod's running. If it dies, the replication controller restarts it. If the node goes away, the master will reschedule the pod on another node. Remember, the route synchronizer itself maintains minimal state, so we can do that if something bad happens. If it dies, if the node goes away, we just restart it on another node. To get the information from the master, the route synchronizer uses Kubernetes Reflector, which is built on the Lister Watcher interface to watch for updates. This enables the synchronizer to perform an initial synchronization of configuration. When it starts off, it gets that configuration from OpenShift. Then as routes are modified or routes and services are created, deleted, using this interface, the route synchronizer gets these updates and updates F5 Big IP dynamically using the REST API interface. So it updates the F5 Big IP, creates updates and deletes F5 pools corresponding to the OpenShift services, and creates updates and deletes F5 local traffic policy rules corresponding to OpenShift routes to route incoming requests coming into F5 to the appropriate pool and then to the appropriate pods on OpenShift. So while I'm here, I want to make a couple of technical side notes just for completeness and correctness. So like I mentioned, we are using SSH and SCP for a couple of things here. We're not strictly using the REST API because of limitations in the REST API. And second, I want to mention while we use local traffic policy rules in F5 for plain and encrypted that is edge terminated where F5 termates encryption or re-encrypted where F5 decrypts and re-encrypts connections, we use local traffic policy rules for those connections. However, because of limitations in F5, TLS pass-through requires special treatment. So with pass-through, you're passing the encrypted data directly through to the pod. The F5 local traffic policy feature was implemented before SNI was popular, so it's not supported by the local traffic policies. And we have to have a custom iRule. The iRule parses the TLS client hello handshake packets for the SNI server name, looks up the server name in a data group, and routes that way. So pass-through routes are handled a little specially. I just want to mention that for completeness. Are there any questions on that? Or should I take questions at the end? Yeah, that'd be best. Okay. All right. Next slide, please. Okay. This diagram shows what I just described. Upper left you have the master. We're connecting actually to the API server. The F5 router is running on an infrastructure node connecting to the API server, which is part of the OpenShift Master. That's how it gets route and service information. And then it's connecting to the F5 Big IP host. So the route synchronizer is acting as this intermediary between the OpenShift Master and F5 Big IP. Again, the master is the source of truth. The route synchronizer itself does not store any state outside of its own configuration and some caching to avoid repeating too many REST calls. Next slide, please. So once the route synchronizer is configured, F5 Big IP. Next we need, I want to talk about how connections and packets get routed to applications or services on OpenShift. So a client initiates a connection with F5 Big IP to the virtual IP, the VIP of the virtual server configured in F5. The F5 Local Traffic Policy Rules or the Custom Eye Rule use the HTTP host header using virtual hosts or the TLS-SNI server name to map the connection to a pool corresponding to the OpenShift service. The pool contains members corresponding to pod endpoints. And I want to be clear about this because it is a little confusing. We do not route through services. We do not route through CUBE proxy with this solution. We're relying on service definitions at configuration time in order to map, to figure out how do we map this virtual server to this route. How does it map to the pod ports or the pod IP address? Services help us in figuring out that configuration. But once the route synchronizer has configured F5 Big IP, we're not going through the services. We're not going through CUBE proxy where the pool members will be pointing directly to the pod's SDN IP address and port. And that means in order to route to these pods, we need to go through the OpenShift SDN, which takes some extra configuration. You have to set up an IP over IP tunnel to a ramp node, what we're calling a ramp node, which is just a node normally you would disable scheduling. So it's just a node that's only there to be on the SDN and act as essentially a gateway. I'll talk a little bit more about this on the next slide. And for high availability, we would recommend creating multiple ramp nodes using a VIP configured in OpenShift, and OpenShift's IP failover to make sure that even if one ramp node dies, another one will come in place. And so this tunnel that F5 is using to get to the SDN, there should be an endpoint. The OpenShift end of that has replicas of the multiple ramp nodes in case one dies. Well, that peer endpoint will be highly available. Next slide, please. I'm just looking at the chat. Passive termination is supported. I was just trying to explain we had to use a different approach to implement it, but it is supported with our integration solution. I hope that answers your question, Vier. Okay, so looking at this diagram and starting from the box in the upper left, we go clockwise. We have the client connecting to a VIP on F5 Big IP, and this is a normal TCP connection, HTTP, TLS. F5 Big IP will pick the appropriate pool and some member from that pool, which corresponds to a pod on the OpenShift SDN to route the packet to the pod. F5 Big IP needs to route through its tunnel to a ramp node. And so at this point, this is an IP tunnel. So we have IP encapsulated an IP, that IP and IP packet goes to the ramp node, and the ramp node decapsulates it. So now we have a packet with the pod's SDN IP address and port, and then the packet's routed from the ramp node on the SDN or over the SDN to the appropriate node and the appropriate pod. Let's see. Yes, the pod, IP, endpoints, and routes. Routes and services, that information is all stored by the master and at CD. It's not going directly to at CD. It's doing a watch on the resource in the OpenShift master. So we do have a nice interface. It's the same interface, the template and HA proxy routers use to get updates to routes and services. Next slide, please. So to use the F5 Routes Synchronizer, you need to have an F5 Big IP deployment configured with virtual servers for HTTP and HTTPS. Strictly speaking, you can, we don't require that you have both enabled, although typically you would have a virtual server for which you would want to support both types, encrypted or unencrypted. You'll need to have a VIP or if you prefer multiple VIPs for one for each, or even multiple for each virtual server. So you need to have your v-servers and VIPs configured. As I mentioned, you need to have this tunnel configured in F5 Big IP. And on the OpenShift side, you'll need to have these ramp nodes configured, one or better multiple with a VIP and IP failover to provide high availability for that endpoint. This is all documented in our official documentation. So we have the TMSH command and all the information on how to set up, configure F5 for the tunnel on F5, how to configure the ramp nodes. That's all documented. I won't go over it here in detail. Once that's all in place, you can use the OADM command, the same command, or similar to what you would use to start the template or HA proxy router. You'd use a similar OADMN command to start the F5 route synchronizer. And you need to specify the address and credentials for the F5 Big IP host to the location of the private SSH key with which the route synchronizer will connect to the F5 Big IP host to upload certain keys for routes. You'll need your OpenShift credentials because you're creating this OpenShift object so you need to authenticate with the OADM command. You need to specify a service account under which the route synchronizer should run. Typically, this would be the router or possibly the default service account. Next slide, please. Some challenges in developing this, the F5 integration. The first was that the F5 iControlRest API is relatively new. It's much newer than the F5 SOAP API or F5 itself. And as one might expect, the documentation has a lot of gaps still. There are a lot of useful resources online. People in the F5 IRC channel are really helpful. But it still took a lot of trial and error to determine the correct REST API calls, required parameters, and responses. Another challenge was handling TLS pass-through. And again, this is routes where the route itself... The route itself does not specify TLS certificates. We're just passing traffic through, encrypted, coming into the F5, pass it through to the pod without decrypting it. And we do support it, but it did take some extra effort. So we had to create a custom iRule that parses the TLS client hello handshake. So it's doing... It's kind of parsing the package, doing some low-level work there to figure out how to route connections of the TLS pass-through type. So that required some custom TCL code. The route synchronizer will add that custom iRule automatically when it first starts up if it doesn't find that rule on the F5 Big IP HUS. So it will create that rule. But that was a challenge during development to figure out how to deal with that type of connection. The biggest challenge was in figuring out how to route connections to pods on the SDN. Routing through Qt proxy would have been inefficient and practical for several reasons. And without a way to route to pods, of course, we can't really do anything. So I want to say many thanks to Ram Ganathana or Ram R and Rajat Chopra who came up with the ramp node and tunnel approach to providing connectivity between F5 and pods on the SDN. These are a few general techniques that I use during development. These are not things you do in production. These are just some things that come to mind when thinking about the development process. So first thing is rather than launching and deleting the pod, the application controller, the deployment config, the secret containing the SSH key, creating that. And then you can do that with the OADM command, but then you have to clean it up. That's a little much to do every iteration. I found it was a lot easier to troubleshoot as I was developing and iterating quickly just running the binary directory on an OpenShift host in my development environment with log output going to my terminal. And as I mentioned, it took a lot of trial and error with curl commands against the iControlRest API to work out the needed API calls. I also used curl against the F5 VIP to test the configured routes to make sure policy rules were correct, that the certs, the F5 was using the correct certs. I tried to connect to applications, that sort of thing. I used the Python and JSON tool to pretty print JSON. It's on standard on Rails systems, so it's pretty convenient. First I was using that to make it easier to read through API responses, and later I realized it's really nifty because it validates your JSON and can find errors in the JSON I write or that my tool is creating. I did find that Python's JSON acceptor accepts some JSON but F5's parser will not. It turns out Golang can generate JSON that Python accepts, but F5 rejects, which necessitated a workaround, and that was one of many gotchas. A third challenge? Well, this is not so much a challenge. A third big thing was, well, in OpenShift and Kubernetes generally, testing is really heavily emphasized. Having a ready set of example services and routes with example certs was really helpful, so kudos to Paul Weil who developed those, the Hello Engine X Docker examples for his work with the template router. This development progresses as the code base becomes more complex. Having the test color, unit tests, integration tests is really helpful in identifying mistakes quickly. We have roughly 75% test coverage in the standard tests. That run with every build. These tests essentially create a mock F5 big IP, implemented as an HTTP test server, recognizes the same routes, spits out the same JSON as F5. We also have integration tests for testing against a real F5 big IP host. One thing I maybe could have done better would have been to be a little more proactive in writing tests, writing test cases before or at least while writing the code that was being tested. First of all, to reap benefits of testing sooner. Second, to reduce the failing of thread when you have this big pile of code and you have to write a big pile of test cases for it. These are pretty standard recommendations for software engineering. Always bear repeating though, tests are good. I think that's it. Okay, lessons learned from the V2 routing daemon implementation. Right, the V3 work is actually very similar to the V2 routing daemon. Conceptually, the V2 is a little more complex. We had the broker plug-in and the separate routing daemon. It was a little more cumbersome, but just having that knowledge of how we can integrate with F5 and V2 is very helpful. Having some experience with the REST API interface. We have a similar approach. Roughly you can think of routes that are sort of like aliases and V2 services are sort of like a domain or an application V2. Not exactly, but follow the similar approach. The SSH and SCP thing was kind of something, it would be nice if there were a way to do it in the REST API. Using SSH feels almost kind of like a workaround for this gap in the REST API. That was something we had to figure out in V2. That's something else we carried over into V3. F5 is the same, so we're using the same technique in V3. I can't really think of more specific lessons learned. Does that kind of answer your question, Nicholas? Okay. Diane, we have documentation that should be being pushed out with 302. There's documentation, the official OpenShift Enterprise. I believe it's also in the origin documentation for setting up the ramp node in the F5 tunnel. We have documentation on launching the route or the route synchronizer. That's all on docs.openshift.com. I can give you a more specific link if you like. Are there any other questions? Using different load balancing methods, algorithms. F5 provides that functionality. Right now, the route synchronizer doesn't make use of it. It hardcodes the, I believe, the round robin load balancer method. That's something we could definitely add if you open an RFE. That's definitely something that would be feasible to implement in a future release. Okay. Interacting with other modules other than LTM. We don't do anything with GTM or with AAM. It's pretty much just LTM. If you have anything specific use cases signed, feel free to suggest and we can look into implementing more, some more features to make use of more features in F5. Okay. Nicholas shoots the configuration sync and commit. That's not in the current release. It's something it shouldn't be difficult to add. But that is missing with the version in 302 right now, sorry. Follow the number of connections to specific applications. That's not something we support with the route synchronizer right now. I don't know offhand how we would implement that using features in F5. If you know what features in F5 we would use to implement that, let me know and we can talk about use case and how we can implement that in a future release. Are there any other questions out there? Micaiah, the two components that actually run on the Kubernetes cluster. One is the synchronizer and the other is the ramp node. Is that right? Yes. I wouldn't phrase it quite like that. It's really a regular open shift node. Typically you would mark it unschedulable so you won't be putting pods on it. It's only there to act as a gateway into the SDN. I wouldn't think of it as a component. I guess you can think of it that way. But yeah, the ramp nodes, ramp node or ramp nodes and the route synchronizer, those are the two components. Outside of open shift itself, the master, the other nodes with your applications, and F5 itself, the route synchronizer and the ramp nodes are the two additional concepts. You could say that you need to pull in for F5 integration. Okay, does the route synchronizer run on a ramp node itself? Well, you could run it on the ramp node. We're recommending running it on a regular in-front node and having the ramp node be dedicated just to acting as basically a gateway. So it won't have high resource requirements. It's just shuttling packets between F5 and pods on other nodes. I'll have to check with ROM or Rajat. I think there were some reasons we thought it would be better not to put the route synchronizer on the ramp node itself. I don't remember offhand things or performance reasons for that though. You certainly could, but it's not what we recommend. Atomic. I'm not sure exactly how we implement open shift SDN. I think it would work the same way with Atomic. All we need is a host that's on the SDN where you can configure, that you can configure as an endpoint for the tunnel and that you can assign a VIP in open shift. I don't see any challenges Atomic would pose, but I'm not certain right now. Right then. So hopefully we've answered most of your questions, but if you haven't and you have another question, you can always jump on the IRC channel and track Mike Makaya down or Mike Barrett or myself and find us and we'll try and get your questions answered. We look forward to hearing a lot more from everybody on different use cases and different features and functions of F5 that we might also incorporate in new and upcoming releases. So thanks again. We'll send out a copy of the presentation and we'll hook it up to the website at openshiftcommons.org slash briefings along with this video probably tomorrow morning when the site gets rebuilt overnight each night. So thanks Makaya and thanks Mike for bringing this topic and for doing another under the hood. We always look forward to these and we really appreciate you taking the time out from your day to day.