 All right, let's go ahead and get started. My name is Mike Heath. I work for the LDS Church in Salt Lake City, Utah. Just real quick, because I'm being recorded, my employer makes sure I say that everything I say is my own opinions, not necessarily the views or opinions of the LDS Church. So just to give you a quick introduction of what we're doing, so we've been running Cloud Foundry for about three years now. We've had it in production for a little more than two and a half years. So we've been on this ride for a long time. We're currently operating three Cloud Foundry instances. One of them is on the campus of Brigham Young University in Prova, Utah. It's our primary deployment where most of our applications are running. We also have an instance running in our main IT offices in Riverton, Utah. It's about, I don't know, 9 or 10 miles south of Salt Lake City. And as an organization, we're still trying to figure out Amazon. We have Cloud Foundry running up on AWS East. We're not using it heavily. But as we start to adopt Amazon more and more, Cloud Foundry is really the front door that our developers will use to deploy their applications to Cloud Foundry. And we're hosting about 450 applications total right now, somewhere between 20 and 40 of those are production applications. And just to give you a little idea about our team, we have engineers, two and a half software developers. I say a half because our manager spends half of his time doing management, half of his time doing development work. We have one UI engineer. He's actually a .NET engineer. Because Cloud Foundry doesn't support .NET yet, he's kind of migrated over to being a UI engineer. We have an operations guy. And then our team is also responsible for our enterprise load balancing and caching. And so we have an engineer that is responsible for that. So let's dig right into this. The whole point of this presentation is to talk about routing into Cloud Foundry, how the Go Router works, and how we've been able to run a Cloud Foundry without the Go Router. So I think this is a pretty typical architecture. Granted, it's fairly simple. It doesn't have all the Cloud Foundry components. But it has all the main components of an HTTP request going from a client into an application. For us, we're using an F5 local traffic manager in LTM. In other cases, you might be using something like HAProxy. If you're running on Amazon, you might be using ELB. But yeah, for us, we're using the LTM there. From there, it load balances traffic into the Go Routers. And then the Go Routers then forward the request into your application. So let's talk about these components a little bit. So the first thing, you have this load balancer that sits in front of the Cloud Foundry. In our case, like I said, we use this LTM. It also does all of our TLS termination, all of the certificate management for the different domains that we are running on Cloud Foundry. And we didn't go out and deploy an F5 LTM just for Cloud Foundry. This is something that's fairly pervasive in our organization. And so we're just using the existing standard that we have, the Go Router. It's part of Cloud Foundry. And the Go Router is really part of the magic of Cloud Foundry. One of the really cool things with Cloud Foundry is you push your application up. And within a few moments, it's actually running. But not only is it running, but you can easily route into your application and start sending your request immediately. You don't need to go talk to the DNS team and the network team and all those things. So what the Go Router does is it dynamically updates its load balancing tables based off of where the application is actually running. So when you start the application, when you scale it, the application crashes and it gets brought up somewhere else. The Go Router gets updated and notified of all those different routing changes so that it knows exactly where your application is and where it can run the traffic to. And the Go Router, it's more than just a load balancer. It also provides logging and metrics, which is important. And it also scales really well. You can scale the Go Router horizontally and vertically. And it works really well that way. But in the three years we've been using Cloud Foundry, I've always felt like this. So I actually stole this meme from the Garmin guys that presented yesterday. I think it fits really well to what we're talking about here. We've already got this Enterprise load balancer in front of Cloud Foundry. And right behind it, we have another load balancer. It's not necessarily bad, but we've already invested a ton of money into these FI of LTMs. And it's one of those few Enterprise products where I feel like, yeah, it costs a lot of money, but we get a lot of value out of it. So I don't feel so bad about giving them a lot of money because the LTMs are fast. They're incredibly flexible. It's really easy to add and drop new virtual IP addresses on the LTM. They're also programmable. They're really cool in that you can take a script and upload it to your load balancer and how that script run as traffic comes through the load balancer. They have all the reliability that you come to expect from any Enterprise product with automatic failover and things like that. And one of the features that the developers and operations people in our organization have really come to love is support for these things called custom health checks, where at the load balancer, you can say, hey, here's a URL to go hit. Depending on the reply from that request, it determines where the application is actually healthy or not and whether it should be receiving a traffic or not. And that's a feature that our customers have come to expect. The LTM also has support for REST APIs. So you can remote control the config on the LTM. You can automate it programmatically and things like that. So this is the current architecture, but I think what we'd really like it to look like is like this, right? Let's take the go routers out of the picture. We've already got a capable load balancer. Why don't we just let it do all of the load balancing into Cloud Foundry? And I think the question that comes up is why does that even matter? I mean, the go router already works today, right? What's the big deal? And I think the big deal for us is that everything that sits between your client and your application affects the reliability of your application. And that's the critical path into your application. So we have a customer that has an application running outside of Cloud Foundry. And they go and run these massive load tests that take like a whole weekend to run. And they'll get very few errors or no errors, not running on Cloud Foundry. But when they do run on Cloud Foundry, they get like a few hundred errors. And this is a few hundred errors out of like 100 million requests. So for our team, it's like, who cares? It works just fine. It's just these corner cases. And for them, they're comparing to the other environment. Like, hey, this is important to us. And so then it becomes our responsibility to try and troubleshoot what's going on. And other issues, like I said, the custom health checks, that's something that our customers have come to rely on. And another big thing, so right now, most of our IT operations, as far as the data center and things like that, happen in one data center at BYU. But as we're expanding out and embracing the cloud and moving to AWS, supporting global load balancing has become a really big priority for us. And it's something that F5 provides for us using a GeoDNS solution. But it's a lot harder for us to integrate with Cloud Foundry when you've got the go routers between the LTMs and Cloud Foundry. We could still do it, but it would be a little bit easier for us if that wasn't there. Another issue that we've had is configuration mismatch. So specifically, we've had an issue where we adjusted the TCP timeouts on our LTMs, on our load balancers, and didn't make the same change on the go router. And that caused a lot of interesting behavior. So the Layer 3 security, it's actually a pet peeve of mine that we have a lot of applications that depend on this. But whether I like it or not, it's reality. And that we have applications that expect traffic to be coming in from a specific address, where they've had a lockdown through firewalls and other mechanisms, the route that their traffic comes in from the public internet or from some VPN or whatever it might be into their application. And with the go router in the middle, we really can't do that later through security. The other thing is, if we take the go router out, things are faster. There's less latency there, right? There's one less component to route through. But if you're going over the public internet, it's really not that big of a deal going through the go router. In fact, it would be almost impossible to measure whether or not the go router is actually running, coming in over the public internet. But for a lot of our customers that are building out large microservices, where they're making hundreds or thousands of REST calls, that performance has an impact on what they're doing. So what we've done, we've built a component that we call no router. And I think no router in a lot of ways is a testament to the awesome architecture of Cloud Foundry and that we're able to take a component of Cloud Foundry, rip it out, and replace it with a different implementation. So what no router is, is it's a Bosch deployable agent that runs on a VM. And it collects routing information the same way that the go router does. So today, we're using NATs. It's a messaging bus user with Cloud Foundry. And we collect all the information saying, hey, an application started up. An application scaled out. An application crashed and got brought up on another VM. We capture all of that information. And then we use that to dynamically update the LTM using the LTM's REST API. Currently, this doesn't work with Diego. Diego has a new API for managing routes. We don't support that yet. I was hoping to have that done in time for summit, but it's not done. So another week or two or so, and we'll have initial support for using Diego. And the new routing API. And it's currently beta quality. We know there are some issues with it. But for the most part, it works pretty well. And of course, it's open source. It's licensed under the Apache Software License 2.0. And I wrote it in Java. I know it goes the new big, sexy programming language. I'm not a huge fan of Go. I don't hate it. If people want to use Go and they're productive with it, I think that's great. I'm usually twice as productive in Java as I am in Go. So I chose to write it in Java. What was that? Yeah, and it's out on GitHub. So we also have a Bosch release for deploying it. So it's super easy to deploy. If you know how to use Bosch, it should be fairly easy for you to deploy no router. If you don't know how to use Bosch, you should learn how to use Bosch. It's well worth the investment. So I want to talk a little bit about how we've architected Go Router. So there's two main components. There's this no router core component. And right now I have this no router F5 component. And they're very loosely coupled. And the responsibility of the no router core component is to collect all the routing information. All the routing updates is applications move around on Cloud Foundry to collect all that information and then broadcast that out within the application to the no router F5 component to let it know, hey, the status change, you need to go do something about it. So those two components operate independently. And then the no router F5 component goes out and updates the LTM. So the reason that we architected it this way is not everybody's using F5 LTMs. And we wanted to provide ways that other people could come in and use no router to perhaps integrate directly with Amazon ELB or some other enterprise load balancing. So the Go Router itself has some fairly distinct behaviors that are unique to the Go Router. One of them is how it does session affinity. So if you've got an application that needs to have sticky sessions, the way that works with the Go Routers, you send a J-Session ID cookie down to the application. The Go Router sees that and then it makes sure that any future requests go to the same application instance. So we're able to mimic that behavior on the LTM. We're able to do that through the fact that the LTM is programmable. We can write our own scripts that run on the load balancer. It uses this awful programming language called TICL. If any of you have any desire to look at TICL, I would highly discourage you from doing so. It's awful. But it works. And it actually works very well, to be fair. If I had to choose between TICL and Go, I'd take Go every time. Anyway, so the Go Router also has its own error responses that we've been able to mimic on the LTM. And one of the really nice features of the Go Router is if your application is starting to come up, but it's not fully up, and it tries to route a request to that application and it fails, it will retry two different instances of the application. And we have that same behavior using the LTM. So right now we have our No Router running in our production Cloud Foundry environments. But it's not for production traffic. So we have multiple LTMs in our organization. We have production LTMs and then some non-production LTMs. And on our non-production LTMs, we're using No Router. And so far, it's actually working really well. All right, so I want to change gears a little bit and talk a little bit less about networks and load balancers and talk a little bit about troubleshooting. Because I think this is the aspect that really pushed us to build No Router in the first place. So on our team, we managed Cloud Foundry, but we also managed the Enterprise Load Balancers. And when I joined this team about three years ago, one of the things I thought was really interesting is about once a week we'd have a developer or some operations guy come to us and say, hey, my application is not working and it's your load balancers fault. And nine times out of 10, it was because they had misconfigured something in their application or something that had nothing to do with the load balancer. But because the Enterprise Load Balancers, this opaque system that sits out there, it's really easy for them to say, I can't prove it's not the load balancer, so I'm going to blame the load balancer. And we get blamed for all kinds of things. And what's interesting is applications have migrated to Cloud Foundry. We see a lot of developers come and say, oh, it's the load balancers fault. We show them it's not. And then they come back and say, oh, it's got to be the Go Router's fault then. And so in a lot of ways, it's kind of doubled our effort to troubleshoot things and to prove to them that it's not the load balancers fault, it's their application or some other factor, except for the few times where it is our fault. So we have had some issues with the Go Router itself that were fairly challenging to debug. I don't want to get into what actual issues were, because really, they're issues with the Go HTTP library, some quirks in there that kind of percolated into the Go Router. But we were deploying a COTS application, and so because it was a COTS application, we had a lot less visibility into what was going on with this application. And we're seeing a lot of really weird behavior as we're accessing this application through Cloud Foundry, and it became, it was a really difficult challenge to debug exactly what was going on. But to be fair to the Go Router though, I think we see a lot of those problems, people blaming things on the Go Router because the Go Router isn't nearly as opaque as something like an enterprise load balancer, they have some visibility into it because of the logging and the metrics that the Go Router provides. So you get access logs from the Go Router, they got published over LoggerGator, so it's really easy as an application developer to be able to go in and see what's actually going through the Go Router, as well as metrics. So you can collect metrics through LoggerGator. Although collecting metrics today isn't as easy as maybe it should be, but we'll get there. But the fact that we can actually go in and see that in the fire hose and be able to say, oh yeah, traffic is going through the Go Router. That's an important feature. And that's definitely a feature that we don't wanna miss out on with No Router because it's a really important feature. So we're actually able to publish the same access logs the same way the Go Router does over LoggerGator, as well as metrics. And it's a bit of a challenge to do it with the LTM because the LoggerGator depends on its own protocol called Dropsond. Dropsond depends on protocol buffers and there's no protocol buffers at all on the LTM. And there's no chance I'm gonna go and try and implement protocol buffers and tickle. And so we came up with a solution that actually works really well. So on the LTM it has its own, it's a feature that they call HideSpeed Logging. It's a way of sending a logging out to a third party over the network. And so we're able to send text-based logs to No Router. And then what No Router does is it's able to convert that text message into a Dropsond message and then pass that message onto Metron. So Metron is a component of LoggerGator. Metron is what receives all of the logging events and messages and then forwards that onto the magic that is the LoggerGator system. So we still have all of that and it works really well. But because a lot of the problems that we've had, troubleshooting the LTM and getting visibility into what's happening in the LTM available to our customers, there's a lot more that I would like to do as far as integrating the LTM with LoggerGator. And I think this is a feature that's, even if you wanted to continue to use Go Router, whatever, that's fine. But I think with No Router what I would really like to build is a system where we can actually collect more of the metrics and information on the LTM and make that available over LoggerGator. So if you are using an Enterprise F5, you might have a traditional deployment with Go Router, but still be able to collect a lot of these things and send them off to your applications. But yeah, so the LTM, it collects a lot of really valuable statistics like connections per second and not just the actual TCP connection. But if you have multiple HTTP requests over single HTTP connection, it collects statistics about all of that, how much data is going in and out, data rates and all those kinds of things. And so really what we like to do is be able to take all of those awesome statistics that are available on the LTM and make them available to our customers that are running applications on Cloud Foundry. All right, so I forgot to start my timer. I have no idea how much time I have left, but since lunch is next, I don't think anybody would mind if we got out a little bit early. So I wanna talk a little bit about the pros of this type of architecture using No Router. The single layer load balancing, I think that's big for us, especially since on our team, we manage the enterprise load balancers, but we also manage caching appliances and a variety of other things. And we've seen lots of problems where the more proxies you have, the more prone you are to random failures. We've had issues where we've had testers come and say, hey, we're gonna run a load test. And they told us it was gonna be a small load test and it ended up being a huge load test and it brought down a whole bunch of intermediate proxies and things like that. And we've had Go Router outages because of load tests. And so the fewer components that we have there, the less components that are that can fail. And that's a big deal for us operationally. It also simplifies leveraging a lot of the F5 features that are there, such as the global traffic management, the GODNS stuff that the F5 provides. And there are a lot of other things that we'd like to start using, like security profiles and caching, that no router makes it a lot simpler for us to use. Like I talked about, it simplifies troubleshooting. And there's a lot lower latency when you only have one hop. And the LTM is a lot faster than the Go Router as far as forwarding requests. That's what it's built for. That's how F5 makes all of its money. It's through load balancing, so they built a pretty amazing product. And then the layer three and four capabilities that we get by going through a real load balancer is pretty significant. There's been a lot of talk here at Summit about adding support for not just HTTP routing into Cloud Foundry applications. I would also generic TCP with no router. We can do that today. We can just create a virtual IP on the no router that goes to one of the no router managed pools on the LTM and do TCP instead of HTTP. So it gives us a lot of flexibility there. Of course, there are significant cons. One of the big ones is it's non-standard. That's just the thing that we've built. It is open source. We would like to see our community get built around it for people that are interested in running in an architecture like this with a single layer of load balancing. But yeah, but IntelliMatures, it's just something that we've built. And currently, we only support the five LTMs. And then another big inhibitor for a lot of organizations I think would be gaining access to your enterprise load balancer. For a lot of people, the load balancers are managed by a network team that's somewhat disconnected from the platform team or a DevOps type team. So gaining access, that might be an issue. Feature-wise, we'll always lag behind Go Router. Go Router is the standard. Fortunately, Go Router is open source and it's really easy to see when something gets committed to Go Router and see exactly what's happening there so we can keep up with Go Router. We are missing some features. We don't support wild card routes right now. Like I said, we don't support the route API, so this won't work with Diego today. Context path routing is coming, so soon with Cloud Foundry, you'll be able to bind not just a host name to your application, but also a path. So we're gonna have to figure out some way to support that. And then there's this idea called Route Services where you can create a service that affects how requests get routed into your application using the Cloud Foundry services mechanism. We're gonna have to figure out how that works, although that hasn't really been built yet, but it's something that we'll have to play catch up with. All right, so we've got five minutes left. Does anybody have any questions? Okay, I'll repeat your question. Go ahead. Select before. Is there an after a benchmark? Oh, thank you. So I didn't wanna focus too much on the performance actually, because that was more of a plus for us building No Router. It wasn't the biggest motivation, but go out to my Twitter account and you can see a graph that I posted a few months ago and all the feedback that I got from it. So that being said, so I did do a benchmark and just going through the LTM was significantly faster. And the awesome thing that came out of that though was the pivotal guys turned around immediately and we worked really closely with them and not identifying some performance issues. So actually the next reason for Cloud Foundry to go Router will be quite a bit faster, latency wise, so. Hi, this is also regarding the performance issue. You already mentioned the session keep alive between the Router, go Router and the apps is not there. Does this fix that issue? Meaning of course you have F5 can keep alive the session. Have you enabled that for your setup right now and has it given you a performance boost? With the sticky sessions, the session keep alive features that... Oh, okay. So the connection is about... The question's about HTTP keep alive between the go Router and the application. Yeah, so we've actually enabled that on the LTM. So that is an issue with the go Router and for every HTTP request that comes through when it hits the go Router, the go Router will open a new TCP connection into your application for each request. That does cause some overhead, although it's not that big because typically they're on the same network right next to each other. But using the LTM it is faster to keep that TCP connection alive. The LTM also has a feature called One Connect where you can tell it to keep a TCP connection open for multiple requests originating from different clients. It's not enabled by default because it's a security issue with some applications, but I think with most modern applications it's not an issue. And that's something that we've played with and enabled and it does have a significant improvement as far as latency goes. And Mike, another question is like, also in your opening slides, you cited Layer 3 security as one of the motivations. Can you help us understand like how some of your applications are trying to restrict inbound IPs and how moving to F5 actually enables you to do that again? Okay, sure, that's a great question, yeah. So one of the things that the LTM enables is from a network perspective it's configured so that you can have any number of IP addresses on the LTM, so virtual IP addresses. So I can go out to the LTM and say, hey, I wanna create a new virtual IP address and it'll give me a new IP address that's unique to that virtual server. And then what we can do is we can say when you load balance requests that come in on this IP address send them out over the same IP address. So the application can say, I've got this particular virtual IP address, it's got whatever security settings, authentication or authorization or whatever it might be and then we can forward that off to the application from that same IP address. So the application can then say, I will only accept connections if they come from the specific IP address. So it's a very fragile way of doing security, I hate it, but whatever we gotta support it because we have apps that need it. All right, any other question? Oh, another question right here. So the question is about load balancing between applications and services basically. So a lot of that's left up to the application itself or the service itself depending on what it needs. For a long time with like our databases, we'd load balance to database instances and like an Oracle Rack cluster, but Oracle's changed their architecture so we don't need to do that as much. All right, well, I've got a hard stop. So if you have any more questions, please come talk to me in person. All right, thank you. Thank you.