 So welcome to my talk about setting up a multi-cloud enterprise great Cloud Foundry installation. My name is Jürgen Graf. I'm a developer at SAP at the infrastructure team there. So our team is concerned with basically setting up the stage for Cloud Foundry installation on various cloud providers like AWS, Azure, GCP, or also our own data centers and customer data centers and OpenStack installations. And I'm going to talk about, especially how we use a tool called HAProxy to bridge certain infrastructure differences we faced there. So we have to deal with requirements that come from customers or from internal, via certifications we have to meet, or also from legal. And there are certain things that you can do with stuff that is given to you by cloud providers, or you can do some with one provider, but not the other one we see with others. So it's difficult to get everything together under one hood. So the main problem we face is, well, the goal is we have a customer, and the customer wants to use Cloud Foundry. So it's basically very easy. But in order to have Cloud Foundry, you obviously need something like a cloud. So you have to provide a cloud to Cloud Foundry, and then why choose some specific cloud? Why not give the customer the choice to decide on which cloud the installation should be put on? So we have various options there available. And everyone looks and feels a little bit different under the hood, but it should feel to the customer like there's no big difference at all. So therefore, what we do is also why not put some common interface in front of it, some add some tools to manage the system, measure performance, process application logs, write audit logs, and so on and so forth. Also, we need to add some billing frameworks there. So many, many stuff gets that adds value for the customer gets added there. Then things that happen also is, well, we have some requirements there. Maybe that's an internal development installation that should only be accessible to developers or to certain parts of the organization. So we need a firewall, some kind of firewall there. Maybe you have legal rules. It's a productive installation for customers that have ongoing trade embargoes with certain countries. So we need to block access from those countries to the installation. Then another problem we face is, well, maybe the customer is already a good customer of SAP and has some on-premise solution running and wants to use them like you have HANA installed and wants to use them from his Cloud Foundry installation. Then another thing that can happen is maybe the customer has his own server somewhere installed, his own special system that he also wants to access from within Cloud Foundry. So now you quickly notice what was once a simple setup looks a bit confusing now. So what happened here? So it's a bit strange. We just wanted to give the customer access to Cloud Foundry. And you should not have to deal with all the underlines that are there with all the things that can go wrong and what's under the hood. So what we want to do is to build some kind of black box that puts, that allows access to Cloud Foundry, no matter where it runs on and what other stuff is behind there. And inside this black box, and there is a program working for us that covers parts of this integration that is called the HAProxy. And now I'm going to talk about what HAProxy is. So HAProxy is basically a very reliable and major open source project. The 0.1 version was released already in 2001. Since then, it's been constantly developed and improved. Currently, version 1.8 is in development. It focuses on high availability, load balancing, proxying for TCP and HTTP. And it's also very secure. So no intrusion in 13 years. The good thing about HAProxy is it's so major. It has huge configuration options. So the configuration manual alone has 16,000 lines of text. And there's also administration manuals. So there's not one manual. There's several. So everything you can think of, you can configure with HAProxy. And it's also used by major players like GitHub, Kernel, or Airbnb, Instagram, and so on and so forth. And the main developer, Willy Taro, is also a guy who knows his stuff. He's a Linux kernel maintainer. Now, why do we need HAProxy? So what are part of our requirements we have to fulfill? Yes, well, we have obviously to route and balance traffic. We have to be able to use multiple SSL certificates in case we want to support custom domains for our customer. So they can have multiple different domains in one Cloud Foundry installation with the apps running. And we also have the requirement that the apps that run on Cloud Foundry need to know the real client IP. So there should be an so-called x-forwarded-for header set in the HTTP request that contains the actual IP of the client that connects to the app. This is, in some installations, a problem. I will come back to this problem later on. Obviously, we also want to scale and perform well, have login capabilities, and so on and so forth. And it should work on every Cloud infrastructure. And another additional thing that we have to meet is we have really to be able to configure restricted access for development landscapes, and also in case of trade embargoes. Then now, let's take a look at how a normal setup could look like. So we have on the left side our customer that wants to access Cloud Foundry. So we put some kind of local lens in front of it. And then on the right side, we have a Cloud Foundry installation where we have some instances of routers running and some instances of access nodes, SSH proxies running that allow SSH into applications. So first approach would be, well, first thing that enters our installation is basically the packets come to the local lenses. So we put the access restriction on the load balancer and we terminate SSL on the routers. The problem that we face here is on some installations, the load balancers provided by the Cloud providers have limitations on the number of rules. Or when you have to deal with trade embargoes, blocking whole IP ranges for countries, you're talking about 5,000 rules of different IP ranges you have to put in there. And normal Cloud providers have a limit of about 25 rules. So we can't actually use some. They have varying features depending on which Cloud provider you use. Sometimes if you have your own installation of Cloud Foundry running on your local compute facility, maybe you have to use some different kind of local lenses that's already installed and so on and so forth. So on the side of the router, we faced the problem with SSL terminations that multiple certificates were not supported. I heard that this is no longer the case, so maybe you have to take a look at this. We also have to look at the performance of the router if it is able to handle SSL termination among other tasks it has. So it's already very busy. So we were a bit concerned about putting too much load on this. And we have additional features for logging and stuff that I can come back later. Well, it did make sense to maybe try something else. So one thing that also was a problem is when you put all of the access restriction additionally on the router, that this obviously puts even more load on there. And it has also problems with some of the load balances to be faced here. So the other thing would be, well, move everything to the load balancer, terminate SSL there. You do access restriction also there. Then you have the problem when you terminate SSL on the load balancer. Load balancer has to create a new connection to the router. And what you lose there is you lose basically the information of the client that connects. Because now the load balancer connects to the router and the app no longer actually sees which client requested access to it. So this is a problem in this case. So it's also an additional problem when we do stuff like terminate SSL and restrict access on the router. Because access restriction in case the load balancer creates a new connection does not work because you only see the load balancer IP. So this was another problem we faced on some cloud providers and some local installations where the load balancer always made a new connection. And the router basically did not know the actual client IP. So what we do then is basically we decided to put something in between that can cope with all these differences. This is the age of proxy. It terminates SSL for us. And it does access restriction. So what it does is it the load balancer connects to age of proxy instances and age of proxy forwards the traffic then to the routers and to the access nodes in SSH proxies. And it also has this feature that you can that the X forwarded for header that tells the router the actual client IP so that the application still knows which client tries to access it. And there's an additional feature that we use is in case the load balancer always wants to do a new connection. There's a thing called the proxy protocol that adds basically prefix to each TCP connection that tells the original IP. So basically the load balancer can tell the age of proxy what the client IP is. And the age of proxy can use this information then and put it in the X forwarded for header. So the application still knows about the actual client IP. This is basically what we have to use in some circumstances. We know it is a workaround, but sometimes it is necessary. And it also it is not too bad because you see basically what is put in front of the data that is transferred where TCP connection is a simple line of text that contains the IPs of the actual client. And it's only done once per connection, though only on when you establish the connection. So there's no huge overhead. What age a proxy can do is put basically the client IP into the X forwarded for header. And it also adds its own IP as the second field in the list there. This comes in handy later on. So obviously you do not have to configure age a proxy all by yourself. You do not have to read those 16,000 lines of configuration manual. There is already an age a proxy push release available on GitHub. It's in the Cloud Foundry Incubator. It's ready to use for your Cloud Foundry installation. It can do all the things you need. It can terminate SSL, supports multiple certificates, automatically can forward traffic to the routers and to the access nodes, SSH proxy stuff. It supports Bosch links. So it's actually future proof. And it also supports manual configuration in case you need it. It has additional logging features. You can turn the proxy protocol support on and off. And it also supports multi-threading. Though this is also crucial, we notice this is a huge deal when you have to be performant with SSL termination. And last but not least, we have implemented black and white listing that can be easily configured via the push release. I'm going to talk about how we do embargo blocking at SAP. So this is sometimes a legal requirement due to trade embargoes. Our customers want some access from some countries blocked to their installation. And what we have to do here is we do not have to do this once. And we are done. So we get a list of countries. This list may change from time to time. All of those countries get new IP addresses every once in a while. So there are countries that get new IP ranges on a monthly basis. So you have to keep updating them. And you also have to keep track of what you blocked maybe a month ago or something. So for auditing purposes, and though we have to be able to look back in time and see what we did there to be able to do this. So why do we not just use IP tables and set up a firewall? So this is obviously a problem when you do not have actual access to the firewalls or when you're behind a corporate firewall. This is often done by a separate team. It involves big processes that can take a long time. So if you have to constantly update stuff, this is not ideal. And there are also different responsibilities there. So it made sense for us to put it into the configuration of our own installation and use HAProxy for it. So our setup is basically we have split it in two parts. We have one internally running set up that builds actually the blacklist that runs periodically once a day or once a week. I set it up. It gets a list of countries and regions that needs to be blocked for a specific installation. It fetches the IP ranges from a GUIP database, such as GUIP IP donations. Those are freely available on the internet. Then we convert those ranges to a format that HAProxy can read. And then we keep this list in version control so we can look it up later on. Then we take this list and put it into the manifest of the HAProxy. We deploy it, and then we can test it actually while the proxy protocol where we can put arbitrarily client IPs in and see if some are blocked or not. So if it is as we expect it to be. We also measure the performance of this thing. So we basically try to block all of the internet once, created a list for all IP ranges that was in the GUIP database. This is about 230,000 fighters. And we measured no significant slowdown at all. So it seems to perform quite well. All right. So this is just, again, to summarize how we do it. So we get a list of countries. We convert them to IP ranges. Then you can basically with a three-liner from Ruby, you can convert those ranges to ciders. And then we can put the ciders into the manifest of HAProxy. And only call deploy, deploy will create HAProxy instances that use this blacklist for access control and deny access from clients that match those IPs. So now I'm going to show a short demo how this works. So let me take a look. So what we have here now is I have set up a local Cloud Foundry installation. Whoops. Well, two apps are running. So I have deployed a Hello World app and an X4-Volid4 app. We can basically, we can browse to these apps using the Safari or Hello World is running here. And I also have an additional app deployed, the X4-Volid4 app. This comes in handy because it basically, the only thing that it does is it prints out the X4-Volid4 header of the request. This allows me to see my own client IP. Well, this is this IP. This is me doing the request. And the IP of the HAProxy instance that basically forwarded the request. So the setup that I have here is when you use BUSH. So I have, let me look up here. I have a Cloud Foundry installation running where there's a single router deployed at IP ending with 43. And I have HAProxy deployed running on the IP ending with 142, just basically what you see here. And I use basically HAProxy to connect to the router to get access to my application. So now what I can do here is I can do a manual curl to HAProxy. And it gives me basically the request from the app. Then I see, well, this is my IP. I try to connect to it. But what I do now is I go to two different machines and I'm going to block one of them. So in here, I'm just using the Hello World app as some kind of machine that I can block later on. So let's SSH into Hello World. And let's put here, this is my local machine. And I can do something like constantly running the curl. So it runs every two seconds and curls basically the application. And I do the same thing now on the container and the application. Let me just copy this. Run it constantly. Now we have both running. We see here the application container has a different IP than my local machine. Now what I'm going to do is I'm going to edit the manifest of HAProxy and block my application. So you see here the manifest of HAProxy. And the important part is the properties here. So you can basically add an additional property that's called blacklist. And I can just add here the IP of my application and redeploy HAProxy. It shows me that I had some changes in my configuration. I added the blacklist. So what happens now is HAProxy gets redeployed. We will see a short downtime for both curls because I have no load balancer in between. So there's a very simple setup. You have to wait for it to start up running again. And what you expect to see now is the local machine still has access while the application can't access the X4.4 app anymore. So you can also fuse this in the statistics of HAProxy. So it has a very convenient statistics page that has been enabled here. Now let me quickly make this bigger here that you can read this to see here. So we have basically a front end called HTTP in. And the back end that you could see all the routers that are configured currently, we only have one. See all kinds of stuff. One interesting thing is here the number of denied requests in the front end. This is basically what we block. So you can see this very easily. And basically it's very nice to have this kind of statistics report here. So far, this was about how we can block access. Now let's go on to a slightly different topic. So additionally, what we also had to look at is the SSL support in HAProxy. So we have to deal with multiple certificates. We have to support certain ciphers. So we have customers that want to have a certain set of ciphers enabled or disabled. We have to have the option to enable and disable old TSL versions. And it also should be quite fast. So we don't want to spend additional money provisioning unused VMs because it's just a slow. So we measured how performant HAProxy is when doing SSL termination. What we did here is basically we set up a simple Cloud Foundry installation and used the Apache benchmark tool to do some requests to the API. So we measured it on two different instances on AWS and one in our custom OpenStack landscape. So it's differed in the number of CPUs. So you see here, all with two CPUs on AWS, we get around 81 requests per second and it's a bit bigger for larger machines. But you would expect a little bit more here. So you have put in double the amount of CPUs and you only get a little bit more performance. So that's not too good, in my opinion. So we also compared this to what the go router does. So to keep in mind, this is a simple test we did. So maybe 47 on the very top. I'm not completely sure if this is a real number here. Maybe the go router has to deal with different tasks, so he was kind of busy. But in all, so when you look at the larger machines, the go router is quite good at SSL termination, so it was very fast. But what we noticed is, well, HAProxy only runs on one thread. So we try to enable multithreading, depending on how many CPUs there are. And now you see, suddenly, you get very many more requests through. And it's also very interesting that the number of requests almost doubles on the systems with two CPUs and with four CPUs. So what we see here is that it almost scales perfectly, only when you go from 2 to 4 in the AWS case, where we have four CPUs, it's not scaling perfectly anymore. This was due to the virtual CPUs you get from some cloud providers, so they are actually two real CPUs running in hypersweating mode. So you can't expect to get a full four-time speed up here. So to basically sum it up, what we have seen here is that HAProxy is a major and future-rich software that is stable, can run fast, and it can do all kinds of things that you can think of that you may need, like the multiple certificate support, black and white listing, can be implemented quite easily. It supports multi-threading. It supports workarounds like the proxy protocol. It has very nice statistics that you can use. And the Bosch release is already there, and it's future-proof with Bosch links. So we are quite happy using HAProxy right now. But if something better comes along, we are free to just use something else. So we are not fixed on this thing, but currently I think it's the best option we have. And we are quite happy with it. So this is basically the end of my talk. So if there are any questions, I'm glad to take some. Come again? So the one reason we always have is we have to look at different cloud providers. So we want to have a solution that looks the same basically everywhere, because when we do special cases and use the special features of some installation, maybe it's better for this single installation, but it complicates stuff and the large picture. So we certainly should take a look at it, but we have to also look at does it work on Azure, does it work on GCP, does it work on AWS, does it work on whatever comes around next. So this is basically the trade-off you have to think about. But sounds interesting. That's not up to me to decide. So I hope it won't take too long. So we have a meeting with Shannon today. Maybe if we... I'm not sure what we discussed there. Maybe I can tell you something about this tomorrow. I don't know. Well, I can only tell that we are using it productively and we do not experience any problem. So this is one of the components that basically makes no trouble at all for us. Welcome. So if there are no more questions, then thank you for your attention.