 So then good afternoon everybody and welcome to this session. My name is Stefan Lai and I've been working for three years now at SAP in the project which tries to build and run a cloud foundry based cloud platform. I'm part of a team which is responsible for building the bridge between the infrastructure as a service. You have heard maybe that we are deploying on many different infrastructures as a service and the teams who are building the Bosch deployments on top of that. So for example we are responsible for defining the network layout and bringing it to the infrastructures, defining how the access to the cloud platform is done for the end users with the load balancer and HAProxies for the developers and operators with the jump, so called jump boxes. And today I want to focus on one aspect of this infrastructure which is called IPSEC. If you don't know exactly what it is, it's not a problem. I will explain it to you today. And here is what I'm going to tell you. So first, I want to introduce you an example scenario where we use IPSEC where we want to prevent some malicious guys maybe called Eve or Mallory. I don't know if you know these people. Where we want to prevent them from doing harm to the data on the cloud platform. And then I will go more into the technical details. So who of you is familiar with IPSEC? Some people but not so many. That's good because I would like to explain you the basic principles of IPSEC because not many people are familiar with it. But we have the impression that it's a quite useful tool. And especially all of you may know TLS, Transport Layer Security, which is a basis for HTTPS for example. I want to compare it a little bit. And then I want to tell you that you don't have to know all the details of IPSEC to use it in your cloud because there's a Bosch release which you can use to make it to simply run it on your VMs. So simply run IPSEC encrypted traffic between VMs. And then I will show you a live demo on Bosch Lite locally where I show you how IPSEC works. And if you have any questions in between, feel free to interrupt me because I think we have enough time. So now let's focus on the example scenario where we use IPSEC productively in the SAP cloud platform. So you may imagine that most customers use SAP cloud platform not because we are the best technology provider in the world, but mostly because they're already SAP customers and they have a lot of running SAP systems in their backend. So one very important scenario for the SAP cloud platform is to expose such backend systems into the cloud. Maybe for one reason that to just make it publicly available in a web browser for everybody or just to develop extension scenarios. On the right hand side we have this on-premise systems here. There is an established solution, an SAP, which exists for several years, ten years maybe already, so-called cloud connector, which can establish a secure tunnel from the customer backend into another system running somewhere else, for example, in the cloud. And for example, in the SAP cloud platform. And then the end customer can access the backend from the application front end in a secure way. So it looks at least here in this picture. So if you use HTTPS from the front end to the platform and then the secure tunnel, you have a secure connection from the end user to the on-premise system. But is it really so? So if you focus now on this SAP cloud platform, we see that, of course, the platform also consists of many different building blocks. For example, in our scenario, the secure tunnel is built by a process on a VM which we expose as a backing service to the applications. So the traffic between this process which connects to the backend may be secure, because it's secure as a tunnel. But the connection from the customer app to this backing service is not encrypted. It's like it is. It's HTTP. And this is just an example scenario. So a lot of other backing services, you know, databases also are not connected in a secure way to the applications. And so we had the challenge to make this connection also secure. One reason is also that customers require such end-to-end secure solutions. And if you don't provide such encrypted traffic on the whole way when they don't go into the cloud with SAP. Okay, so this is now where these evil people come into play. So basically, security is about confidentially, integrity and availability. And one person, often called Eve, just harms confidentially. So Eve just reads the traffic on the line. So imagine you know that SAP is offering, or you may know, the cloud to multiple customers. All the customers are deploying the apps on the same SAP cloud. So it's a really multi-customer cloud. And we don't control who runs applications on this cloud. So we offer even trial accounts. You can register your user if you want. And tomorrow you can today even deploy applications. And if you are Eve and you may try to find some security issues, break out of your container and sniff network traffic. Or whatever you imagine as an evil hacker. And... Or even you could just intercept traffic as a man in the middle, often called Mallory, and just manipulate data of the customer. For example, that's with banking transaction goes somewhere else than intended. Okay, now we asked how can we secure this traffic. So there is an established way to secure traffic on a network level called TLS, transport layer security. On the left-hand side you see the well-known OSI network model with the seven layers. And just show it here because now you can better relate IPsec to the TLS solution you know. So TLS is on the transport layer. And IPsec is just one layer below on the network layer where IP data grams or packets are sent over the wire. Here are some aspects when we compare IPsec and TLS. So IPsec encrypts traffic, encrypts traffic between hosts. So IP traffic is between two hosts with certain IP addresses when this traffic is encrypted. Well, as TLS encrypts traffic between processes running on these hosts. For example, a web server and then a client application. And IPsec is really widely used. Many of us I even also didn't know that that IPsec is really well established in the Internet world. So most VPNs or VPN tunnels rely on IPsec. So if you have your company, iPhone, and you're connected to your company network, chances are high that IPsec is in the play. Whereas TLS, you know, web apps, for example, more and more use HTTPS and also other client server scenarios use TLS. So now the next three aspects are now some criteria which we use to decide if that we use IPsec. So first one, IPsec is configured on a kernel level. With STLS, you have to configure in the client application. For example, the build pack has to provide root certificates and you have to use the correct client's web app which make use of TLS. So with IPsec, an application developer does not have to care about certificates because it's just one level below. Whereas using TLS or HTTPS, the application developers have to care about and we also have to care about that. We have officially signed certificates which per default are in a build pack. So the root certificates are there or we have to create or sign them with our own certificate of authority and somehow bring this root CA into the build pack. So this is much more complicated for the developer. Moreover, so IPsec has a much broader range so we could not only use IPsec for this HTTP server but also we could use it for other backing services for example. So because of these reasons and because we have access to the kernel when we deploy VMs, we decided to use IPsec for this scenario. Now I'd like to go a little bit into more detail of IPsec so if you want to have a high level overview of how IPsec works really then I hope you can learn it now. So it's really built into a Linux kernel. It's version 2.6 which was published in 2003 so it's quite old technology. And there are some open source tools how you can manipulate this or how you can configure this IPsec. So the most commonly used tools are called IPsec tools. First, if you want to use IPsec you have to configure so-called policies on the kernel. Here's an example if you can read that. So with this tool set key you can create but also look into the entries in the so-called security policy database and here you define the traffic between these two hosts so 34 and 35 should use IPsec. So it's an opt-in. So any other traffic doesn't use IPsec so you don't affect any other traffic but only this traffic here. And on the last line you see some details about which kind of IPsec is used. First, ESP, there are different ways how IPsec can secure your payload. ESP is called encapsulating security payload which just encrypts your content inside the IP packets. So the second one, there are two modes in IPsec. One is called transport mode and the other is called tunnel mode. The tunnel mode is mostly used in VPNs. It means that the whole IP packet is just packaged into an enclosing IP data grammar packet. Whereas the transport layer which we use here is not so for tunneling but really for a host-to-host communication. And the last one, require, could also be optional. So require means if the other party, so in this case the 35, has no IPsec configured when the communication would fail. So both sides have to use IPsec. So now if you have such an entry in this database if now my host, one, let's call it Alice, wants to call host two, let's call it Bob, first time when they have to establish a secure connection. This you can compare to a TLS handshake. So if you call a server with HTTPS when at first there's encryption and shared key and shared secrets are calculated. And in this establishment phase there's a so-called security association build which is also another database where the symmetric key is persisted for example and some other information. And after this establishment phase is done then in the data exchange phase the traffic is encrypted really in the network layer. There are two major types of IPsec mode what is called authentication header which does not encrypt but only provides the integrity within hash but the major one is called encapsulating security payload which really encrypts traffic. So let's look into more detail in this establishment phase. So there is an entry in this security association database created, the whole thing, so the exchange of the key is done with IKE or IKE protocol which is done on UDP port 500. So this is important if you set up your cloud boundary on the infrastructure of the service you have to open the ports in the communication between these VMs. There are different possibilities how this internet key exchange is done. So both parties have to trust each other which can be done with a shared secret which you just bring into both sides before everything for example with a Bosch deployment but this is not so secure because when the secret is there in the machine and if somebody knows the secret they can decrypt each traffic on the whole landscape. So what we use is the last one we use certificates on both sides and the certificates are used to create public-private key pairs and with that both can establish trust with each other. So this is one of the contributions we did to this IPsec Bosch release. And let's go on. So how does it now look like on the network layer? So if you have a regular IP exchange you have a header and for example here TCP as payload if we use this ESP-dvence-fair mode then still have the same IP for two headers so the IP addresses are still visible to everybody and then instead of protocol TCP you have your protocol 50 which tells the network layer that you use IPsec with ESP. Okay now if you are confused you still can use IPsec Bosch release because in the Bosch release we encapsulated all these things which happen below. So if you use the IPsec Bosch release you can just deploy it together with your VMs and then this VM will communicate with IPsec. It's open source under Apache license so basically we forked it from cloud credo but this was because there was no response anymore it was in three years on this Git repository and we wanted to introduce some additional features so we decided to fork it one year ago but also of course put it into open source. And we use tools from this IPsec tools. How do you deploy it? You just code deploy a job on your VM and configure it with this configuration parameters so you can switch it on and off. You can say that it's required or it's optional and you have to configure on each VM the other VMs where IPsec should be used for the communication. In our scenario all the Diego cells where the applications run on top use IPsec if they communicate with this VM which connects to SAP back end. So now I hope there's time for demo, yes. Let's go out here. So it's like to mirror display. One moment, find my mouse pointer here. So I have here deployment manifest. It's quite simple one. You have two VMs, one is called Alice and it has a static IP on Bosch Lite and it has such the IPsec process as a job, nothing else. Its configuration is that if it talks to the other VM there's a certificate which has this IP when it should use IPsec. So I hope you can read it. So I hear about two VMs on Bosch Lite. So if I SSH into it, into Alice. On this terminal SSH into Bob and now open a listening socket here. On the other side, just talk to it. I'll type it. So if I just communicate from one VM to the other then I can just see on the other that it's arrived and now on this VM I simulate just Eve who wants to listen to this traffic. I hope we have time enough. I guess I'm not logged in to the director here. So I have to log in to Bosch. Now it works and now I can just do a TCP dump here. Here I have to network interfaces. Okay now if I type hello and my password then I can see here Eve can listen to this and just see the content of the packet. And now why is that? Because IPsec is switched off here and if I switch it on both sides and just deploy it and I don't have to restart even my processes once the deployment is finished the traffic between these two processes will be encrypted. It takes Bosch Lite 20 seconds per VM. We'll also see in the TCP dump all this negotiation stuff with UDP so first you won't see that the traffic is encrypted but a lot of other things. You can just modify my TCP dump where it matters when you will see how this ESP payload will look like. So okay now if I just do any other traffic and you will see a lot of strange things happening on ESP 50 for example is ESP protocol and UDP. If I modify my TCP dump parameters so just don't show this UDP stuff and type something and you see here there are two packets from Alice to Bob with ESP and you can just read anything anymore. So this was the demo and I'll quickly go back to the slides. So what have we learned using IPsec? So once we had the IPsec Bosch release how we wanted it and it was quite easy to set up the communication between the Diego cells and the backing service and it runs very smoothly now for over a year where we're stable and we had no issues at all so it's really a well established solution in Linux kernel. What is nice is that we don't have to hassle about creating certificates for the backing service. Application developers have no TLS configuration overhead where maybe drawbacks are if you... We have only one communication secured with IPsec if you want to have a lot of communications when you have to configure all this IP addresses in the Bosch release manifest files. So far we have not a good solution maybe Bosch links can help here. And if you have this backing service in an HA setup with a load balancer between it gets more complicated because you may have to secure the traffic to the load balancer with IPsec and from the load balancer to the backing service. But I would like to encourage you if you also develop a platform or build a platform to try out IPsec. It's open source and you can... We are happy if you have feedback or even contributions. So now I'm quite nearly over time. Are there any questions? So we have not... The question was how much overhead there is performance wise on network. So we have not observed overhead. So this first phase where the symmetric key is established is quite quick and when the symmetric key is used you can configure it for one day and as it's built in a Linux kernel it's quite performant. So we have not done any measurements but we have had no reason to do any measurements right now. Any more questions? So there's a non-possibility with IP tables. If you have access as a root user on VVM you can just manipulate IP tables so that the traffic does not in the network stack directly go to the normal way. But with these commands you have no time to show it to you. You can just redirect inside the Linux network stack the traffic to some other place called NF log which you can then read. It's a good question because I had just this light here as an appendix. Okay, if you have no more questions if you have more questions you can of course contact me afterwards anytime and then thanks for attending the session. Thank you.