 Welcome to our talk about the rocket science of deploying Cloud Foundry in a multi-cloud environment. My name is Fabio Berchtold. I've been working for the past three years for Swisscom's application cloud project. Naturally, this means I've been working with Cloud Foundry, Concourse and Bosch and various other Bosch releases. I'm also part of the Cloud Foundry ambassador programme and I'm a passionate Golang developer who has contributed to various Cloud Foundry projects. Hello everyone. My name is Andrea Emo and I've been working for now more than one year in Swisscom for the application cloud project. Started in the infrastructure team, was mainly involved setting up the whole multi-cloud environment based on OpenStack and joined the team at the beginning of this year and now doing all the stuff which Fabio does. Okay, let's start with our talk. Before we start, there is a brief outline about the topics we'll show you. We'll introduce the Swisscom application cloud as a product to you. We'll discuss why we've chosen a multi-cloud environment, how the multi-cloud architecture actually looks like at Swisscom. We'll also give you a short introduction to the Bosch multi-CPI feature which actually enables all of this. Then we'll give you an in-depth look about the Bosch deployments that power the Swisscom and application cloud. You also get a brief detour to our continuous deployment setup with Concourse and last but not least, we'll show you one of the cool features this enables to be able to migrate easily between different infrastructure providers. So let's start the journey. The Swisscom application cloud, what is it? The Swisscom application cloud is Swisscom's custom platform as a service offering managed by Swisscom that we offer to anyone who wants to use it. We have the public offering available on developerswiscom.com. You can go there. You can create an account. You can register, put in your credit card details and you can start pushing apps to this platform as a service which of course has Cloud Foundry at its core. We also offer a great variety of services in our marketplace. They mostly consist of the usual suspects which I'm going to show you now. Let's have it in some services. We have MongoDB, RabbitMQ, MySQL based on MariaDB. We have Elasticsearch, of course, Redis is also there. We have an S3 compatible object storage named Dynamic Storage. And last but not least, since we are a telecommunications provider and company, we also have a service called Smart Messaging which really is just a fancy name for SMS or short message. Our main business actually though is the Swisscom Enterprise application cloud which is a managed platform as a service that we offer to our enterprise customers. The Swisscom Enterprise application cloud is the same as the public offering but it's installed, deployed and managed by us within our own data centers for our customers. This platform as a service in our data centers then is network-wise interconnected to the internal network of our customers' companies. This network interconnectivity is actually one of Swisscom's specialties. The Enterprise application cloud offering also has the same services in the marketplace and all the data and everything resides within our own data centers. Hence one of our main selling points is that all your data stays within Switzerland when one of these famous three-letter agencies comes calling, you know you're safe with us. Now I'll hand it over. So yeah, some of you may ask why we set up a multi-cloud, so first of all our customer wants to have reliability, failure tolerance and also high availability. And of course the customers expect a cloud to be always up and running and probably they're also limited to external or internal regulation which they have for their own projects they want to push onto the application cloud. So it was quite clear that we need multiple data centers to fulfill these requirements. Secondly it's also a story about our own internal lessons learned because the maintenance work on the infrastructure below should be without downtime for any deployment or workload running on top of it. So with having different customers and having them agreed and coordinated to a single maintenance window that's nearly impossible. So this would probably lead to the situation that your infrastructure is less maintained which would in turn bring more box or missing features and yeah, you can't fix them because it would affect multiple customers at the same time. On the other hand we also want to be somehow flexible for the future so it's kind of a baseline for a hybrid cloud setup. So next to provide our customer the best possible experience we came to the conclusion that we need multiple data centers. So we engineered the Swisscom application cloud around three different data centers as its backbone. As you can see from this picture we set up our infrastructure in this case OpenStack and VMware on these three different data centers. Each of these data center represents an availability zone from Bosch's point of view. You see there the Bosch director is placed in one availability zone and it orchestrates the different ES in the different availability zones. It has each of these availability zones configured in its CPI configuration and they are mapped to names like zone one, zone two and so on. When deploying cloud foundry all we need to do is to specify what we would like to have. Let's say we want to have three instances one in each availability zone and then Bosch will automatically spread these across those data centers. The really cool thing about this setup is that all these availability zones are directly connected network wise means any VM in any availability zone can talk to another VM in any different availability zone side independently. And this allows us to let's say a really true multi-cloud which means one cloud foundry installed, oh I'm sorry, one cloud foundry installation stretched over those three different data centers. The blue box where you see there is DR-5. It serves as an application firewall but more importantly as a load balancer our entry point for any traffic going to the go routers of cloud foundry. Each site has its own local entry point for any traffic for any traffic for each site and it routes east-west bound traffic. On top of this we have what we call the global load balancer and it serves as our single entry point for incoming traffic from outside. It distributes globally all incoming requests to each site. The special thing about this setup is that in most so-called multi-cloud environments there is a separate cloud foundry installation per data center or per availability zone or whatever but the big disadvantage there is that there has to be some somehow a logic in place which manage the distribution of for example pushing an app. Usually this is done by some kind of API replicators or low balancers but for us this was not really a practical solution so we try to improve that and engineer this great setup which enables us to do something like this like stretch one cloud foundry over different availability zones which translates in our case to different data centers. So yeah that means in fact that we manage to realize one single push for our customers and they don't have to care about what's happening behind the scenes. But thanks to Bosch and its concept of a CPI which abstracts away the interaction Bosch has to do with any particular infrastructure provider any deployment is completely infrastructure agnostic and can also be deployed on a hybrid cloud platform. We can mix and match various infrastructure provider as needed and Swisscom is in the unique position to be actually able to pull this off thanks to being an ISP itself because we own and control our data center and all the network pipes by our self therefore it was possible to guarantee also a minimal latency in this case. So in the worst case we have between this physical locations in the worst case we have a latency of about two milliseconds which should be still fast enough for any component running on top of it. And through features such as isolation segments we can even give our customers the choice on which data center or infrastructure they want their app to be pushed and running on. And also a big advantage of this is that this allows us to have this threat maintenance window because we can take down a whole data center and Cloud Foundry will still happily keep on running. It provides us the high availability and fail tolerance we need. So now a short topic the Bosch multi-CPI feature I don't want to go too deep into this topic but I think it's really an enabler for this story. So all this magic which Bosch allows us to do comes from the core concept of a cloud provider interface. In the past it's not a long time ago the Bosch director wasn't able to handle multiple CPI configurations at the same time. It was only able to talk to one configured infrastructure provider. When we started our journey in the middle of last year we quickly realized that we want to be able to talk to different infrastructure provider from the same Bosch director from the same at the same time. And Bosch fortunately had already the concept of this native built-in availability zones. We thought why shouldn't we apply this concept to the infrastructure too. And yeah that's what we did. The Bosch multi-CPI feature was developed by Swisscom in collaboration with Pivotal last year to allow a Bosch director to handle multiple CPI configuration at the same time. This not only allows us to configure multiple availability zones for one infrastructure provider it allows us even to have multiple infrastructure provider and then deploy software across it. And the cool thing is it enables us also to provide a hybrid cloud environment and from the end user's point of view it's all completely abstracted. They don't have to care about it. So let's have a closer look at the inner workings of the Swisscom application cloud. It always starts out with the initial Bosch director when we set up a new customers environment. As mentioned before we have three different availability zones. Now the reason for choosing three zones was because this plays particularly well together with various leader election algorithms like the RAFT consensus algorithm and so on. That for example we have in our standalone console cluster deployment. We also have ETCD clusters. We have distributed MongoDB enterprise clusters, Redis and so on. So it seemed to be the reasonable choice to make to choose three different availability zones which map to three different data centers. The first thing we usually deploy is what we call the core infrastructure. Here we have a DNS deployment based on unbound. This DNS deployment serves as the whole DNS backend for all the other components that will follow. This goes together with the console cluster that we deploy. Here we actually have three console nodes per availability zone which translates into a nine node console cluster. Now you might think this is a bit of overkill and actually poses some performance problems and trouble for us but we're really only using the console cluster for its service discovery features based around DNS resolution and we're not really using any of its key value store based stuff so we're fine with having a nine node cluster. In fact it actually improves the high availability and especially failure tolerance for us because having a nine node cluster means we can take down a whole zone for one of these maintenance windows and even one additional node can fail on any of the availability zones and the cluster still remains or still keeps, maintains, quorum and stays healthy so this was an important choice for us to make. We also have a Docker registry per availability zone which is used by Kubernetes and Concourse and other Docker based services we have. We deploy one concourse worker per availability zone and the reason for only deploying workers will be explained in the next slide. We also have a Galera cluster and we have Stark and Wayne's shield. The Galera cluster serves as the SQL backend for all our RDBMS needs we have like for example club foundry needs and shield is there for obvious reasons it serves as the safety net we need to be able to sleep well at night and it provides us with backups. The next components are the business integration components. Here we have the end user facing web portal. We have the CF extension API filter whose responsibility is to intercept and extend any API traffic going to the cloud controller. With this we can implement our own custom API features and we also have the billing component who you can guess what it does by its name. It handles all our billing needs and we have to earn some money after all. Next is the centerpiece, club foundry I'm sure you've heard of it before. Here we deploy multiple isolation segments. Now the reason for this is rather special because we use or rather abuse these isolation segments to be able to provide to our customer to have the different versions of GoRouter and cell deployments in these isolation segments. In the past we've had issues where even the tiniest change in GoRouter behavior or changes in the root file system were actually able to break our customer's apps. And through having different isolation segments with different versions of these components we're able to empower our users to first test these new versions with their apps to see if the behavior is still as expected and if everything works as expected. We also have a big Kubernetes cluster deployment which we use for our container-based services. Any small to medium-sized Redis instances or MongoDB instances for example that you can consume from our marketplace, they're powered by this Kubernetes cluster and running inside a Docker container. Last but not least we have what we call the Bosch-based services. Here we deploy our own in-house developed service broker, the open service broker that's available also on GitHub. We also deploy any additional Bosch directors here whose responsibility is to orchestrate these bigger services. We have Galera clusters in here, MongoDB Enterprise, Redis Enterprise, RabbitMQ is also in there. These bigger services they're basically meant for those times when you need to bring out the big guns and you need a cluster of big sized standalone VMs rather than have the small service instances running in containers. Now as I've mentioned before we only deploy one concourse worker per availability zone and here's the reason why. We have a special central management plane from where we control those workers in this central management plane. We deploy a central concourse ATC and actually also the concourse database and some additional workers and all the workers on all the other environments are configured to talk back to that central ATC. The workers have been specifically tagged for the ATC to identify which environment they belong to or which customer they belong to and all our continuous delivery and continuous deployment pipelines they're located in the central ATC. When we want to do an update all we need to do is we just need to trigger these pipelines there and it goes to the worker and everything is done there. Now these workers most of what they do is they talk to the local Bosch director there to issue deployment commands but they also do various other things like pushing images to the Docker registry or setting up some files or some infrastructure configuration and so on but it's really mostly commands issued to the Bosch director and this Bosch director then deploys or updates the local components there like the core infrastructure, club foundry, the Kubernetes cluster, all the Bosch based services and the business integration components. Now in the beginning when I showed you the outline I mentioned that one of the cool features that these three availability zones, three data centers set up enables us to do is we can easily add, mix and match new CPI configurations with different infrastructure providers life while everything still keeps on running on top of it. For example we can, here in this example we have three availability zones based on open stack in the three different data centers and what we can do is we can just take out zone three for example either for a maintenance window or to really get rid of it completely and in this case here have it replaced with a new zone four which is based on VMware. And the Bosch director will take care of all the stuff on top of it thanks to its magic the club foundry deployment for example it will just keep on running and all the apps on top of it they will just keep on running the end users or our customers they will not actually even realize or what has just happened on the infrastructure side and this is one of the things that we actually aimed for to be able to do these maintenance windows when we wanted and not when our customer was saying it's right to do it. We can continue this little game even further and for example now we can take out zone two and have it replaced by a new zone running on Amazon AWS or to go completely crazy we can also now replace zone one and have it running on Microsoft Azure. Now of course when you start using these external infrastructure providers like Amazon, Google or Microsoft Azure you start to you have to start thinking again about possible problems this introduces especially regarding network connectivity and network latency of course the multi-cloud setup we have wouldn't work if you were choosing let's say AWS regions or in Europe Asia and the US I mean network latency stretched across all over the globe this would be a killer but if you're like us and you have your own data centers and they're somewhat closely located physically and you have good connectivity between them then I say it's absolutely worth it to do this setup and to go for it I mean it's perfect and this was all made possible thanks to our ambitious goal of having a true multi-cloud setup for our club foundry deployments yeah and now it looks like the application cloud rocket has reached its destination the eagle has landed and multi cloud is born so with this we conclude our presentation and we thank you all for your attention and we're now open for questions yes yeah if you only have a single instance of an app then of course it might happen that it's running on a cell which is in this one availability zone we take down and then it might be that you realize that there is a downtime but actually I mean when you issue the redeployment command for club foundry from the Bosch director the drain scripts of cloud foundry will run of these particular cells that the Bosch director is intended to remove from this old zone and the drain should be responsible to relocate the app first to a cell which is on another zone to keep it running so I mean in theory if even if you have only one instance of an app you should not realize that there's any downtime and if you have two or three instance then of course you really should notice because it's cloud foundry's job to to try and spread these app instances over the different cells on the different availability zones I mean the cells or Diego is availability zone aware and it tries to keep it spread no no it's a club foundry should handle this all for you you shouldn't really be and there shouldn't be no need to actually do something yourself as an operator or as an app developer you really just push your app and we can take down this zone and your app should keep running because the drain scripts of the cell they will first before the cell goes completely away they will relocate it on another cell in another zone yeah but while this is happening the old one is still running so the drain scripts in cloud foundry they will not take away the app before it has been relocated I mean of course unless there is a really long time out and it takes really long time to get your app up and running again or it might also be the case if your app has let's say a bad health check which reports back immediately oh I'm healthy again but let's say your app is a big Java application and it takes ages to start up the the application server inside there then of course it could happen but that's why you deploy more than one instance of an app I guess yeah so have you done any testing on like what the the network latency between the different data centers as you said that you own all of yours and they're closer but to the idea of trying to stretch that across data centers that could be further away do you have any estimates on what that latency no we didn't do any testing on how much was possible we just we have our data centers here in Switzerland and of course Switzerland is rather small so I mean we just tested between our data centers currently the maximum latency we have is two milliseconds and for us this was enough so far we have not tried to spread it let's say across with or across Europe we haven't tried this yet so I can't really tell you what the limit would be for it to still work unfortunately hi how do you handle data consistency between the three data centers what you mean exactly because the Bosch director has these three data centers configured as availability zones and there is really no need to handle everything I mean you we just deploy one single cloud foundry deployment and the Bosch director stretches it across these three availability zones so it's still it is one deployment and for example also when we deploy console or etcd it is one cluster deployment stretched across these three availability zones which are on the three data centers the the it's the responsibility of the software itself to keep itself healthy and in consistency we don't really we don't really do any let's say we're not stretching the infrastructure itself just the deployments on top of it across these three availability zones and the Bosch director has this concept of availability zones and it will look after the deployment so when you say you deploy a three node etcd cluster it will deploy one node in zone one zone two and zone three and it's the responsibility of the software to to keep consistent and the three zones network wise they're completely interconnected and the traffic east westbound it's for it for the components running in these data center it looks like a flat network just to elaborate on this little bit have you done anything specific to to to try to address the case in which you completely lose east west connectivity basically because it's a single deployment right so if you lose that connectivity then yeah of course if we'd lose completely the connectivity from every zone to every zone then of course this would stop working I mean it would be like everything goes down but we can safely lose one of these three zones we let's say unintended it was unintended that we tested it once but it happened and funny enough none of our customers noticed so it still kept on running while just having two zones interconnected with each other so the your the global load balancer is able to detach the missing zone yes yes of course the load balancers the local ones and the global ones they have health checks to really see eyes is are these components all still up and running can I send traffic there and so on and if if one zone goes down unintended or actually because we do have a maintenance window then the traffic will only go to the remaining zones and as long as we still have two zones running and connected they will basically maintain quorum and the deployments on top should stay safe and keep running thanks so you talked about migrating things from open stack to vSphere for example which is easy enough if the stuff you migrate doesn't have any persistent data how did you migrate things that do have persistent data yeah that's actually funny we we use or we trust the software itself to do that when we have for example Redis clusters and we take down this one zone on open stack and bring up a new zone on VMware it's our intention is that the Redis cluster is responsible for by itself to see ah I have a new member which has not synced yet and then the Redis cluster should sync all the missing state to this one new node that came up so so we leave it basically all to the software to keep in sync two questions this bush multi multi CPI feature is already in the master branch it's already yeah since I think the beginning of this year it's in the master branch of Bosch and you can use it there is usually you have the command for cloud config and runtime config and beginning of this year another command was added the CPI config and with this you can load these multiple CPI configurations into your Bosch director and some documentation how to provision multi multi cloud solution I'm not sure if there is documentation on how to really provision with multiple different CPIs but there is some documentation on Bosch IO on how to use the multi CPI features but I think that the examples are only for having the same infrastructure provider with just different credentials on something like that but it should be fairly straightforward because all you do is you configure these different CPIs the same as you would before with the same properties for a single CPI it's just multiple of them in one YAML file thank you so you're saying that the end users they don't really notice much of all of this they don't have to care about it but what if I want to care about it as an end user what if I have a space where I want to have the apps deployed into a data center in Switzerland and then another space where I don't care about it they can go into AWS or anything so is there any way of how I can influence that yeah this is actually one of the things we are currently evaluating and probably will have as on as a feature next what we will do is we will deploy isolation segments and have them specifically mapped to one of these availability zones that are for example on AWS and then at deploy time as the app developer when you push your app you can choose to which isolation segment it goes and you can choose for example an isolation segment on an availability zone which is solely located on AWS if you choose to do that since you have one cloud foundry installation where do you place your singleton jobs the singleton jobs which availability zone which singleton job because I don't think we have a single instances of anything we have everything at least three instances I mean famously there was the the cloud controller clock global but that also recently has been made h.a. and we deploy three instances of it now and for the cloud founder deployment we have everything at least in pairs of three and spread over all three zones shield yeah shield unfortunately currently is not yet h.a. so we have it only in one zone but the shield is not let's say not a user facing component it's just for our own infrastructure when we do backups and restores so it's not really a problem for us if we take down the one availability zone where we just have one shield instance we then either either we bring it back up again because it was just regular maintenance window or we redeploy shield just to another zone and it will run again the blob stores they are not actually on these availability zones we have an external s3 provider which we use and all our blobs are distributed there the external provider I think we are using ECS for this and we have it also spread across data centers for us it's an external service that we consume we don't have to worry there about keep it up and running sorry for the stist that I still have a question I was wondering if you have full network visibility from your on-premise installation to the public clouds or do you have any limitations to that that is up to you to choose when you want to have the Swisscom enterprise application cloud it's up to you to say yeah it should be like fully available from the internet or it's completely closed off into your company's internal network or you also want to be able to reach our public offering because you have some apps running there or some microservices which you have actually on the public offering and then on the private offering it's up for the customer to decide we could basically configure it as wanted or needed yeah still one more question one more when you have one cloud foundry installation you have some components that spawn multiple data centers maybe console at CD clusters did you experience any problems upgrading them because when you upgrade cloud foundry and you have to upgrade your at CD cluster or console let's say so far not you never know but I think the the console and DTC deployments inside cloud foundry they're quite okay with it currently and I mean there's an ongoing effort to actually remove console from cloud foundry and our own console cluster as mentioned before is actually a nine node cluster so there we really play it safe we've not had an issue so far where we took down an availability zone and lost data or we had a cluster of console ETC D or so that suddenly lost quorum we were so far lucky enough that this hasn't happened of course it always might happen you never know if suddenly an additional node goes down that you didn't expect but at the moment it it's working how is your deployment work like have you sliced down like different isolation segments or different components for different asies or it's a single deployment the cloud foundry deployment is the actually it are it is multiple deployments the cloud foundry deployment itself the core cloud foundry it's just one deployment that is spread over three availability zones but when we do this additional isolation segments we actually do them as different deployments of course we could also have them in one cloud foundry deployment but we felt because these additional isolation segments there not all of our customers want or need them so we have them separated in separate deployments but the main platform like the public offering or the basic enterprise application cloud offering it's just one cloud foundry deployment but then if you have the management plane as a single deployment how do you manage the compatibility between like the some of the older versions running the isolation segment and a different version of the management plane well that's so far in this in the isolation segment the only thing that are in there are go routers and cells and we've not yet run into the issue where the the API between management and these two components in the isolation segment was breaking I I guess it could be a possible problem if these versions are too far apart but so far we've only always had it in steps of one we always add one new isolation segment for our for customer to test the latest version and we're just one version back and then we upgrade after a few weeks so far we've not had any breaking API changes between management and the isolation segments because it can be a issue between the cloud controller and the the CC bridge yeah for sure it would translate to the VBS so that would be possible yeah but we try to avoid any changes I mean we have integration and testing environments and yeah thanks okay no more questions we're done yay