 All right, shall we start? Konnichiwa, yoroshiku onegai shimasu. This is the only Japanese like I started yesterday and the two now I don't know how to speak. Thank you for coming to our session in Japanese, so I will say thank you again for coming to our session. Okay, let me introduce myself. My name is Tom and I'm now working for IBM China Cloud Foundry development team and today the three of us we will help to share our experience on deploying Cloud Foundry on OpenStack and There are two other speakers. I want to introduce the first one is Dr. Max and Max is our well-known expert in Cloud Foundry community and the other one is Edward and Edward is now pair programming with me in China in the Cloud Foundry community. So this is the agenda that we want to discuss with you all today and we wanted to divide the agenda into three sections. In the first section, we want to introduce briefly about the Bosch project and how to use Bosch on OpenStack and followed by a demo to deploy Cloud Foundry using Bosch on OpenStack and the second section before the summit we did a survey in the community and we want to show the feedbacks from the developers and the users in the Cloud Foundry and OpenStack community to you all and finally we want to share our lessons learned during our trial on deploying Cloud Foundry on OpenStack. Okay, so let's start. So first thing, what is Bosch? I just want to know how many of you have ever heard or used Bosch before? Would you please raise your hand? Okay, cool. Maybe half of the people here in this room. And Bosch of course is a Cloud Foundry project and the Cloud Foundry as a platform, as a service platform, it can be built based on some infrastructure as a service system such as OpenStack. So when Bosch was born, it was designed and implemented to deploy Cloud Foundry on ice systems such as OpenStack and Bosch, generally speaking, is a release engineering and deployment management tool and in the past the main target for Bosch is to deploy Cloud Foundry on various ice systems such as OpenStack, SoftLayer, AWS, but now the target turns to be more general. Now it's targeting on deploying distributed, large-scale distributed software where in the cloud. Here I just listed one link at the bottom of the page and this is the official website for Bosch. And if you are interested in Bosch, please take a look at this website. And there are many useful resources and the Bosch releases on this website. So as deployment management tool, Bosch uses many new concepts in the deployment area and the most important concept in Bosch world is release. So what is a release? A release is a versioned collection of configuration properties, startup scripts and software source code and all the installation artifacts that will be used to deploy distributed software in the cloud in a reproducible way. So this is the definition for the Bosch release and we can divide a Bosch release into several jobs and each instance of the job for a Bosch release can be deployed in the cloud on a single virtual machine and job can depend on a set of packages, which stands for the software source code or the installation files. And of course a package can depend on another set of the packages. So here we can see an example. The size is not very perfect, but this is a typical Bosch release of Cloud Foundry. And in this folder we can see that there is subfolder named jobs and in this folder it will collect all the necessary job that to be deployed in the cloud. And for each single job we can see the spec of the job and it will show the definition of the job and also a list of the packages here just part of this job. And go back to the top of the the folder we can see that another important subfolder is the packages. And here we listed all the necessary packages that need to deploy Cloud Foundry in this folder and each package will contain a reference to the source source code in this folder and by organized in this way Bosch can deploy the full Cloud Foundry into the cloud like OpenStack. So the second important concept in Bosch is deployment manifest. Deployment manifest is written and provided by the user and in this file, this is a YAML file and in this file it contains two important sections. The first section is will include all the release relevant information such as the jobs, the packages that needed to be deployed in the cloud. This will tell Bosch what to deploy and the second section is the configuration properties of the IS system that Bosch want to deploy this release to into. For example, if we want to deploy Cloud Foundry on to OpenStack system, then we need to configure OpenStack environment relevant information in this section and So for each kind of the IS system, we need to prepare a deployment manifest but the convenient thing that Bosch provide us is we only need to write the deployment manifest for once and then we can replicate the most content of the manifest for each of the IS system and everybody who has this deployment manifest can deploy and manage the deployment in the cloud everywhere. So Bosch itself is distributed software and the two of its most important component is Bosch director and the Bosch agent. So Bosch director is something like the Bosch server or the Bosch center. We use Bosch client to talk to the Bosch director and tell Bosch what to deploy and where to deploy. The Bosch agent resides on the single virtual machine in the underlying IS system and the Bosch agent will communicate with the Bosch director periodically to deploy the jobs and maintain the deployment. So how can Bosch to communicate with the underlying IS system? The right component is the external component cloud provider interface which is short for ECPI. In the past it is called CPI because CPI is part of the core Bosch code, but now we have separated this part from the core Bosch code base. So we call it external cloud provider interface and the Bosch when during the deployment process Bosch need to create and manage the resources in the IS system such as create virtual machines, create volumes, attach disks or manage networks. So all this work are being done through the ECPI component. So this component will try to use OpenStack REST APIs and try to manage and create resources in the IS system. And also the last important component is the stem cell. Actually stem cell is a special VM image in the IS system. This VM image will contain a pre-built Bosch agent inside the image. So each virtual machine that launched based on this stem cell will have a Bosch agent running inside the virtual machine. This will help the Bosch director to communicate with the Bosch agent in the virtual machine. So understand the other concepts. Now we can see how to use Bosch to deploy software on OpenStack. The process is quite simple, only four steps. The first step is to prepare OpenStack environment. This is of course. And the second step is to deploy Bosch itself. As we said before, Bosch itself is distributed software in the cloud and then followed by using Bosch to deploy as a distributed software in the cloud. In our case, we'll be deploying the Cloud Foundry in OpenStack and finally we will verify the installation. So this picture typically summarized the whole process of using Bosch to deploy software in the cloud. For the first step, we need to use a Bosch client tool named Bosch init and prepare a deployment manifest. It's also a YAML file and the Bosch and OpenStack CPI release and use the Bosch init deploy command to help deploy the full Bosch in one single virtual machine in the cloud. So the rectangle here shows all the components of a Bosch and after Bosch is deployed, we can see that then we can use the Bosch CLI, the Bosch client and prepare the Cloud Foundry release, upload to the Bosch director and then use the Bosch director to deploy the full Cloud Foundry into our OpenStack, the right side. And this picture shows the typical topology of the Cloud Foundry deployment in OpenStack. Notice that this is just a prototype and minimized deployment of Cloud Foundry and each rectangle, the blue rectangle here stands for a Cloud Foundry node and each will resize on a virtual machine in the cloud. Knowing that for the production level deployment, we might need multiple nodes for each of the Cloud Foundry node. For example, if you are familiar with Cloud Foundry, all the applications will be running on the DEA node. So in this case, if you want to deploy a production level Cloud Foundry, you need to maybe hundreds of thousands of DEA nodes in your cloud. So this is just a prototype and an example. So now we can see a recorded demo for using Bosch to deploy Cloud Foundry on the OpenStack. We have already uploaded the full demo onto YouTube and in the last page, we have added the link to the video. Okay, so the first step is to deploy Bosch itself, as we said. There's a word named micro Bosch. Micro Bosch means all in one Bosch in one single virtual machine. It means that we deploy all the Bosch components in one single machine. And in this demo, we will use OpenStack Kilo version. But the same process can be used on OpenStack, Icehouse, Juno, Kilo, and Liberty version. First, we have a clean environment, OpenStack environment, no virtual machines. And before the Bosch deployments, we need to prepare several things. The first thing is to provide a new security group. This will define some special part that the virtual machine will open to communicate with the Bosch director. The detailed configuration rules can be read on the Bosch.io website. And second, we need to prepare a key group. And finally, we just need to assign two floating IPs to deploy. One floating IP is to deploy the Bosch and the second floating IP is to deploy the Cloud Foundry. So here, we can see the micro Bosch deployment manifest. We just need to prepare the release information in the manifest to tell Bosch what to deploy first. And as we said, and also the jobs we want to deploy. For the first step, we want to deploy Bosch itself. And finally, we must config the OpenStack relevant properties in the manifest, including the endpoints and all the key names. And after that, we can use Bosch in need tool to deploy Bosch first. This process may compile and install your OpenStack CPI on your local machine and use this CPI to talk to OpenStack environment. Now, the Bosch in need will try to create one virtual machine to deploy Bosch in OpenStack. Okay, so now finished. Then we go back to OpenStack. We can see that there is a new virtual machine created and all the Bosch components now is ready on that virtual machine. So we go back to the console and use Bosch client to target to the Bosch director. Now we can see the detail Bosch director information, the IP that we just, this is the floating IP that we just assigned. And this is the UID of the Bosch director. And now the first step is finished. Then we can see the second step. The second step is after the Bosch is deployed, then we can upload the Cloud Foundry release and deploy Cloud Foundry on OpenStack. It's only one virtual machine, the running micro Bosch. First, we want to upload the stamps out, the VM image onto Bosch director. And Bosch director will try to upload the virtual machine image to the grounds. And then we can upload the Cloud Foundry release. It's a table. Here is lessons learned. The Bosch deployment manifest can be very long, although it's a YAML file. For instance, the Cloud Foundry deployment manifest we used in this demo can contain 2,000 lines of code. So it's not suggested to write the manifest manually. Cloud Foundry community provide a useful tool named the speed to help you generate automatically of the manifest. So what we need to do is just to prepare a YAML file. This file is short to list all the properties needed for Bosch to deploy the Cloud Foundry and then use speed command to generate the manifest. Here we can use this command to generate the full version of Cloud Foundry deployment manifest. And then we set this deployment manifest. Now we deploy the Cloud Foundry on OpenStack. You can see that during the process it will create a lot of virtual machines in OpenStack. Now the process is finished and we go back to the OpenStack console. And we can see that there is a lot of virtual machine created. So now we can target to the Cloud Foundry system and try to push one sample application to verify the installation of Cloud Foundry. This step is just to target to the Cloud Foundry environment. Use Cloud Foundry client. Okay, this is the final step to verify the Cloud Foundry deployment. We all know that for almost all the past system the most simplest way to use past is to push some applications into past. Here in this demo we prepare a Hello Cloud Foundry Ruby web application and we try to push the source code into the Cloud Foundry environment. This is the folder that we put all the source code and then we use the single command cf push. And now we can see that for each of the application deployed in the Cloud Foundry system it will be assigned a unique URL. So we can use this URL to visit the application. Now we can use curl command to pin the application. So the application responded Hello Cloud Foundry. It proved that the application is running in the Cloud Foundry and Cloud Foundry is running on the open stack. Okay, so this is the demo part. Okay, so for the next session I will hand over to Dr. Max. Thank you Tom. So what you just saw was when everything works the reality is it doesn't always work and the reason, well I guess we'll find out some of the problems that we've experienced. But importantly what I want to mention is that all of us participated as part of the Bosch team itself. So we spent time at Pivotal and Tom and Edward even have contributed in the past, I would say six months directly into the Bosch backlog. So they have a lot of experience directly working with Bosch and various other clouds. And what we've always seen and certainly I have also when I was working with the Bosch team last year is that open stack was the problem child. We had IBM of course here because we believe in open stack so we'd love it to be the best cloud. But the reality is it had a lot of issues. And some of it were based on instability of the clouds. We tried various different providers. There's no need to mention their names but including IBM's, you know, clouds and there was always a problem. And every time a new release of open stack would come out there would be even more problems. So what we kind of thought it was maybe we should survey the community and find out also what kind of problem they have. So maybe as part of the survey and as part of our effort to try to get open stack to be the best, we can identify maybe the top problems and then see if we could solve them. So that's kind of the results you're gonna see here. And then after that, my colleague Edward will go over some of our experience directly trying to solve some of those problems. So this survey was done to about, to various communities, so Bosch, CF and then internal groups inside IBM, SAP and various other places. So what you're seeing here is the first time we're releasing the results. So the first thing we ask is, you know, of course we wanna identify who you are if you're responding to the survey. The survey was open so it's not targeted to people. And we found that about 60% of them are operators which is exactly who we're trying to target to the people that are managing Cloud Foundry. Of course we want to make sure that they are familiar with CF because otherwise that particular response may be less interesting. And we found that the vast majority of them were sort of familiar with the different concepts. So that was good. Another problem is of course, because we wanna focus on open stack, we wanted to make sure that the survey responders were familiar with open stack and we could see that about 50% of them felt they were intermediate users. So not newbies, you know, for open stack. And then of course since the whole point of the survey is to try to find out if they experience any problems, it was very interesting to see that no, very little number of them, like 5% experienced no issues. Most of them experienced some issues, some of them saying significant issues. And we've also collected a lot of the responses in terms of what kind of issues which I'll show you a little bit. So of course, you know, what we also wanted to know is if they had to customize open stack to get it running. And about half of them you could see had to make some special customization. Ideally, you would want your cloud to be, you know, out of the box, something like CF which is a complex distributed system should just deploy. Why would you have to customize open stack for a complex system? I thought, oh, we're building a cloud, right? We should be able to deploy those things. So we're finding out that it's not quite there yet. And some of the things we're seeing, as you saw, is that a lot of them are seeing instability in open stack. You remember this complex web that Tom showed you has different components that are pinging each other all the time, right? Because we want the system to be open running. So the VMs have an agent on them, at least the Bosch board, right? That's communicating with the director to find out what it needs to do next. There's a bunch of jobs running on it that are being monitored. So when one goes down, it resorts them. Then you saw there was also the console agent and various other agents. And then there's also the health monitor. So the system is, I wouldn't say it's self-healing, but it's trying to be self-healing. So it's trying the best it can to keep itself open running. So because of that, it is the network to be open running. It needs enough storage. It needs the VMs to be running. So when things go wrong, they kind of restart. But when things go wrong so much, like it becomes unstable, you kind of get stuck. And you have to redo your deployment or restart. And we've seen this multiple times. So this is the pain. So this instability of OpenStack, at least in the various versions in the past, caused us in the community a lot of issues. Another very important issue that we've experienced and that the community has also experienced is issues setting up networking on OpenStack. And I think this is an issue where a lot of people are shaking their heads that other people have experience as well. And I guess with the keynote today, we saw that a lot of focus is being spent on the networking part of OpenStack to make it a little bit better, hopefully, that will be fixed in future versions. But it's definitely a real issue right now. And you could see there is a few other problems, of course, that they've experienced. Finally, we wanted to also figure out what version and most people are actually using newer version. I mean, of course, they would see they're not using much newer than when we did a survey, but you can see most of them are on Juno and Keto. And like the demo we did was on Keto. And of course, another point of interest, we wanted to see, were they just using remotely managed, meaning using big vendors like IBM or HP or SAP to run the OpenStack versus their own local? It turns out that most people are running their own local. I mean, this beauty of OpenStack is you could have it any way you want. And what we saw is that they're mostly doing local management, more than 50% of them. So with that, let me pass to the lessons learned from our experience. So what you saw was the community's experience. And then after that we'll conclude and take some questions. So Edward. Thank you, Max. So OpenStack and Cloud Foundry are both complex platforms. So to keep it simple, we start from a very basic environment. We use the DevStack and very basic tools, Bosch, to deploy Cloud Foundry on the one box. So here all of the work are based on the very basic environment. Here's the summary of our work. We have successfully deployed OpenStack release, including Icehouse, Juno, Kilo, and Liberty OpenStack release. The Cloud Foundry looks at this environment in the CPI level. So from this point of view, I think it's thanks to a component of Fog, which encapsulates the OpenStack API in such a way that it is very stable for Cloud Foundry's Bosch project to consume the API. So we can see the result is very stable and very encouraging us. But we still have some lesson learned. We will cover later. And also a second point is when we deployed the OpenStack environment, we found the configuration that the basic primaries of local accounts is almost the same except the branch name. So it is pretty amazing, I think, at least from this point of view because it is very basic. And last one, I think on the deployment manifest, we didn't make any changes on the deployment manifest for release of OpenStack. So that means it is pretty efficient for us to start this work. And it is very quickly for us to deploy the environment. But the first one is the most difficult one, I think, because you have a lot of configuration problems and a lot of troubleshooting, we will cover later. The first lesson there, I think, you need to prepare a network environment for this box to deploy Cloud Foundry or OpenStack. For Cloud Foundry, we need to launch more than 16 VMs as a basic deployment. It requires the IP, two floating IPs, one floating IP for Director, another floating IP for the Cloud Foundry to use to intact with the outside of the world. And we also need to the static IPs and private dynamic IPs in OpenStack. So here we can see the two YAML files. One is for Director, another is for the Cloud Foundry itself. This network configuration is very basic, but when you go to your cloud, it will become very complex. You have to deal with the full environment. Another lesson there is about disk space. Because this is one box, we didn't find this problem in one box, but we found it on another box because the disk space is not enough. We, I have spent a couple hours to work on that. Finally, I found the one box we used have not enough disk space on the computer, on the normal computer. No, because all of the VM flavors of the Cloud Foundry used has a disk size, which accumulates more than 680 gigs. This is a very basic, minimal requirement for Cloud Foundry. But for this box, it doesn't have so much disk space. So it always reported no valid host that was found. When I get deep into the desktop, I found the problems. The fails the disk filter of the Nova scheduler. Because the Nova scheduler will use the default disk allocation ratio config, which is equal to one. So it is assuming that the disk space requirement will be satisfied, but it is not true. So I changed the primaries of disk allocation ratio to increase the, to allow the overcommit of the disk space, which does work. But for, if you have a more big environment or over stack, this might not grow a problem. And here we also see the quota of tenant. By default, DevStack have defined the very basic quota of each tenant. For example, it only has 10 instances for each tenant. So before starting master, increase the quota of the tenant. For VCPU, we did a simple calculation here. If we have, for Cloud Foundry, we have four small flavors, 10 medium flavors, and one big flavors. So we got 40 VCPU requirements here. So if your Cloud Foundry environment is bigger than this, and you must consider the quota of the VCPU. And for RAM and instance quota, it's the same. This is very basic requirement. And just as Tom has showed, we need to create a script group because Cloud Foundry has a lot of traffic, network traffic between each component. So you must allow the communications between the components. So this is some example of the port you have to allow opened. And this is very interesting why we need internet access. Because we, we use DevStack to install OpenStack from, by putting the source code from GitHub. And so it will install that it will pull the packages also from the internet. So, and also Bosch can be deployed by using the UIL to provide the address of the stem cell or Bosch release. So if you want to use this feature, you must allow the internet connection, the internet access for your OpenStack environment. And another very annoying issue in China, we have a great war. So some of the service are blocked, such as S3. Some of the Bosch stem cell are blocked by S3, by firewall. So it is very tough to deploy, deploy the Cloud Foundry. And the network issue, in the journey of this work, we found a very, we accidentally found this problem. We ran into a very special situation that we have two box. Two box has fully deployed Cloud Foundry and OpenStack. So, but when one box is ready, we deployed a second one, the Bosch director, we blocked, we cannot successfully deployed Bosch director due to the network issue. Because we use these two box in the same network and use the Linux bridge plus flat GHCP. And also we use the same, deploy the manifest, which has the same network IP address configuration. So the two director will conflict and they will, you will see the Bosch CLI will fail to connect the Bosch director due to the IP address, the physical address or the MAC address of the director are inconsistent state because of the LPE package was sent out through the bridge. And it get a two bridge to reply the different MAC address of the, of the, your client, your Bosch CLI. So it always shut off the connection between, shut off the session between your Bosch CLI and the director. At last we found this problem and use a different subnet with a different IP range to resolve this problem. So even in such a simple, such a simple environment, you still be careful about the network isolation. So make sure you have a robust network isolation. When the, when your cloud grows, you have to plan how to manage this network. So what's the next? So we have, we will continue trying to deploy Bosch for every new open stack release and continue survey the community's feedback to see where is the problem exists. And also we will come the community to give us feedback. And I think for the community, they should drive more coherence from the perspective of API and the backward compatibility. And also open stack community should engage other communities such as Cloud Foundry to help to resolve these differences. Here is a list of some reference, the Bosch IO official website of Bosch project and the Bosch init tools. There is a very good article to introduce the Bosch project. And also there is a CPI pipeline is online for you to take a look at the project status. The last one is our video. Thank you. But if you do have questions, maybe we can entertain a couple. Maybe the question is how to get started or how is your, let's say from the point of view of operator. How can we quickly get started on Cloud Foundry, for example? So for Bosch community, we have a lot of tools to help the starters. We have a box named the Bosch light to install all the Cloud Foundry nails all in one single virtual machine. So it's a very easy tool to try Cloud Foundry deployment using Bosch, so you can take a look at it. Is that available in the Bosch website? Yeah. Right. That article, there is an article link, you should read it, it's very helpful for you to start. Yeah, Maria with this, so this is very good to start with. Any other questions? Say that again? Yes, of course, yes, of course. Thank you. Well, thank you for coming. Appreciate it. Thank you.