 Okay, we are going to start. Can you hear me? So we are going to give a couple of more minutes for people to take a look. Okay, so we are going to get started. Hello everyone. Thanks for joining. My name is Animesh Singh and these are my colleagues Daniel Crook and Jason Anderson. We work for IBM Cloud Team and we have been focusing on Cloud Foundry and OpenStack integration over the last one and a half years. Now today we are going to talk about how to optimize Cloud Foundry deployments, large Cloud Foundry deployments on OpenStack. Now before we go into some of the details, I would like to ask how many people here have actually either used Cloud Foundry or have deployed Cloud Foundry. So that's roughly around 40% of the room. How many people have actually heard about Cloud Foundry? Exactly. I mean, that's the reason you're here. So that's good. One more quick question before we get into the contents. How many people here have actually experienced number 11 syndrome? It's a pity you didn't because this is a term which I just coined two days ago. So let me explain to you what it means. We all have bosses, right? So you're given a task by your boss and you make a list of the 10 things you need to do so as to accomplish that task. Great. You do the task to perfection. You go back to your boss and what does he say? Why didn't you do number 11? So that's what I mean by number 11. So I'm hoping all of you at some point have experienced that number 11 syndrome. So with that, we'll give a very brief overview of how IBM Cloud is being built on open-source standards and technologies, be it at the infrastructure as a service layer, with OpenStack, Open Daylight, OSLC, or at the platform as a service layer with Cloud Foundry. Database is like MongoDB, portable workload templates like Tosca and even at the SAS layer where we are focusing a lot on HTML and JSON standards. As you are seeing through the conference, IBM is committed to the success of OpenStack. Throughout IBM, we have around 380 IBMers actively working on OpenStack in different capacities. And specifically with respect to the OpenStack community, that has grown exponentially from 8,000 individuals in March of last year to around 16,000 individuals as of now. And now begins our number 11 syndrome. So in last year, at the beginning of last year, IBM announced that all our cloud products will be based on OpenStack and seeing the success of OpenStack, we were asked by our boss or a set of bosses in terms of exploring Cloud Foundry and see whether we can replicate the same success at the past layer. We started looking at the Cloud Foundry. We deployed it in-house. We did a couple of tests. So what is Cloud Foundry? It actually passed a lot of those tests which we wanted. First of all, it's open-source, truly open-source. Secondly, it fits the real paradigm of platform as a service. As a developer, you're not worrying about virtual machines, network storage. You just go with your code. Cloud Foundry is smart enough to detect what runtime you need. You need Java, you need Ruby, Node.js, and you tell what services you will need to actually run your application. It will provision and bind it to them. So it's open-source. Target's developer needs. And third and most important, it actually has a very strong and vibrant community. As of now, we have more than 1200 contributors actually working on Cloud Foundry and more than 700,000 lines of code have been written and contributed back. In this chart, we give a very brief architecture overview of Cloud Foundry. At the very top is the router layer which is responsible for managing all the incoming traffic to your applications. User Authentication and Authorization, as the name suggests, it is responsible for making sure users are authorized and logged in. And Cloud Controller is actually the brain of Cloud Foundry. It's responsible for managing the lifecycle of applications. Applications are deployed on DAs, which are dedicated VMs, which get provisioned for applications, and the containers are actually embedded in what they call build packs. So, for example, for Java, you will have a different build pack. For Ruby, you will have a different build pack. Similarly, Service Gateway and Service Connectors are very much self-explanatory. They allow you to actually write your own services and broker it within Cloud Foundry. Health Manager here is responsible for making sure that everything is consistent. And if there is a mismatch, it reports it back to Cloud Controller to take some actions. Great. So we investigated this. We went back to our boss and said, Cloud Foundry seems like a very, very good fit. Do you think he was happy? What's the next thing he asked us to do? Can you integrate it on top of OpenStack? So we started looking at how Cloud Foundry can integrate with OpenStack. So Cloud Foundry uses a tool called Bosch, which is a release deployment and lifecycle management tool which allows you to deploy Cloud Foundry on different IS platforms, be it VMware, be it Amazon web services. And in case of OpenStack, it uses OpenStack CPI to do it or Cloud Provider interface. Now we'll go a little bit into the vocabulary of Bosch itself as we will be using some of these terms through the presentation. So what you see here is a job. A job is basically a collection of software packages with some config files. For example, if you're deploying MySQL, that's a job in itself. If you're deploying PostgreSQL, that's a job. And a release is a compilation of all these different jobs together. A stem cell is a base OS image with DEA agents embedded in it. And all the Cloud Foundry deployments actually go on the stem cells which are converted into VMs. And finally, at the heart of Cloud Foundry deployment is the deployment manifest, which combines all these things together, takes the jobs, takes the releases, takes the IS infrastructure information, you construct a big manifest file and tell Bosch to deploy Cloud Foundry. And in the end, Bosch deploys virtual machines on your IS environment and converts it into a Cloud Foundry deployment. This is a sample manifest file. Just to give you an idea, some of these manifest files can actually run in 2,000 lines, if you see. And part of the reason is if you're deploying multiple jobs, multiple software packages, if you're doing a Cloud Foundry deployment with caching service, queuing service, a lot of other services, plus you need to feed in a lot of infrastructure information. For example, in case of OpenStack, you need to tell what is OpenStack network information, what are the credentials to get in, what is the public-private key pair, what four different jobs will be running, etc. What are the used IP addresses? So there is a lot of information you need to manually enter in this manifest file. This chart gives a very brief overview of the OpenStack CPI or Cloud Provider interface. Now, any IS provider who need to deploy Cloud Foundry as we talked in the beginning, they need to implement this interface. In case of OpenStack, it has been implemented using Fog Gem, which is a Ruby gem written to interact with OpenStack APIs. It provides some of the basic functionalities like launch a VM, delete a VM, provision a disk, attach a disk to a VM, etc. Okay, so we figured out all the things we will need to actually deploy Cloud Foundry on OpenStack. But there's one more thing, a vanilla install of OpenStack is not sufficient. We need to customize OpenStack, we need to make sure some of the prereqs of Cloud Foundry are set up on OpenStack. In this case, we needed to make sure that there are static and floating IPs which are enabled, there are persistent disks. The VMs being deployed have outbound internet connectivity. You can get rid of this by doing a lot of work in terms of setting up a lot of Ruby gem servers inside, etc. Custom flavors need to be created for different components of Cloud Foundry. For example, for a router, for a DEA, which will host the applications, you need to create custom configuration sizes. And we need to make sure that there are security groups which are specialized to handle Cloud Foundry traffic in terms of making sure there is connectivity to the router from outside or the VMs in Cloud Foundry are able to communicate to each other. So great. We completed the integration. We actually deployed Cloud Foundry on OpenStack and we are happy. We go back to our boss. What do you think he will say? What to do next? Anyone? Integrate, automate. So we completed the integration, but he said, okay, there are a lot of manual tasks we needed to do. Specifically, when we talked about the manifest file, you saw that we needed to enter a lot of those entries. We needed to feed a lot of parameters. So with that, we actually went down the journey of automation. So a few of the things we did on the automation side. We extended the fog gem, which we saw the OpenStack CPI uses to discover a lot of information from OpenStack. So rather than going to OpenStack installation and finding out information in the config file or going through the APIs, we actually used this fog gem and invested a lot of effort in writing automation code to discover a lot of these things, like key pairs, flavors, your VM subnet, where is the DSCP server running, where is the DNS server running, what are the used IP addresses, because all this information we need to tell Bosch. So that was one step in terms of automating it. Step number two, as I talked about, a vanilla install of Cloud OpenStack is not sufficient, you need to customize it. So we actually invested again effort in terms of customizing OpenStack environments using the same fog gem. So what we are doing is we are actually creating key pairs, we are creating different flavors for routers, DEAs, cloud controllers, and we are also creating security groups, etc. We are upping up the tenant quota so that it can absorb a Cloud Foundry deployment. All through automation using the fog gem. And finally, automation tip three, which in our experience was the most cumbersome task of all, creating these manifest files. As I mentioned, some of these manifest files could be very, very long. So we invested a lot of code in terms of taking all the information we are getting either through discovery or through the setup we are doing and feeding it in the manifest file and generating it. So that was a big back through in terms of getting this done very quickly. And one more thing, I talked about the stem cell or the base OS images which a Cloud Foundry deployment uses and converts it into different components. Now in our experience, the community images, the stem cell images which are there, they didn't work as is. We needed to make some modifications. So we have invested in writing automation code around it as well. I mean, there are different ways to get around it. We experienced that the FS tab entry was not there. So you could either inject it manually or if you are just using a dedicated OpenStack for this only, you can change the OpenStack configuration parameter or the best of all, you could have Cloud in it installed in your image and it works in conjunction with the OpenStack metadata service. So great. We did all these things and finally we are happy. We are happy that we actually automated the whole Cloud Foundry and OpenStack installation and we are getting this done now in two hours. We go back to the boss and what do you think the boss will say? Number 11, guess it, scale it. So with that, I'll pass to my colleague Daniel Crook to talk about scaling Cloud Foundry and OpenStack. Okay. Can you hear me okay? Okay. So we thought our boss was going to be pretty happy with the integration, the automation, but what they really wanted to know was how do you actually run applications on this thing. So we asked what sort of, what would represent a large cluster out there and we picked 1000 applications written in Ruby, PHP, Java as a benchmark. You showed them using 252 megabytes and taking this list of this number of 1000 applications, we wanted to see what sort of resources we would need to support that. So for that, we estimated there'd be about 60 virtual machines. 20 of those would be high memory virtual machines that run the applications. There's 11 of the core fabric, the routers, the controllers, the authentication and there's the services themselves, brokers and then the open source services like MySQL, MongoDB, Postgres, Redis. And for each of those VMs, we needed an OpenStack infrastructure that provided them with local storage, root storage, ephemeral storage as well as blocks, object storage and of course all the network private IPs and public IPs to support the system. So we took this attempt of 1000 applications to run on an OpenStack cluster and because we were also experimenting with OpenStack in addition to discovering Cloud Foundry, we basically took the simplest possible approach we could to stand up a cluster. So we had basically all of the open stack components on a single controller node. That's all the databases for the components, the messaging, Cupid, the APIs, the scheduler, the user authentication and of course the UI. And we moved the rest of the servers within our cluster to be compute nodes for the actual virtual applications to host those Cloud Foundry components. And we also reserved one node only for storage which had Cinder and Glance. So it's a great first approach to working with OpenStack, getting the simplest thing to possibly work but it wasn't very good for what we wanted to do with 1000 applications. So the architecture there that we have used, it had plenty of single points of failure. There was also some bugs within Folsom and how it worked with Cupid that caused that head node to run out of memory. That in turn caused some issues with lost messages. Bosch not only provisions resources that Cloud Foundry needs but it also checks on their state, checks to see that they're there and a lot of messages and hits the API database result from that. We also, because it was also an open stack environment with a lot of other innovation experiments going on to clean the area of big data and analytics, there was a lot of network traffic that went through that head node that caused a lot of congestions, that caused Bosch to time out and caused other problems with that deployment. So in addition to being kind of an inefficient system that we first came up with, we also had implemented it directly on bare metal Red Hat servers. We had selected the nodes that were going to serve the roles and that's what it was. We could add more compute nodes but that was about it. So based on the lessons we learned there, trying to run Cloud Foundry at scale, we came to a more distributed architecture of open stack that kind of mirrors what we've done with Cloud Foundry. So we've broken the controller node out, we've kept the databases and RabbitMQ nodes, we moved away from Cupid, we have three of those each in clustered formation, and we've got the Nova API, the scheduler, Keystone and Horizon still on a head node but we've duplicated that onto two machines. And in front of all that, we've got a bunch of load balancing nodes implemented with HAProxy and KeepLive. Those are all virtual machines which gives us flexibility to move them around, add to them, and we remained with dedicated servers for the storage nodes and the compute nodes. So this gives us the IO we're looking for as well as the better networking. So this open stack cluster then gave us what we were looking for for our Cloud Foundry experimentation. It provided a foundation that was highly available, very flexible, could add more nodes, we could move nodes around, and it was also very scalable to match our experimentation with Cloud Foundry past 1,000 applications. And of course we could have replaced an upgrade much easier than with the single point of failures we had before. So there's a bunch of other IBM sessions, two other IBM sessions that are going on tomorrow that go into a little more detail about the hardware, the actual specs on the racks that we used to our morning, and in the afternoon there's a talk on the testing and the architecture decisions that went into that HA architecture. Okay, so we now could integrate, automate, and scale. So we were pretty proud of ourselves, but there was still some more work to do. And what was that? Optimize. So Jason's going to take you through. Alrighty, so now our boss thinks we tirelessly work and we have one more task to do. We have to optimize. So we have our Cloud Foundry environment, it's up, it's running, we now have the ability to scale it out, but now we've got to fine tune it. We have to make it, to tighten it up, make sure it's secure. So the first thing we want to do when we're looking at optimizing Cloud Foundry is to go and look at where the bottlenecks are. We want to make sure that those bottlenecks are alleviated. So the first is the communication between Bosch and Cloud Foundry, or Bosch and OpenStack. So the OpenStack API is what Bosch uses to request the infrastructure that the Cloud Foundry VMs will be using. So Bosch, though, in its implementation, is pretty aggressive with the API. It says, hey, OpenStack, I need a VM, and then it consistently goes and says, are you done yet? Are you done yet? Are you done yet? And so OpenStack and the API gets overwhelmed, especially when we start talking about 60 VMs and we start scaling out. So in order to fix this problem or at least allow OpenStack to allow this communication, we had to update the rate limits that it has in place. So to do this, you can go into the NOVA configuration files, and you can edit and update them. So right now we've set them to 9,999, and so that seems to be enough to allow this scale version of OpenStack to be deployed. One of the other optimization bottlenecks that we have to perform has to do with Bosch itself. So as these Cloud Foundry VMs are coming up, there's an agent in the stem cell that Animesh had talked about. Now, this agent communicates using a bus inside a Bosch called NATS. Now, originally, NATS was only a single node could be deployed at a time, and this one node starts to be overwhelmed, especially when we start looking at 60 plus VMs that are all communicating through this one message bus. So in order to allow this communication and allow a little more lag, inside of the NATS configuration, we increase its timeout ping to 30, and that allows enough for this scale for every VM to be able to communicate. They just have to wait a little bit longer. And so actually now with the latest version of Cloud Foundry, version 169, they have the ability to cluster NATS and to scale it out a little better. So we'll try again with the new version, and hopefully we don't need this work around with the ping interval increase. All right, so we've looked at the bottlenecks. We've taken care of a couple of them, but now we need to start looking at a little more some performance during the deployment time. So there's an issue we found with the implementation for Nova Network and the way that it implements the security groups. So a security group in OpenStack allows what VMs to communicate with each other, what ports are allowed to be accessed. And so there's a couple of ways you can set these up. The first is by the source address. So you can set up a network saying anything in this network is allowed to come in and access these ports. Now there's an alternative to this. Instead of allowing a network or an IP range, what you can do is actually specify any other VM in this current security group can access another VM in the same security group. Now the implementation for this basically has to go every time you add a new VM to a security group, has to go to all the other VMs in the database. It's put messages on the bus and says, okay, this VM is now allowed to talk to all the rest of them. And so you can see this is a problem as we start scaling out, because for that 60th VM, it has to go to the 59 other ones and update it to allow access to that one VM we just added. And we add the 61st, 62nd, it starts to become more and more of a problem. So we were seeing deploy times for the VM increased by two to three minutes, just based on these security roles. So instead of using these name-based security groups for the source, it's much better to use the IP-based security roles. And the other thing related to scheduling and how do you distribute your Cloud Foundry VMs, you want to make sure that you're using a OpenStack scheduler that evenly distributes VM load. So when Bosch comes to OpenStack and says, hey, I need 60 VMs, if you have a scheduler that fills up one compute node, moves on, fills up another one, you're going to be deploying maybe 20 VMs on one compute node, 20 VMs on another, and you're not really using the full infrastructure that you may have. Maybe you have 15 compute nodes, but you're going to have a lot of empty ones if you're not using a distributed scheduler. So the default scheduler, luckily, has that capability to distribute the load. So as long as you don't customize or OpenStack for those different types of schedulers where it's going to co-locate all of your VMs, you're going to be okay. All right. Now let's start looking at the security. So the guiding principle when you have a secure system is you want to make sure you're only opening security holes where they need to be. You don't want them to be bigger than they need to be. You want them to be as tight as possible. So the first, and when we were first doing some initial tests, we were guilty of this ourselves, but you want to make sure your credentials are as tight as possible. So instead of using the full OpenStack admin credentials to access and deploy VMs for Cloud Foundry, you want to use, you want to create a tenant and use that tenant credentials specifically for that purpose. Similarly, you want to not open up all the secure ports on all the VMs. It's not a good idea. Instead, what you want to do is to get the list from docs.cloudfoundry.org of all the ports that Cloud Foundry needs to operate, and then go in and create your security groups logically. You can actually see, I think on the website, even they have a list, a screenshot of how to configure OpenStack, essentially step-by-step, to set it up correctly. So it's not that not that much effort to do it the right way. Another point on security, you want to make sure that you have good network isolation. So you want to keep your Cloud Foundry VMs on a separate network than your OpenStack management, and you don't want to allow necessarily communication between the two. So there's no reason why the router component in Cloud Foundry needs to talk to the Cloud Controller of OpenStack. So why allow it to do that? So you want to keep them on separate networks and only pinhole security points when you need them. One good example of this is Bosch. In order for Bosch, which is deployed on Cloud Foundry, to communicate and get those VMs deployed for Cloud Foundry, you need to have pinholes opened up just for that reason. And then finally, you want to also keep the services that you're going to be using in Cloud Foundry separate as well. So you have your OpenStack management network, you have your Cloud Foundry core components on a separate network, and then finally you should have a third network for the services that you're going to be deploying. So for instance, if I'm going to have a DB2 instance that I'm going to expose through Cloud Foundry, I want to make sure that's on a separate services network, and then I pinhole access as needed for Cloud Foundry to communicate with that DB2 service. Great. So that's how we optimize our Cloud Foundry OpenStack environment. So we come back to the boss. And is the boss happy? Yeah? Well, you're going to have to join us at the Cloud Foundry Summit next month to find out. So for those of you who don't know, there's a Cloud Foundry Summit, similar to the OpenStack Summit. Everyone from Cloud Foundry goes to the summit in San Francisco. It's June 9th to the 11th, and a lot of great talks, a lot of great speaking. So if you're in the Bay Area and wanted to come by, feel free to. There's a registration discount code here on the right hand side. You get 50% off. Great. So with that, we have any questions in the audience? If you wouldn't mind, come up to the microphones and then we can hear you guys a little better. I'm not sure if this is a question. It's more of a statement. I assume you will be taking all the information that you learn optimizing Cloud Foundry for OpenStack and work that back into Cloud Foundry. Because I know we've had implementations of Cloud Foundry that, again, those timeout values are critical. Once you start doing anything with it, we've had problems just deploying Cloud Foundry because it hits that timeout and the whole thing has got to be redone from scratch. And it seems like you guys have actually solved a lot of these problems that are not, it's not an OpenStack problem. It's a Cloud Foundry problem. Right. Definitely. I know just finding the timeouts and how to change them was one of our main problems. So contributing back, at least in documentation, for how do you change these timeout values? Because you don't want them larger than you necessarily need for a small deployment. But when you start looking at scale, nobody's really documented what needs to be changed for it to work. So definitely contributing back through documentation and where it makes sense, especially with NATS, they've implemented the clustered NATS. So now hopefully that's less of an issue even in a configuration level. That's good. Could you make sure is Micazon as well, please? And I would like to add to his comment as well. I mean, we do a video blog of our Cloud Lab site and where we post a lot of these materials as we are finding out. So we'll definitely make a note of it. Make sure that it's in the slide, which we send out. And we actually have some sort of documentation which is then contributed back to the Cloud Foundry community. And part of our larger organization, they are committers to the Cloud Foundry project. We are committed to improving OpenStack as well as Cloud Foundry. So we've got dedicated resources to do that. So we'll take that back as a request. Thanks. Can you talk more about your NOVA scheduler? Are you using an anti-affinity filter for scheduler? Right. So right now we're just using the default scheduler. We haven't had a need or a reason to change it, really. So you've been finding that it distributes the DEAs well? Right. Yeah. So I mean, as far as Bosch is concerned, because Bosch is really requesting the VMs. So its job should really be to know where the VMs are being located and therefore distribute the DEAs, for instance, a little more. So it's actually Bosch that's handling it. Right. So that's a gap currently in Bosch. But that's something that's also being looked at. Great. Any other questions? All right. Well, thank you, everyone. Thanks for joining. Appreciate you guys joining. Thanks.