 Hello, everyone. Thank you for coming to our talk. It's entitled, How Cloud Controller Makes the Foundry Go Round. My name is Ravashi, and I'm an engineer on the Cloud Foundry API team. And I'm Simon Nel. I'm from IBM. I've been working with Ravashi and some other people for a year on the Capitime for a while now. As the developers of API, we mainly work on this component called Cloud Controller. And many times, we just refer it to CC in short. So over the past years, we've been soaked up a lot of information about Cloud Controller. And we play our hand in some of the bit changes and refactoring in the components as well. And so I'm really excited to give you guys an overview of what Cloud Controller is and the role it plays inside of your Cloud Foundry deployment. At the end, we're going to wrap up with a quick synopsis of what new features you can expect from the API. So Cloud Controller is the API that orchestrates some of the most common interactions inside of your Cloud Foundry deployment. Most of you are probably using version 2 of the API in your current Cloud Foundry production environments. If not, then you're a bunch of really big risk takers. But what it is, is a RESTful API with a, and the app itself is written in Ruby with a few supportive components written and going. But what you really need to know about Cloud Controller is that it is the source of truth in Cloud Foundry. With over 300 endpoints, it can tell you a lot of information about your apps and services. And it helps you to do a lot of things inside of Cloud Foundry. We like to think of it as the mastermind or kind of the brain behind the operation. So let's take a look at what's inside of this brain and from the perspective of a Bosch deployment. So there's three main jobs here. We have the worker, the clock, and the API. The worker performs all background jobs. We have two different types. We have the global worker and the local worker. The local worker really just takes local disk files and uploads them to the Bob's Store. Simon's going to walk you through that in just a bit. And the global worker does pretty much all other jobs that happen with external resources. The clock simply schedules some of these jobs and does some daily app cleanup and expired droplets, things of that nature. And the API is literally everything else. This is where all of the complex logic inside of CC lives. So when we're talking about CC responding back with helpful information, this is the thing that's going out and talking to the external components inside of the Cloud Foundry deployment to get you back the information that you need. So let's take a look at what a typical, really simplified version of a Cloud Foundry deployment looks like and how CC interacts with some of the key players in this. So you have a Cloud Foundry deployed and you have some developers who are using this installation. Every request that they make will go through a routing layer. The go router will direct traffic to CC appropriately. It's then Clock Controller's job to talk to the UAA, which is our identity service, to make sure that the permissions and scopes line up so that we can respond appropriately with the information. And then based on the user's request, Clock Controller may need to reach out to its backend and talk to DEA or Diego, whichever the backend provider is, and speak to them about the running instances, maybe get some metrics about the application itself. Well, thanks, Fashi. That's a pretty good overview. So now we have an idea of what the CC actually is and what the role is in the ecosystem. Let's take a deeper look at how Cloud Controller actually do its internal operations. We can take a look at a sample, such as a pushnab. Pushnab is the most important feature in the Cloud Foundry. In the user's perspective, you do a safe push, your source code, and you just get uploaded to somewhere in the cloud, and you'll run it somewhere in the cloud. It just work, unless you write really back code, and it doesn't even compile at all. Usually, that's not the case. So Cloud Controller is the thing in the background actually to make sure all the components, it's orchestrating all the component, make sure they are working together as a whole, and make sure everything goes smoothly. So let's take a look at this. Here, we have CLI command, safe push, the testing app door. We used to test environment all the time. And on the right hand side here, we have the output for when you do a safe push in CLI, this is the output. You see there's a lot of stuff going on. And in our case here, you have more surfaces going on. That's the reason that is we turn on the trace, and CLI itself essentially is just a Cloud Controller API client. So if you turn on the trace, it's going to show you all the requests it's making to Cloud Controller, and it's going to give you all the responses from Cloud Controller back. So if you do a push, and you will notice, one of the first things that it does is to create an empty app. It does it by doing a post request to Cloud Controller to the V2 apps endpoint, along with the body, the metadata, the name of the application, your memory limit, this quota, et cetera. And when Cloud Controller receives this piece of information, what it does is it's going to create an entry in the CCDB and remember that. CCDB is the database the Cloud Controller uses to remember basically everything it knows about the state of the world. So after that, the next thing the CLI does is to create a route. So we can route traffic to the application. And again, it does it by posting to endpoint V2 routes. And when CC receives this piece of information, it's going to create an entry in the database and remember that. So after this is done, CLI, the next thing it's going to do, so far we have an empty app with a route, but in no way they could relate to each other anyway at all. So it wouldn't function unless we bind them together. So that's exactly what CLI does next. So it binds the app to the route and you make a request to CC. And again, CC is going to remember the information in the database. But at the same time, this time, CC is going to broadcast the newly created route to the rest of the world. So components like Go Router will knows which application to route the request to when it's being requested. So now we have the app and we have a route to route traffic to the application. And it's probably time for us to upload a source code so we can run the application, right? Not really. And the reason of that is if we upload all the files every time we want to update our application, it's problematic because you will just end up uploading a lot of applications to Cloud Controller. And that's not the best way to leverage your system resources. So the system is actually quite smart about that. Before the CLI uploads anything to Cloud Controller, it will do these things called resource matching. And you'll actually see it in the CLI output, right? Basically what it does is the CLI is going to tell Cloud Controller what files I'm about to upload. And it's comparing to what's already uploaded to Cloud Controller. So this is how it goes. The CLI upload a file list saying, this is the files I'm going to upload now. And in return, Cloud Controller is going to come back with a list of all the files you already know about. So by comparing the two lists, now you know the delta. You know what file you're supposed to upload. And CLI is going to go ahead and upload those files that's not on CC. And it goes to the best endpoint for uploading. And all the app bits and file source code is in the format of multipart form data. And after the files are uploaded, CC will receive all these files. And you will create this in what we call a package. A package, essentially, is just all the source code and the app bits that your application needs to use in order to be running. And CC now is going to take this package and store it into something we call Blob Store. The Blob Store is basically a backend storage that Cloud Controller uses to store things like all the files, all the built-in packages, and the binaries. So now we have the source code, the app, the route created. And it's really time for us to start the application. To do that, CLI is going to tell CC what we want the application's desired state to be. So it goes like this. It's telling CC I want my application state to be started. And the CC is going to take that request. And it's going to remember that in the database. And at the same time, now we know the desired state is to be starting the application. And how do we meet that in reality? So it's time for the CC to send a request to the backend saying, I want to stage and start the application. When the backend receives that message, what it does is it will go to the Blob Store and take the package. We upload it earlier and compile that into something what we call a droplet. So basically the droplet is something what we get from compiling the package and you become the binary version of the package. And it is a binary that's ready to be turned into a running process later on. Once the backend have that, it's going to store the droplet back to the Blob Store for later use. And at the same time, it's going to use that droplet to turn into a running process. And that's how your app gets staged and run in one of those backend containers. The process of staging and running actually is quite complex. And it gets carried out in many different ways, all depends on which backend you're actually running. The choice right now is a Diego or DA. And based on those two backend providers, Clock Controller has to pivot and accommodate both of them in different ways. And so as Simon mentioned, the two options are DEA and Diego. DEA used to be the only choice for a really long time. And it is written in Ruby. It stands for Droplet Execution Agent. And it describes some of the processes that Simon just went over, staging an application, running them. Diego is also the alternative. We've seen a lot of Cloud Foundry deployments transitioning into Diego. It's written in Golang. But during that transition phase, we still have to support both of these backends. And really, there's four main operations that Clock Controller needs from any backend. And the first is to be able to tell it to go and stage a thing. The second is to be able to tell it based on a user's request to start and stop an application. It needs to be able to get information about running instances from the backend, just overall metrics and the health of the application. And lastly, it needs some mechanism for consistency. It needs to know that the ideas that it has inside of its database are actually what's going on in the current state of the app. So let's look at what Clock Controller has to do to be able to support the DEAs. Since this was a choice for the longest time, the DEAs, Clock Controller took on a much more dominant role of the staging process. CC took a look at the available DEAs and start filtering them based on memory, stack, and disk. And then it would randomly select from this pool of DEAs to go and stage the application. And then at this point, whenever a user would want to start or scale or start an application, it would interact with the CEA to do that. I had mentioned that CC needs to be able to make sure that what it has its ideas in the database are the actual state of the application. In DEAs, it relies on a component called the Health Manager to do that, or HM9000. And HM9000's job is to watch the app very closely and to make sure that it is the actual state that CCDB knows it to be. When it sees a discrepancy between these two worlds, it'll actually hit the CC API to make the two worlds consistent. As for Diego, Clock Controller needs to be able to talk to the Diego API, which is called the BBS. The BBS creates something called LRPs and Tasks. LRPs are long-running processes. And in order for Clock Controller to communicate with Diego's BBS client, we use something called the CC Bridge. And the CC Bridge consists of four components. We have InSync, Stager, CC Uploader, and TPS. So InSync does the job of listening to Clock Controller for desired apps and going over and creating those LRPs. It also makes sure that the two worlds are consistent by carrying out the necessary actions to make sure that Diego has the LRPs that CC needs or takes away the LRPs that it doesn't need. Stager handles the staging requests and response between the two parties. So when CC sends a staging request, the Stager will go and ask the BBS client to make a task. And the CC Uploader plays a really minor role here where it just mediates the uploads for staging between the two parties. Lastly, we have the TPS, which in conjunction with InSync kind of works as to be the HM9000 of the DEAs, where it looks very closely at the apps and reports to CC if there are any crashes and also just provides general information about the running LRPs to Clock Controller. So that's how CC is able to swap out the two backends. For DEA, we take a really big role in selecting the DEAs and filtering them out. And as for Diego, we heavily rely on the CC Bridge to be able to talk to BBS. And this is just one of the flexibilities that Clock Controller is able to provide in your Clock Foundry deployment. So what we've actually just illustrated was one of the prime sample of how CC ties together all the CF component and perform major operations. So another example is how apps and services work together in Clock Foundry. Users, they like Clock Foundry for many reasons. But one of the major reasons is how easy it is for you to extend the functionality of the application by just leveraging some of the CF services. And CC is actually a component behind the scenes doing a lot of heavy lifting for the operation to operate seamlessly together. So let's take a look. So in order for application, for a service to be used inside Clock Foundry, the service itself has to be integrated into CF. We do this by implementing a component, what we call a service broker. What the service broker essentially is, is a bridge between the service provider and Clock Foundry. With the broker in place, now Cloud Controller can learn about the service offering, service plan that's provided by the provider and make it available in the marketplace. When it's done, if an application want to use one of the services, you can make a request to Cloud Controller and Cloud Controller is gonna bind the application to the service. And now, every time when the service is up and running, the Cloud Controller is gonna make sure the information it needs, like such as SSURL or user credentials are available to the application through environment variable. And with all this information, now the apps can leverage the service directly. So you might ask, what if the service on the leverage is not part of the CF ecosystem, right? It's outside of the marketplace. You can do that, right? So for example, now you have a service outside of the Clock Foundry ecosystem and you have a CF app trying to use that. So usually, you will have access to a credential that you need to access a external service. So what you need to do is to take that credentials and create something what we call user provided service and bind that to the application. So again, when you do that, every time your application is up and running, Cloud Controller is gonna make sure your application will receive the needed information in order for you to access the external service. So with this, you can scale your application up and down with different instances. And all the instances will receive the same kind of information, right? And whenever the information to access the service changes, all you have to do is update the user provided service. And the same thing goes for if you have an application that's living outside of CF and you have a service instance, you want to leverage, right? And so what the problem here is like, in Cloud Foundry, to use a service instance, you will have to bind that into a running CF app. But you don't have one, right? Your app is not running inside Cloud Foundry. So what you can do here is to create something called a service key, right? What the key, service key is, it's basically the key for you to ask Cloud Controller to get all the credentials you need to access a service instance. So you can request with the key and in return, you get the credentials you need to access a service instance. With those information, you can just go into CF and access a service. So so far in this talk, we've been talking about all the work currently Cloud Controller has been doing under the cover to allow all these major operations. The CAPI teams actually hard at work at trying to provide some new features. We've been working on some like a set of brand new features and one of these new features is the V3 API, right? The V3 API is the result of some major overhaul of entire Cloud Controller and lots of refactoring and that's how we got here today. We had to do all those in order to implement some of the V3 features we want to implement. The biggest problem was with the current V2 architecture, if we want to have a running application, what you have to do is of course, you'll push and you push all the source code and you'll become a package and then you compile that package so you get a binary droplet. And from there, you can spawn a running application, right? So what if you want to spawn another process type, a different one, from the same application? Well, this is what you have to do. You're gonna have to go through the whole stages again and now like to get that two different process type. And what happened is now you end up with three different packages, you end up with three different droplets just to run some different process type from the same application. So that's quite poorly designed and it's not the best way to leverage system resources, right? And another big problem is like here, in the packages and the droplets is really tightly coupled to its own application. So that doesn't really give us the flexibility we want to implement some of the V3 features we wanna get. What we really wanna get is this. We want to do a single push and then we can get many processes out of that. And at the same time, we want to break up all this tight coupling between the packages, the droplets and the processes, right? To achieve that, we need to re-architect how control itself as a whole. And now we've done that and the big benefit now we get is now we can interact the packages, the droplets, the process independently because now they're on their own entity. And that opens the door to a lot of new features such as like now like Booge Green deployment, something like that, we can bake that into the platform instead of relying on some third-party solution. And also come out of the V3 work track is a one-off task. So from the droplet, you can spawn up different process types, but at the same time, you can also spawn up a thing called task. So the major differences between the tasks and the processes, the process is a long-running process. If the process goes down or crash, it's gonna get resurrected and it's gonna come back up and live again. But for a task, you run at most once and once the task finished doing what it is designated to do, it's gonna go away and return all the resources back to the pool. So with this new feature, it allows us to do something we couldn't do before such as now you can probably periodically collect some intelligent data from your running app, your production app or you can run database migration or backup your database, things like that. So these are some of the new features coming out from the V3 API work, but the team's actually also working on some other new features and Yufash is gonna talk about some of them. Yeah, so Cloud Controller is going through some more changes. One of them is providing support for new storage providers as Google Cloud and Azure Storage become more prominent. We're opening our doors to that. And the other big thing that we're hoping to do is some of the resource matching and uploading bits that Simon had described during pushing, we wanna take that heavy lifting out of Cloud Controller and have someone else completely do it. So there's an entire team called Bit Services that's dedicated to this goal so that we can kind of make that less complex inside of Cloud Controller. We also want to start moving the CC Bridge inside of Cloud Controller and this is the components that are needed as a dependency. They brought it in through the Diego release and Cloud Controller needs them to be able to interact with the BBS client on Diego's end. We're hoping to make that move into Cloud Controller so that CC can directly talk to the BBS client without needing this extra layer of dependencies. But this is just a really short list. There's so many more changes that are coming to Cloud Controller. And if you're interested in learning about those, we'd love to hear from you in our Slack channel. We're at the Cappy channel and we also have our repo open source on GitHub. You can submit issues. We love pull requests, things of that nature. We often dedicate, daily, we dedicate half of the day just to community and a full pair to interrupt every single day. So we're more than happy to interact with you guys and talk to you online. Thank you for attending this talk. Are there any questions? Questions? Yes. So, let me do that. So the question was, how exactly are we gonna spawn different process types from the same drop rate? So, let's say what we want to do first. Like right now, if you want to... So currently with the V2 architecture, a drop rate can only spawn a single process type, which is a web process. And if you want to spawn up a worker or some other worker do their base migration, you couldn't. You will have to do the staging and push the app all over again. From the V3 perspective, you have a droplet and it's actually independent from the package and the process itself. This droplet, all you want to do, all you need to do to spawn up a new process is to provide a metadata, like how much memory you want, how much, what kind of start command that you want. And then you will spawn a process with the starting command that you provide and that is one kind of process type. And you can define a worker and other process type with different process starting command. Not necessarily. They might not in the same container. Yes, what's been done for CLI to expose all these new features natively in the Office of CLI. But I don't think it's in there right now, but it's been worked on. Yes. So the question was, can you scale a process within the droplet? So a droplet itself is basically, you can think of essentially just a V2 application. But the difference is, we can now launch different process types. So that being said, it's a V2 application, that means that you can do any operation that you can do to an application today, including scaling or changing memory limit or this quote or things like that to a V3 process. Any other question? Yes. No, so the droplet is compiled from a package. So what you upload to the package is what you're gonna get for the droplet. You're just not gonna change the content of the droplet. What's changing is the starting command and all the metadata. Any other questions? All right, so welcome to be here. If you have any more questions, you know where to find us, Slack or email anywhere you want. Thank you. Thank you. Thank you.