 My name is Michael. I work for SAP since a couple of the years in the Cloud Foundry team and I would like to talk With you about our experience with admin UI and why we ended up deploying it as our Cloud Foundry application The admin UI is a very useful application for Cloud Foundry operator That allows you to visualize a lot of metrics and operational data For example, you can look for a specific application and see in which Diego sells The application is deployed or on one Diego cell have a list of All the application that are deployed on these cells and many many other useful information our old architecture with admin UI was Bosch release if you're not familiar with Bosch, it's a great tool to manage life cycle for virtual machines so our application was deployed in a virtual machine and we had to keep care about the Ruby Framework actually Bosch release it's open source. It's on the Cloud Foundry community, but it started to be an effort to update and operate as to just update the source code of application. We had to spend a lot of time To create a new Bosch release to test it to deploy it and every deployments takes time as it's a Virtual machine that gets deployed and not just an application So we started to think to use a better Framework a better platform to deploy our application and We started with Bosch because there was already a Bosch release available for the admin UI We are in a team where we deploy Cloud Foundry with Bosch. We deploy concourse with Bosch We deploy our monitoring and logging stack with Bosch so we started using It also for the admin UI, but why not making use of Features of Cloud Foundry also to deploy our application We deliver with our product so Why not on Cloud Foundry? This way we would get The advantage of simple and faster updates and we would stop caring about the underlying framework So we don't need to update Ruby anymore. We don't need to check for vulnerabilities on the Framework we are using What do you have to keep in mind when you want to port the admin UI as a Cloud Foundry application? First thing is that we don't really need persistency. The admin UI has a database, but it's used as a cache So we don't really need that to deploy it on Cloud Foundry What we need is still the same setup of a UAA client So this is as it is described on the Admin UI GitHub project you can put that in a shell script. You can use that on your Automation system like concourse on Jenkins before Pushing the admin UI. We're not going to talk about that We also need a special space on Cloud Foundry. You cannot just deploy the application on a trial developer space Because you need special permission. The admin UI needs to have connection to various Cloud Foundry endpoints to gather its data What we also need is to move the configuration file from a file in the in the file system to Environment variables. So we would like to to Set the configuration of the application Using the Cloud Foundry manifest. We want to use only one place to put all the configuration of the application Not use the manifest for just the Cloud Foundry Related part and then push the application. We've already the configuration inside. We want to keep the source code and the configuration in two different places Let's start with security groups. What do we need to? To run the admin UI on a on a Cloud Foundry space We need a special security group to allow connection to the Cloud Foundry Cloud controller and UAA databases. We need to connect to that to get information about org spaces and more We need to connect to the NATS endpoints. That means we need to Connect to the private IP of NATS VMs in your Cloud Foundry deployment We need to connect to the API endpoints and this means the internal IP of the API virtual machines not the Public one which is exposed by the load balancer. After that the main issue is the configuration file the resolution is not the best, but You can trust me that this is just a YAML file with a list of Configuration parameters you need to set to run the admin UI. One can be the Address of the database, the Cloud controller database, the UAA database and so on So the admin UI is expecting a YAML file in a specific place of a configuration folder But we want to set that through the Cloud Foundry manifest So how can we do that without touching the source code of the admin UI? This is the template of the manifest we are going to use to deploy the application of Cloud Foundry So we see that we pass all the configuration we would like to set as environment variables and then We launch the application with a different command. So you would expect to start immediately the Ruby process of the application, but instead We launch a bash script This allowed us to to read from the environment the configuration. We need a write that to a file This is a really simple bash script that allows you to run ERB to evaluate a template and write the configuration file inside the container This happens inside the container after the application is pushed and then we start the application This is an example of a ERB template So just the configuration file plus the ERB code that allows us to read from the environment and write to the file This is a really simple trick to immediately push an application that needs a configuration file but We don't want to too hard code the configuration into the application. We push we want to pass that as a Environment to the application manifest. This is a suggested way From Cloud Foundry and has other advantages like you can look at the configuration by running CFM and so on So to do that we just need to push our application with two additional files inside The the source code. We just add something. We don't touch the existing code. We add our star script and The ERB template that will read from the environment Let's look at how does it look like on the file system From the bottom we have a star script. We just saw the configuration template the ERB file. We just saw We have a manifest template But we will use to generate all the configuration we need for example We can read that configuration from a Cloud Foundry deployment or from other sources We have a nap YAML file that just includes the URL On GitHub of the admin UI source code and the commit ID So we don't even need to keep the admin UI as a submodule as it was on the Bosch release So if we need to to fork the application for whatever reason we can just change the URL on the YAML In the action folder, we have three files that are just batch scripts That allows us to create the manifest using the template Every one of you Sure, if you deploy a Cloud Foundry application, you will have your own way to to generate the manifest I don't think you just hard code in the manifest all the Credentials, but you just have a template and then you add the credential depending on where your which Cloud Foundry deployment are going to to use We have a deploy script that just will clone The the source code of admin UI and will add the stab folder and then run a CF push We have also pre-deploy script that we'll just check for the existence of the security groups that we need This is the comparison with a Bosch release You can't even read but there are more than 60 files Most of them are generated by Bosch that's true, but we still need to take care on a lot of things One of that is the the Ruby framework this way we make use of Cloud Foundry Ruby build pack and we get for free on updates on every updates of the platform What we gain also is the speed of redeployment of the application Before it was over five minutes to start and recreate especially when we need to update also stem cell the OS of a virtual machine we use to To run the application and it takes time It's a single turn so you cannot even use that during that time and if you want to try to change something it's still An effort to change the Bosch release and to start it again with Bosch instead of just a CF push But we still have a performance issue it's one of a issue that we didn't talk much before and It's about the load of this application that minimize a great tool, but when it's deployed on a Productive landscape we start to get a lot of loads because the application subscribes to Doppler and as an admin user and it gets all the metrics and all the logs all the information from a Cloud Foundry deployment this means that We get one metrics for example as a latency for each and every get requests that hits your landscape On a productive landscape this it means a lot And the admin UI is a Ruby process on a single thread And it's not able to handle all this information so As we are not Ruby experts we started again to think how can we improve the performance without touching the code of application can we put something in Front of that and filter the data. We don't need before it reaches the application and This is the idea to put another Cloud Foundry application written in go Between the admin UI and the Doppler endpoint This way we would be able to Drop all the metrics that are about to reach the application and we would improve the performance So how can we do that? first of all we need to Make the admin UI target our nozzle our application our go application instead of Doppler endpoint in Cloud Foundry Then this nozzle should be able to authenticate to Doppler, but we don't want to write the admin UI token admin UI credential into the nozzle because this is a Cloud Foundry application and it's public So how can we do that? but if we just set the Doppler endpoint on the admin UI to be the nozzle instead of Doppler the admin UI will already Send to the nozzle the authentication and the subscription ID necessary to request data to Doppler This way on the nozzle we can read the subscription ID and the token used by the admin UI that is thinking to It's believing to Connect to Cloud Foundry, but instead it's connecting to our application with that as a proxy Our application we use this data and we connect to the real Doppler endpoint At this point we have two Websock and one client and one server on the nozzle the server will wait for the admin UI connection and The client will connect to the real Doppler When the connection goes through we make all the Swim of events pass through a filter written again in go and this is open source. You can try that and Our filter will look just for value metrics and for container metrics This is the type of information that the admin UI cares about On the value metrics we also filter out all the latency information That is the big chunk of data that is hitting the The admin UI that we don't need at all What are the results of this? nozzle put in between the admin UI and Cloud Foundry Doppler Well, the admin UI on a productive on our productive landscape landscape Took six seconds to load just the first page and this becomes frustrating when you need to look for some data You need to change that up and it takes three four seconds every time After putting the nozzle in between we managed to take that six seconds down to point three seconds Looking at the logs of the nozzle we see that we filter out around 97% of the events that would have hit the admin UI instead On our landscape we have around 800,000 events every 30 seconds and Most of them are useless for the for the admin UI Looking also at the CPU load of the application. The blue one is the Default tab the new I just deployed on Cloud Foundry that is always at a hundred percent But that was exactly the same case also when it was deployed via Bosch on a virtual machine Also, because if you add more cores doesn't it doesn't help as the the application is a single thread Using the nozzle we managed to get the CPU down up to 40 percent. We still have spikes We still hit the second bottlenecked on the on the line, but this way We have better chance to not lose data Because when the admin UI is taking too long to process data from Doppler Doppler we start dropping out information before sending it The nozzle is open source Since one and a half day probably you can look at that you can try it you can deploy that as a Cloud Foundry Operator and you can put that in between the admin UI and your Doppler endpoint. You just need to set The address of a real Doppler endpoint on of your Cloud Foundry installation on the nozzle settings and Tell the admin UI to connect to your nozzle which is a Cloud Foundry application What would be the plans for the future? First of all would be to make the admin UI more Cloud Foundry friendly What does that mean? We'd like to get read of this workaround that we use to wrap the admin UI with a shell script to read the from the environment and write the configuration file it would make sense just to Make a pull request on the application and make that able to read from the environment directly But this way we managed to deploy that in a couple of hours without Asking anyone we are asking the owners of the application to change anything and this is a great thing Would like to try to use a persistent database even if it's not really necessary This could improve a performance as every time we restart the application. We lose the the database and Then takes maybe one minute to to to be populated again The last thing would be how can we scale it up? This is something we still have to start to look into that and would be really interesting because we managed to to to solve a bottleneck, but it's not enough and Getting on bigger and bigger landscapes. We will still have issues in the future actually the nozzle is also the nozzle it's a workaround as Logregator now has announced a way to Filter the metrics to request only what you need But the only thing it's implemented right now I think you can only ask either for logs or for metrics you cannot ask for specific metrics and our issue is that It's the metrics of the issue and we are getting all the metrics and we don't want the latency metrics that are 90% of the metrics we receive and this will Hit the application real hard To conclude We managed to move out the configuration file of the application Into the cloud fund we manifest without touching the application code at all This is not the nicest way to do that, but you can do that in a couple of minutes you can just Add a shell script that will start instead of Directly starting the application and you can read from the environment and write in the configuration file We managed also to greatly improve the performance of the application again almost without touching the source code say almost because The application did not support an override of a topler endpoint So we had to add this as a configuration parameter, and this was a really small pull request, but this way On cloud fund we managed to create a new application on a new a different language that's Improved the performance of an application we don't own and this would have took much more time to try and fix that in the In the ad new eye This is not of course the final solution, but this as well allows you to fix issues in a quick way and Without touching the code of the original code of the application With that you have any question The admin UI is only an application with admin rights So it's for the operator that has an overall view of the whole landscape It's for the platform operator is not for for users Yeah, if you don't have any questions I have one and that would be Why would you deploy the admin UI which is for a cloud fund re operator on cloud fund re does it make sense? You do not deploy something that it's a monitoring tool on cloud fund re to monitor cloud fund re but this Makes sense, but it depends on your usage of this application This is a great application to browse for data to get information of On something, but it's not a monitoring tool. So we have a monitoring stack. We have a login stack, but Chances are if cloud fund re is Experienced several problems. We don't need the admin UI to fix it. This is just when you have to find out Where an application is deployed on which cell what are the application deployed in a cell? That's maybe it's behaving in a strange way and you want to see what are the application affected and check if it's really a problem or not and so on but Everything you can do in the admin UI you can do also Directly from the source of information. It's a really painful to go into the database and look for data to start Getting all the metrics from firewalls and do the query on your own, but you can still do that without the admin UI so in our experience the trade-off of Deploying that for on cloud fund re and the all the time we save for the lifecycle of application. It's fine for us to Have a risk of not having it available when we have Huge issues on cloud fund re as also it's only a single application So we don't have many instances deployed, but that it's fine for our use case I was at that code talk and that was really really similar one thing would be I'm not sure if the top plugin of CLI is Showing all the information or just the top of the column you you're ordering to So on the admin UI you see everything so I can show on the very first slide there is a screenshot of the application and You see that we have 1100 tabs of the application list so we can see everything you can look for everything you can see for Crushed application and so on you can have the all details of what's happening on the landscape. That's why also it's The performance is an issue because it's getting all the data of your cloud fund re deployment Yeah, yeah, okay. Thank you very much and we'll be around here if you need something else