 Hi everybody. Welcome to my talk. Upgrading from CF release with CF deployment. The other way. My name is Janik. I'm from Iwala. And so which one of you is still running CF release? Quick hands up. Two guys. Somebody knows anyone who's still running CF release. Okay. So the talk is still relevant. Okay. Some quick disclaimers. This talk comes from a project we did in the beginning of the year. Back then the transition repo was not as good as it is now. So we came up with our own way to upgrade from CF release to CF deployment. The second disclaimer is every deployment is a bit different. So you might need to adapt some of the points. I will talk now about and all the guys who still raise their hands. Now it's the time. So it's CF release is deprecated since February. So slowly you should come to a point where you upgrade. Okay. Let's get started. Our environment at this customer was a vanilla CF open sec. We had a staging foundation only for testing purposes and two production foundations. They ran at the time CF release with the RP version 280. And they had configured an external blob store so non-internal which is a bit necessary for how we did this. You may also do can do it with an internal blob store. But on this talk I will have the requirement that you have an external blob store. So the customer had some requirements during the update. There must be a zero downtime for all apps. So the apps, the customer should not need to intervene in any case. So no CF scale, CF push the apps. They don't should have to switch their services. So they also wanted transition to Cretab which they didn't have at the time. And they wanted a possibility to roll back the upgrade if the upgrade fails. During the time the CF deployment transition repo was not that good as I said. So it wouldn't check all the requirements for us. So we developed another process. We wanted to keep the blob store so we were just in dependency so we can switch back and forth between our two deployments. We might create a databases and deploy a new CF foundation from scratch. First we have to do some preparations. We upgraded the Bosch director which was also outdated, deployed a Cretab, created new networks for the new CF deployment and also adapted the security groups. Then we had to extract some properties from the old deployment like the decryption cipher and the BBS key. And we also started on Cretab because now we have Cretab and everything is secure. Yeah. So we started with this. This is a schema of the CF deployment. And at first we wanted to disable the Cloud Controller API so we have a change freeze so nobody can push on your app or scale the apps. Mainly we wanted to prevent any changing in the blob store because if the blob store IDs change and we have a database backup with an old IDs then the container won't come up in the new deployment. So we have many possibilities to disable this access. We can kill any local answer or DNS. We can disable the Cloud Controller itself. We can also ask our devs not to push any apps during the time. What we did, we disabled the load balancer for the system route so nobody could talk to the API. Next we made our backups for the database and we also did it for the blob store in case anything goes wrong. We can always restore our blob store. So then we had to deploy a new database. The customer provided a managed database which we could use but you maybe could also deploy the internal database so you would at first deploy only the database and scale all other components down. So we would deploy a new database. As I said, you maybe also could deploy an internal database if you scale up down all other components during the first deployment. Restore your backup and after that deploy the rest of the CF deployment which was the next step. So we deployed the new CF deployment and enabled the API access to this deployment so we could check and monitor the containers on CF. So now we have a running second system and we used, for example, CF top to check if all our containers came up back again and could div between the old CF release and the new CF deployment. Check our monitoring in Prometheus and if everything came back up we could switch over. So now we would disable the app domains on the old CF release and enable them on our CF deployment. And now we basically did a transition from CF release to CF deployment. At any time, if we now recognize something is wrong, we can always switch back to CF release because except of the load balancers we changed nothing with our old release. So we have the raw back functionality, some up and downside of this method. For once it's very easy. You can make mistakes and don't have to fear that your production goes down. It's easily testable in production. So what we did many times we tried to visit different domain. So we spun up a new CF deployment and checked if all the containers came up again and could repeat this process until it got perfect. It's a CF downtime upgrade and we always have the functionality to roll back. On the downside, it's very resource heavy because you have to deploy a second foundation for the platform you already have running. It's not an officially supported way. And as I said at the beginning, you might have to make changes for each cloud. So that's all my slides. Do you have any questions or yeah? So the question was just because I have to repeat it. If you had to do it again, would you do it the supported way or this way? It depends on the platform. If you have the resources, as I said, it was very pleasing process because we could test the production upgrade without any fear of breaking anything. So I would prefer this way, but as I recognize when I prepared this talk, the transition repo got better. In fact, it got very good. So I think both ways are okay. But I would prefer this way again. Hi. We are in a similar situation, but I saw in your screenshot there are 84 applications in this cluster. We have a cluster with 90,000 applications. It's a bit bigger. How did you, what kind of things did you run into that you didn't expect on beforehand? Okay. The screenshot was from some of our test platforms, what not actually cloud from the customer. Something that we did run into which we did not think of first was once the BlobSides. So even if we, the first time we didn't disable the API access, though some developer or some pipeline found a way to update the app and we had apps that didn't come up after the upgrade. We also had some trouble with changing the domain names, but it was a dependency of the customer. They lost some of the secrets for their certificates. But other than that, we did not have any bigger major issues. Maybe one addition to that, the cool thing was they could retry and retry the upgrade because they made always a new copy of it. So they could look, is the copy okay? Are we fine with the process or do we try to repeat that with another solution? Any other questions? Thanks.