 Welcome to our talk on metrics driven blue green deployment using spinnaker's cloud front integration. I'm Amit Nambiar and this is... Hi guys, my name is Passup Achala. We're platform architects out of Australia, so not far from Philadelphia, but yeah. So glad to be here. Thanks for coming. Great. Yeah, it's been, this is probably going to be the most challenging talk of my career. You know why I was here one hour earlier and just got the thing working, the presentation. My laptop died. Just imagine what I'm going to do. I don't have access to GitHub, but hopefully we'll get through this one. Yeah, our focus is going to be on cloud font reintegration for spinnaker. Pass will be talking about how the different abstractions built into spinnaker connect to cloud foundry application runtime, right? How do they map to these operations in cloud foundry? I'll be doing an end to end demo of a pipeline using canary analysis to deploy apps onto cloud foundry at the end, right? So spinnaker is a continuous delivery platform which has its origins at Netflix, right? Can we have a quick show of hands on how many of you have heard about or worked with spinnaker before? That's great. Maybe we should get them to come out. Yeah, the CF integrations for spinnaker has some great features added recently and I hope this talk will take you through those features and, you know, get you excited enough to start using them. It's a great addition to the cloud foundry platform to have this integration. This is a great book from Netflix about continuous delivery with spinnaker. It talks about the intent, the problems of doing continuous delivery at scale and this quote actually caught my attention. It says, we defined a paved road that encapsulates best practices for teams wishing to deploy to the cloud, right? That's exactly what spinnaker provides. It provides a paved road for you to start with your continuous delivery process, right? So I just wanted to draw an analogy to the cloud foundry application runtime which in my opinion also provides a paved road for enterprises to run all your apps, right? So what I'm trying to say here is the CF integration for spinnaker is what it's doing is bringing together the best of both worlds for you and I think the cloud foundry community is going to benefit greatly from this integration. With that I'd like to hand it over to Paz who's going to talk to you about the how spinnaker and cloud foundry, how these mappings work. So guys I'll start with this slide. I actually stole this from the spinnaker team but I like it because it says spinnaker is a multi-cloud delivery platform and what that means is that spinnaker is going to manage these resources on your behalf on ISs such as GCP, Azure, AWS for example but it also does it on platform as a service with App Engine and we're going to be focusing on what it does with cloud foundry in this talk as well as container orchestration systems such as Kubernetes as well. But what I wanted to bring up here is if you think the cloud provider interface for cloud foundry so you've got Bosch that has a clean separation between the actual clouds that it's talking to through the cloud provider interface, it's a similar concept with spinnaker. There's a separation and there's accounts that you configure to be able to talk to the clouds themselves. So how is it actually added? If you speak to the spinnaker team and you tell them that you've got spinnaker installed and you don't have HELL, they'll probably tell you that it's not supported. So that's a critical part to spinnaker. You can install spinnaker without HELL but HELL is the actual command line of the administration tool for spinnaker itself and it's the way we actually add a cloud foundry provider is using HELL now. I say here greater than 1.15 but anyone that's on latest versions of spinnaker will know that HELL is actually at 1.18 now and 1.19 coming out soon but you actually just enable the cloud foundry provider like that and you actually then go and add a cloud foundry account. There's really only three pieces of information that's really important. This is the same as when you're using the cloud foundry CLI. That's the API endpoint for your cloud foundry instance, the user and password. The environment can be also added but not necessary. Now if you're using a pivotal cloud foundry instance, a commercial version of cloud foundry itself, Apps Manager URI and Mectrics URI gives us more capability within spinnaker to be able to reach out to Apps Manager and Mectrics of your apps at the appropriate time. So what does this mean for spinnaker? Once I've added cloud foundry, what are some of the things that I can do? So we've got here the ability to do manifest based deployments. That's obviously quite important because all you see it pushes them more than likely going to be using a manifest. Blue-green deployments, multi-foundation view of applications. This one's quite important because I can add multiple cloud foundry accounts to spinnaker. I'm not tied to just one, so I can see all my applications across all my foundations. As you expect application management actions, all these ability to enable, disable, destroy, all these actual terms that I'm referring to are spinnaker terms. They actually map back to cloud foundry. So for an example, enable and disable is actually a CF stop will disable the app. CF start will actually enable the app. So spinnaker make those calls on your behalf. So sometimes when you're first working with spinnaker and cloud foundry, you may find that the terms kind of what does this actually mean, but they're actually mapping to what you're doing with the CF CLI. Recently, they added the ability to actually deploy services as well. So it will actually reach into the marketplace of cloud foundry, display those services in the layer to create stages in your pipeline that create services as well, which is quite neat. And then obviously as creating a server group when we do a deployment itself, we can actually bind to those services that have been created. I'm actually going to show you live on the pipeline what that may look like. And so once you've enabled cloud foundry, you'll get your little cloud provider down the bottom there and you'll be able to select cloud foundry when you create a new application. Okay, so resource mapping, an account in spinnaker is referred to as a cloud foundry user account. That's exactly what that is, a login to cloud foundry. It's not an org, it's not a space. The cloud foundry integration will actually go use the permissions that it's got from the cloud foundry account user that you've got and traverse everything and pull those applications into spinnaker for you. The load balancer is actually the cloud foundry route. There is no load balancer that you create within cloud foundry as you're all aware. The router itself has routes. So in spinnaker terms that's a CF route. Server group is probably the most important concept in spinnaker with cloud foundry because that's mapped to deployment of a specific cloud foundry app to an actual foundation. At that point it goes into an org and a space and I'll show you that live as well. A region is the actual CF space mapping and an instance as you can imagine is a single container instance of your app. So if you've got an application deployed on cloud foundry and you have 10 instances, spinnaker will show those 10 instances as separate instances, exactly the how it shows in cloud foundry. So there's also operation mapping and I'll show you that live as well. So the ability to deploy, roll back, resize, we'll do a CF scale for example, enable, disable. I've already spoken about those. Destroy, completely remove an application. You can do as well. Clone is another one that you can actually take an existing server group that you've had deployed and just clone that and use that as the base of what you're going to do for your next deployment. Map load balancer and upman load balancers are quite new. That gives you the ability to add routes or remove routes from your application. And I'll show you that live in a pipeline that I've configured and terminate an instance is actually go and kill that specific application instance, not the app, the instance that you want to terminate. And I'll show you in the UI how that looks like. So there's a couple of screenshots here down the bottom, but I'm going to show you live. So let's just do that. I think that's part of what we're doing next. So here's the sample pipeline that I'm going to show. This is not a real pipeline. I created a pipeline that will show some of the stages that map to cloud foundry that you could use in a pipeline if you're doing a continuous delivery or some sort of deployment to cloud foundry. So my one here, I've used the deploy stage, the deploy service, disable server group, resize server group and map load balancers. So I've used quite a few of the different stages that map to the cloud foundry stages themselves. And so if you look at when I do a deploy, this is actually a CF push as you can imagine. When I select the stage resize server group, you can see there in this image, I get the ability to change the number of instances, maybe the memory, maybe the disk. I don't know why you'd ever change this, but anyway, it's there. And so as a result of that, I can make changes live within my pipeline as well. I need to show this slide because I like this image. That's the only reason you're seeing this slide. Okay, so let's go to a demo. And why does everyone when they have to do a demo say that, you know, I hope the demo guides with me. If it doesn't work, it's not the end of the world for you guys, but it might be the end of the world for me. So let me go to, we've got a couple of different tabs here. I believe this is my one. Okay, so if I go to, now remember I told you I added a cloud foundry account and that account has gone and pulled in all the applications that I have access to privileges and then pulled them into spinica here. And one of these here is this little application called CF here. I feel better because this means that my back end environment's working. So not smoking mirrors. Okay, so let me start at some of the stuff that's spinica and some of those terms I spoke about server groups, apps, instant instances, some of those commands that we can run. So let's have a look at this. If you ever look at this in the server group, we're going to start there. So is this big enough by the way? How's that? A bit better? Cool. All right, so this is a Node.js app, something new to me. I've never demoed a Node app, ever. So I've been working with Pivotal for five years. So you might wonder, why would he demo a Node app? Why not Pivotal love Spring Boot? Me included. The reason is when I start an instance in Node, let's be honest, it starts quick. If I started in Java, yeah, just have to wait a bit longer. So in the server group here, I spoke about actions. So let's go and do one of those actions now. I'm going to actually resize. Now, in the real world, you probably your administrators won't go in and manage Cloud Foundry this way and do resizes. Everything's done in the pipeline. So I'm going to go to the pipeline section. But what I do want to show is some of the features that are available. So I'm going to resize this, we'll make it one, for example, and hit submit. Now I'm going to go to here. This is definitely too small. How's that? So you can see now that this app, CF app, CD doesn't work in Cloud Foundry. CF does, though. And so you'll see now that it's actually gone from two instances to one instance, right? So what I'm going to do is while we're waiting for that to come up, show you some of the other actions that we're going to do. So we'll leave it at that versus changing it. It's been just giving me a nice UI saying it was successful. So I might have spoke about all these other commands. So you could destroy, for example, and completely take this Node.js out of Pivotal Cloud Foundry or Cloud Foundry in this case. So I'm not going to do that, but there's other different server group actions as well, such as map and unmapped load. I spoke already about clone. So here I could just add a new route to this application if I wanted to do it. So you can see our ties in nicely with Cloud Foundry. Now I spoke about actual application instances. Here's my last surviving instance. And so from here, I'd actually like to turn on with details. One of the things you can see here that's big enough is that actual ID there for the instance is actually the app with within Cloud Foundry. And right at the end, there's a little zero, which is the index of the actual Cloud Foundry application index itself. So if I have more, I have one, two, and so on and so forth. So here I can actually look at the actual routes if I can scroll down. And so there's the actual application if I wanted to go to that unconscious of time. So I want to show you just a quick, we all know there's application running on Cloud Foundry, nothing new. Thanks, Pass. Okay. So if I go to a pipeline, Spinnaker demo, this is how you really use Spinnaker, obviously. So in the pipeline here, I've got this little boring pipeline, but what it does do is show some of the, how is that? So for example, here, I'm deploying an application. And when I deploy an application, I define the reason I'm going through this really quickly is Amit's going to do a proper pipeline, which is going to do a canary analysis. And so you'll see all that sort of stuff there. He'll do it live, right? He'll like, he'll show his demo of the pipeline live. But what you'll see here is I'm using a HTTP artifact. There are many different types of artifacts within Spinnaker in terms of deployment. This one's the most boring, but it's the most easiest, right? Because I just HP endpoint to a jar, for example, and I can just deploy it. But here you get, this is the form based manifest entries that I can put in. Now you get a fair bit here, such as the environment variables, the ability to bind to services, build packs, health check, not quite everything. The manifest within Cloud Foundry are quite exhaustive. So in that case, you could just select artifact here and actually change this to actually go and read a manifest somewhere else. Again, use a HTTP artifact. So I'm going to cancel that. So what else do we have here that I could show? So resize server group, here's an example of what the end of deploying an application in my pipeline, I'm going to go from one instance to two instances, for example, or maybe a manual test has been run and we're all good to go. And so all of these stages are actually manual state judgment is something within Spinnaker, but all of these stages, such as deploy application, resize server group, add route to production, all of these are some of the new, so map load, but some of the new stuff that's come into Spinnaker around the Cloud Foundry integration. So with that, I believe I'm going to pass to Amit. He's going to show what you're really here for. And this is like how we do a canary analysis in Spinnaker for a Cloud Foundry app itself. Cool. Thanks, Paz. Thanks for the great introduction. So I want everyone of you to get a little bit creative. This app has been purpose built for this demo, but I'll explain the intent behind the app. So it's called Time-Till-Pickup Prediction Service. When you book a cab or Uber, you'll see the time the driver would arrive at your place when he's going to meet you. So imagine this as a backend service of a car or a cab hailing app, a bunch of microservices. One of them is this one, which is going to predict given a latitude, longitude, and where the customer is, how long is it going to take? And this has some very stringent requirements, non-functional requirements that a prediction request should finish in, say, 200 milliseconds. So why I'm building up this story is because I want to prove that, okay, with the canary analysis, the prediction time when it goes spikes up, it's going to fail and that change doesn't propagate into production. That's the whole idea I want to show here. So let's go through the Pickup Prediction Service Pipeline. This is what the developers are using right now to commit and propagate the change into production. So developers push the accord into the agit repository in this case, GitHub. We have set up a web hook to notify Jenkins about this change. Jenkins picks up those artifacts, builds the app, and uploads it into GCS. That's the Google Cloud Storage. We have set up a subscription from there for Google Pub's up to notify about that new upload into the bucket. I want to pause here and say this notification is going into Spinnaker, but Spinnaker has a lot of integrations with Jenkins, Artifactory, and other things. The demo we are running right now is in a unique environment. We are running it in a private lab which doesn't have inbound access. So we have gone with this kind of a setup, which looks complicated, but it need not be this complicated. So that notification comes in to Spinnaker, and Spinnaker realizes, okay, there is a new build available and goes to Google Cloud Storage, picks up that build and starts running the pipeline, which I'll be showing you in a minute. But what it does is it deploys a baseline version, canary version. I'll explain what these mean in Spinnaker or in continuous delivery terms or continuous deployment terms. And then it's calling Kanta. Kanta is an automated canary analysis tool which is built into Spinnaker. I'm just showing it outside of Spinnaker here to make it clear that it is using Kanta to do the canary analysis. But Kanta needs some kind of application monitoring kind of a tool. We have used Datadog in this case. It supports Prometheus, Neuralic, and all those kinds which fall in that category. So once that's done, if the automated canary analysis passed, then what that means is it can go into production. Then Spinnaker deploys that app into Pivotal application service as part of this demo, which is based out of Cloud Foundry application runtime. So that's the flow which we are going to show. So just briefly, Kanta is a platform for automated canary analysis. Canaries were formally used by miners to warn of dangerous gasses in the mines. So you can try and relate this to putting a new version into production. You are taking it in there and seeing whether it is okay to run this in a production environment. If it fails, you want to shut it down and come back as soon as possible. Canary release is a technique to reduce the risk from deploying a new version into production. It has some information about how it is done. I have a beautiful slide on that, so I'll just quickly pass through this one. I want to talk about this current situation we are in with this pickup prediction service. What has happened is the developers introduced a deep learning algorithm to predict time and it has failed miserably because it has spiked. The amount of time it takes to respond to a request has gone from 200 milliseconds to 50 milliseconds to like 600 plus milliseconds and that's how the canary has failed. This is the datalog matrix for the actual failed canary. Now let's look at the actual pipeline in Spinnaker. As you can see, this is the last run and the datalog grass which you saw was this failure here. So if you look into this canary analysis here, you can look at the canary summary there. That's the report of the failed canary analysis. You can see that the top one is the canary which is at 650 milliseconds and the baseline is at 250. So what I'm going to do just for demonstration purposes is change that algorithm to return sooner. It's just using a thread.sleep. I'm going to change that so that it just proves the point. What has happened is I've lost my credentials and everything to GitHub. I've somehow managed to get on the web thing. So I'm going to make a change here and commit these changes from here so that you know. What he's saying is his laptop crashed 10 minutes before. Yeah. Unbelievable. Not rebooted and we can come back, crashed, dead. That's what he's saying Australia, right, crashed. Oh, did you say something else here? I don't know. Every time I talk in anywhere I go, I say something and it's not quite right. All right. So what I'm going to do is so this is so what you see here, this is the time to pick up prediction service, right? So all it's doing is just a delay. Just as I said, its purpose built for this one, I'm going to change the delay to 200 to make it right, right? It was spiking. So I'm going to commit these changes. Okay. So I wanted to make another change, but can I do it at the same time? I cannot. So, okay, there is a UI for this as well. I'll show you the current, this is the current, this is the current, current pickup prediction service in production, right? It has a UI which is just a car. So what I wanted to do actually for the demo was change this color to green as well, just to indicate it's a blue green deploy, but I don't think I'll be able to commit both at the same time and get Jenkins to build it. So what I'm going to do is I'm just going to make sure the canary passes by just making that change. Are you guys following what I'm saying here? Great. Thank you. So I'm going to commit these changes which is going to kick off the pipeline, right? So this change, as you can see, a build has started here. While this is happening, okay, what's going to happen is it's going to build the artifact and then upload it into Google Cloud Storage. Then it gets notified into Spinnaker, right? While that's happening, you can see there are no builds happening at the moment here. In probably 30 seconds, you can see there's a pipeline, which is this one, which will kick off, right? I'll probably get started with this, go through the stages, as you and you will see the pipeline in action soon. So the first step is here, this is the configuration here. This is where I'm saying, okay, what is the trigger for this pipeline? As I said, it's a PubSub notification coming in from the Google PubSub subsystem, right? So that's the trigger for this. What's the first step I'm going to do? I'm going to deploy a canary and a baseline, right? Let me just pause there and talk about canary analysis. I think this is the right time to introduce it. Okay, this is what we are going to do, and this is the situation we are in right now, right? Okay, it was an older slide. Anyway, so this is production traffic coming in to the CF router, and there are three instances of the app in production, right? So what's happening is all these instances, the production instances are sending metrics to Datadog right now, and we introduce a baseline version, which is an exact copy of production, but just start up one instance of it, right? What's the purpose of this, and why not use the production metrics, right? The idea is the production system might have been running for, say, 24 hours or more, I don't know, and it might have cached some data, it might have done something which has improved its performance over time, right? We don't want it to compare with the new canary. What we want to do is, how does the new version of the baseline that's the production compare to a new version of the canary? Are you guys following what I'm saying? So that's exactly what is being done here. One instance of the production version is spun up, and that starts sending metrics to Datadog, but we want to identify which one is production and which one is baseline. So that's why we are going to add tags to them. So the tag there for production is, it says deployment is production. For baseline, we are saying deployment is baseline. So Datadog collects these metrics. The next thing we do is the new version which came from Google Cloud Storage, right? The new version of the app. We're going to take that and send metrics with the tag of canary. One thing you need to note here is, all the three different apps, not three different apps, three different versions, not even three different, two versions of the app are all bound to the same URL. Why is that? Because we want to send traffic to all three of them so that they generate metrics, and then we can do an analysis on it, right? So in this case, when the first baseline was spun up, it's getting 25% of the traffic, and when the canary came up, it's again splitting between all these versions, right? So a small portion of the traffic is now going to these instances. I imagine with the weighted routing, I think we can do some really interesting traffic splitting between these instances. But for now, I think it's just doing a round robin, right? So let's look at where the pipeline is. As I said, the pipeline is kicked off, and you can see the deploy canary and the deploy baseline is done, right? So let's have a look here. Previously, when we looked, it was just the production version. Now, you can see four versions. Sorry, there are three apps. Oh, it's not. Sorry, guys. And make the screen a bit bigger. You can see the two new versions deployed in the last three minutes. The baseline and the canary has been spun up, and Datadog is getting those metrics. Let's look at Datadog. So the yellow line is the production version. And since these two were just spun up now, that's why you see the metrics for the baseline and the canary. So I can just disable the canary. So what you're seeing is the baseline, right? What you need to note here is you can see all of them are almost on the same line. That means, okay, the spike is gone. The canary version, that's the new version, is at 250 milliseconds now. So what we can do next is, okay, I have a stage called manual judgment here. The only reason this goes completely against the continuous deployment practice where you don't have any manual intervention. The only reason I have this is I wanted to show you the metrics, show you that the app has spun up itself. That's all. So I can just say continue here, which will kick off the canary analysis and then make a decision about the canary result and deploy to production, delete the canary and then delete the baseline. So I'm going to say continue. Okay, let's look at what exactly is happening here and look at the configuration as well. So as you can see, you can set up all these parameters. So I have said, for the purpose of this demo, I want a warmer period of one minute. You could say whatever time you want it to wait. And the interval there, the second stage, which it's going to get to after one minute is what's the interval for the canary runs. That's one minute again. So it will warm up for one minute, wait for the first minute interval, run the first canary analysis, wait for another one minute, run the second run. And then we can look at the reports, whether it's passed or failed. If it fails on the first canary run, the entire pipeline stops. Let's look at what is the canary config. I mean, what exactly is this doing? So this is the canary configuration here. So the only metric I'm interested in is the pickup prediction time. If you look at what the configuration looks like, it's just looking at Datadoc for a prediction time, predictiontime.average. One of my colleagues within Pivotal mentioned when I was showing this demo to him that average is not a good statistical measure for doing such kind of things. You should look at percentiles and stuff. So it's just for your information. I have not changed it in the demo. But yeah, that's something to keep in mind. Look at this one, the criticality fail, the canary if the metric fails. And also you can say fail on increase, right? Why would you fail a canary on decrease if your performance is improving? There's no point in failing it. It's good for you. So only fail on increase. So that's some really cool features built in there. The other thing I wanted to show was the configuration of the canary analysis stage itself. So here this is where we are saying, okay, use this canary config which I just showed you, right? And this is the lifetime. It's two minutes. The delay is one and the interval is one. You guys know that. Now, this is the important part. I'm saying, well, where is the baseline? Right? When you look at Datadoc and when you send that query for the prediction time, this is the tag which you have to look for. I mean, all metrics associated with that tag. And for the canary, I'm using this one. Now, how does this happen? How do I change that? So if I look at the canary configuration here, I've set it here as an environment variable. When this particular version of the software is launched, this is the tag or this is the environment variable you need to use. And how did I map it? So if you go to this service here, this is, I'm using micrometer. It's a Java base library for sending metrics to different kinds of APMs. I'm saying the prediction time, whenever this function is called, just log the time it took to run this function to Datadoc, right? That's how everything comes together, basically. Let's go back to the pipeline and see where it is. Yeah, it's past the canary analysis. Let's look at what the results are there. So you can see the canary summary. The last one I showed, there was a failure. But let's look at what this one shows. As you can see, yeah, it has come down and they match now. And that's how the canary analysis passed. And you can see the deviation is just 1.4 percentage. And there are some more information as to how many metrics were collected, how many runs were there, what's the average between the baseline and the canary, and information like that, right? So the example I'm giving here is just for prediction time. You could go and say, log the CPU time used, log the memory on different metrics, right? System metrics can also be logged and run as part of a canary analysis. So you can see all these metrics and you can, there are lots of options within this canary configuration where you can say, okay, if this metric fails, don't worry, let's go ahead. If the percentage of change is less than 20%, don't worry, go ahead with the thing. Okay, so that has passed. The last stage is deployed to production, right? Because now we know that the canary analysis has passed. The last stage is to deploy to production. So I just wanted to point out something here. You can use the spring expression language to actually test what has happened in a particular stage. For example, here what I'm saying is, check whether the canary analysis stage has succeeded, right? I'm using it here in deployed to production. I'm saying, only if the canary analysis has passed deployed to production, otherwise what's the point? You shouldn't be deploying it into production, right? And the other stage is the delete canary and delete baseline irrespective of what has happened, whether the canary passed or failed, I want to get rid of it, right? So those don't have any such conditional statements. So I think once the deployed to production task is done, you should see here the new version in production with the lower latency. Yep, you can see that the production version is now four and the previous version which was in production has stopped. Now, this is another cool thing which you can do. If you're not sure, right, you want to keep the old version still there so that if you want to roll back, something's gone majorly wrong. If you want to roll back, you can do that as well. The way you do that is if you look at the configuration for deployed to production, there are a few strategies here, right? What I've said is use a red-black kind of a deployment. What that means is till the new version of the software is up and running, don't disable the other one. As soon as that comes up and the health check has passed, disable the previous one, but don't get rid of it. Keep it there, right? If I want to roll back, I want to roll back. The other one is called Highlander, which is a next strategy, which says just get rid of everything if the new version passes the canary analysis, right? I think... Yeah, one more thing. Sorry, the last one I want to point out is the spawning of the baseline. How do you do that, right? It's basically cloning this stage, whatever is in production, right? So it's a new feature which has been added by the Cloud Foundry integration team within Pivotal. So what I'm saying here is the stage is called clone server group. You can have a... See here, the clone server group. What it does is... What is the baseline? Basically, it's a copy of production, right? So what we're saying here is the source is, look at this region, that is, look at the pickup prediction service organization and production space and pick up the newest server group and spin up a version for me. That's how you create a baseline here, right? Yeah, I think the pipeline is done. So I think it's a successful demo. I'm happy even without my laptop, it got through. So I think that's the end of our talk. Unless you have to... Is there any questions? I think we're happy to take. Okay, yeah. So the previous version of the software is version 13 and... She's the product manager for the Cloud Foundry integration. Spinnaker. Olga. Yeah. And yeah, is there any questions we're happy to take now? Yeah. Do you know what I mean? So let's just say you're spun up at 20 instances. Yeah, that's the production version. Because you have a enormous pressure on production, but you only have one instance of the canary, right? Correct. So when you flip the routing, are you going to spin it up to 20 instances right away? Where are you flipping the routing? From Spinnaker. I'm assuming that when you say the canary is good, you're going to go from 1.0 to 1.1. The routing is still the same. Or you're just going to deploy 1.1 to the production instance. Yes, it's binding to the same route. Okay. Yeah. And disabling the previous one, that's all. So the deploy stage will deploy the new version in 20 instances, make sure they're healthy, then flip the route to them and disable the old version. And then it deletes baseline and canary separately. So if you look here. It will spin from the canary to 28. The new versions to 20. The new versions. Right. And call that production. Right. No, no. The canary, you can control it. How many instances you want to spin up, right? If you look here in the pipeline configuration for the canary, you can see how many instances do you want to spin up of the canary, right? But normally it's, if 20 instances running in production, I would probably spin up just two of them. Yep. Or one, I don't know. Depends on the use case and depends on the app characteristics, right? Okay. So the canary doesn't become the, no, no, it's not. Sorry, I misunderstood. It doesn't. It doesn't become the, it gets deleted. But that version gets deployed as production. I think the bigger question is around like auto-scaling. If you had, you know, your normal instance was five. And like he said, there's tremendous pressure. You're at 20 now because you auto-scaled up. Does it work in that situation with the canary now? Take on 20 because that's what it's currently sitting at because of the auto-scaler or what will happen? In the final, okay, that's a good question actually. I've not thought about it. But if you look at the final deployed to product, deployed to production stage right here, deployed to production, you can actually set some parameters here. Currently it's just manifest based here, right? So I understand your question. If it has already scaled, will the production version have that scale? Yeah, we had to do like custom code for that to pull down the auto-scale settings currently. As of now, I think it'll just scale up to five, whatever the thing was. Yeah, exactly. In this one, my fake pipeline had auto-scale, how you would use that within Spinnaker and then bind it to the out front. So we didn't do it in this one though. Yeah, we are actually four minutes past 11.50. Probably, yeah, one more question. Yes, it does integrate with the Cloud Primary Service. So I think your question is, so one of the stages is deploy service and what that does is effectively a CF create service of the service that you selected and the parameters that you set. So like I said, all of these stages are going under the covers to call, not CF CLI, we're using the Cloud Foundry Java API to make calls, but they're the same risk calls once. Any more questions? You want to go? No? Okay, thanks guys. Thank you. We'll see you in the evening.