 Good morning everybody. And while people are popping in and joining us, I'm going to just let everybody know that this is an OpenShift Commons briefing and I'm really pleased at the number of participants that are jumping in and joining us this morning. So it's a tribute to Veer's popularity and as well as to his expertise. So Veer Mukchandi is our, you see, platform architect and an evangelist for us at Red Hat, but he's also much, much more. He's done lots of work on client deployments and helping customers and folks get all the efficiencies they cannot have OpenShift. And today he's going to talk to us a little bit about implementing blue, green and AB deployments, which I think is near and dear to a lot of our hearts. So I'm going to let Veer introduce himself and give us a little bit about his background and then just kick this off. And we're going to talk for about a half an hour and then we'll do some Q&A. If you want to enter questions and answers through the chat, please do so. I'll read them off and then we can have a conversation afterwards about whatever feedback you want to give him or any other questions you want to ask here. We've got him for a full hour. We take it away, Veer. Thanks. Thanks, Ryan. Hello, everyone. My name is Veer Mukchandi. I am a platform as a service architect or an evangelist. I am also called a middleware specialist. My job is to work with our potential Red Hat customers who are interested in platform as a service, OpenShift, right? And then help them adopt OpenShift in their enterprises. So I go in stages where OpenShift is new for them. They are trying it out, getting their hands wet. I help them set up OpenShift in their enterprises and then coach them on how to use OpenShift kind of things. So in that process, I also come across a lot of situations where I am from Red Hat's perspective. I come to know about what customer situations are or what they're asking for or what the expectations from past are firsthand. So there's a good part of my job that I am exposed to what customers are wanting from a tool like OpenShift. So that's about me. So the topic for today is blue-green and A-B deployments. So what are these and where are they coming from? Let's talk a little bit about that before we get into how to do this, right? Now, blue-green, when it comes to deployments using OpenShift, application deployments using OpenShift, one of the questions that comes up from the customers whom I meet is how do I achieve efficiencies in terms of deploying newer versions of applications? My team is always changing applications, right? And as the applications change, they have their handover from development to QA, QA to production, right? When it comes to especially to production, the operations teams have traditionally they have been like working over the weekends or taking some time in the nights where the application can be taken down and a newer version has to be deployed. And that's painful as well as time consuming and if things fail, then they have to roll back. All these issues would be there, right? So how do we reduce the downtime of applications number one or how do we probably achieve zero downtime deployments, right? And that's the kind of question comes up and there are patterns that have come up in the industry in the past few years and which enable this minimal downtime or sometimes zero downtime deployments. And these are those patterns, the blue green deployments and AB deployments are kind of those patterns. They are more or less the same, but there are some minor tweaks, minor differences. So we are going to discuss what these patterns are and how OpenShift enables these patterns with the kind of flexible architecture OpenShift has, how those patterns can be achieved using OpenShift or I do have a couple of applications that are already deployed for these patterns. I pre-deployed them for the sake of time because this is just a half hour and if I try to build and deploy during this call, it's going to take a lot of time. So I have these examples ready and I'll show you how to achieve these patterns, but let's talk about what these patterns are. First we'll address the blue-green deployments. So in case of blue-green, right, what does blue and green represent? Blue is the version of application that is running right now. Let's say it is version one of your application, right? It's think of it as blue color, blue is version one. And let's say blue has some problems and green is version two, which is with the changes. So the newer version of application, let's think about it as green, right? So you want to move from blue to green. So if you look at how the blue version of application is running today, you have the app running and there could be one or more than one instance of app, which is of blue version, right? So in terms of OpenShift, I'm making an assumption here that people who are on this call have no little bit about OpenShift already. They know what a pod is, what a service is, and all that. So I don't want to explain all those things at this point. I'm just directly jumping into how to do blue-green without assuming, by assuming that you know all the basic stuff about OpenShift, right? So let's say if you have multiple instances of your application running, you may have multiple pods which are the running instance of your application, right? And all these pods or all these instances of your application are frontended by a router. And the router is the one that is load balancing and distributing the traffic across all these instances of your app, right? So there is a blue version of app running, and when your client, which is probably a browser, right, is hitting your application's URL, then it gets resolved to the router, and from router, the router will take it to the blue version of application on one of those pods that is running, right? Now, when you want to introduce a newer version of application, which is the green version, right? How do we make a quick switch from blue to green without incurring a lot of downtime, right? So in that case, what you would do is rather than tearing down the blue version of application, in case of OpenShift, since applications run as pods, you can spin up a new app, right? Perhaps within the same project, spin up a new app with the green version of application. That means a green pod gets spun up. So now both blue and green are running. You can also test your green app in advance and make sure it is running. And after that, you can disconnect the route from the router to the blue version of application and at the same time enable the route to the green version of application. I'll show you exactly how it is done, right? Now, what happens in case where your green version is bad? Since your blue version is already running, you can quickly switch over or roll back to the older version if you have to, right? That's blue-green. How do we switch instantly between the blue version and green version? Now, having said this, let me quickly switch over to the demo. As I said, I have a pre-deployed application. Let me pull it up. So what you will see here, I have a project with name blue-green demo, right? And in this project, I have two applications running or two different versions of application running, right? I have a blue application, right? And this is running with just one pod here. And there is a green application that's running with another pod. And both these applications have their... If you know OpenShift every pod, you cannot talk to pod directly. There's a service front-ending the pod. So blue application has a blue service and green application has a green service front-ending it, right? Now, also, you'll see a difference here that there is a URL for this application, right? And this URL is associated with blue right now, which means that your route is pointing to the blue version of application. So let's see what it looks like. So it's showing a blue rose, right? Now, let's say we want to switch over from this route from the blue version to green version. Let's see how easy it is to do this, right? I'll bring up... Let me switch over to the project first. So I'm getting into the blue-green project. If I look at the routes, I have this route, which is this URL, right? And this route is pointing to the service blue, this service. Now, all I have to do is edit this route. This name is blue-green. Get down here. You'll see that the host name is the URL and it is pointing to the service with name blue. I'll change this to green, right? This is the green service. And I'll just save this file. Instantly, you have seen that this URL now moved from blue to green, right? Now, let's go back and see what happened to the application itself. There's a green rose, right? So that's a quick switch over of app from blue to green. So when you want the newer version of application to be deployed, all you have to do is just switch over. So this can be done at any point of time. You don't have to be in the night to do it, right? So think of the amount of ease with which operations can handle this kind of task, right? That's the advantage of blue-green. Now, there is another pattern. I'll move this away and we'll go to the next kind of deployment pattern. There is another pattern that keeps getting discussed, which is the AB deployments pattern. Now, one more thing I forgot to talk about. If you wanted to switch from green to blue, again, you would make the same exact change. Like if something was wrong with the blue green version and if I wanted to change to blue, all I do is this change and then my app gets back to the older version, right? Now, the next pattern is AB deployments. And AB deployments is more, although it is very similar to blue-green. This is a little different model in the sense that a lot of people ask, like, I have a newer version, but I don't want to switch over completely from the older version to newer version. I want to test it out slowly and slowly move my deployments from the old version to newer version, right? Now, by default, let's say you have n number of parts of your application running in production. Let's say you have your app and in this case, since we are calling AB, let's call the first version as app A. Now, your app version A has, let's say, 100 parts or 100 instances running, right? If you are doing a deployment with OpenShift, by default, OpenShift does a rolling deployment, which means that each part will be taken down when you change the version from A version to B version. A part is taken down and the new part is put in its place and it repeats for all 100. You can also change it to update by certain percentage, which means you can say that I have 100 parts. I want to take down or change 10% at a time, which means 10 parts are taken out. They are replaced by newer version of your application, which means out of 100, 10 will be new, 90 will be old, and then it keeps repeating that until it reaches 100. That functionality is there automatically. You have the rolling deployments configured automatically when you try to do deployments with OpenShift. You can change the strategy at any point of time, but it's configured by default. Some of the customers ask for, yeah, you will do automatic rolling deployments, but I don't want it that way. I want that to be controlled. I have 10 instances of my app running, 10 parts running, but I want to decide when I want to increase the number of instances of app B, which is the new version, and reduce the number of instances of app A, the old version. So they want to have a complete control on increasing the number of instances and reducing the number of instances. Let's look at what it looks like. You have the same model, you have a router, your client is reaching the router by using a URL, and there are n number of instances of your app version 1 running, which is app A. Now, let's say you had a change, you made a change, it's tested, and then it's being driven into production as an deployment, right? You spin up a new part of version 2, which is app B, right? Now, you don't have to take down the version 1, you just introduce one instance of app B, and you have four instances of app A in this case, right? And then if you are, some of the load is getting directed to app B while the rest of the load is going to app A. So both the versions of application are running at the same time, right? And then as you go along, if you are happy with the performance of app B, you start increasing the number of instances of app B and reduce the number of instances of app A, right? And this continues until the app A version is completely taken off, you don't run any instances of app A, you only run instances of app B, right? So this kind of a model where you gradually shift your workload from app A to app B is called AB deployments. Now, how do we achieve this using OpenShift? In OpenShift again, we have the route, right? And we define a service that frontends the parts, right? In this case, what we would do is, we define a label for the application, like for example, you can call this label anything, but let's say we say all these parts are AB members. AB deployment members, right? So I set a flag called AB member equal to true and it applies to all the parts here. And then I also define a service that frontends any parts that have this label called AB member equal to true, right? Which means whether it is the blue application or green or red or whatever it is, right? Any part that has AB member equal to true will be frontended by this service. Now, my route is defined in front of this service. So when I hit my app.mydomain.com, it goes to the router and it is, since this route is frontending this particular service, the router will load balance across any parts with this particular label, right? Now, as we move along, we add app B with the same label. So now the route, the load that is coming in gets distributed across blue as well as green at the same time, right? This is what we do in order to achieve the AB process. And then over a period of time, the blues will go away and our application A will go away and our application B will remain, right? Now, let's see, going back to the example, right? I have another, let me close this. I have another pre-deployed application. And my project, I call it AB demo. And if you see here, I have an app A and there are six parts of app A running here. And I also have an app B and there are currently, there are no parts for this. And both app A and app B are frontended by a service. This service is not relevant because we are not going to use it. However, there are six parts here for app A and there are zero parts for app B right now. You'll also see that there is a service AB, which is the same as what I was showing here. And this has this AB member equal to true kind of a label set, which means that any parts with that label will be frontended by this service and in turn this route is going to load balance across all this, right? Now let's pull up and show what I'll do now is, you see this URL if I what is getting displayed here. It just says I'm version one and my product IPs so and so right now. Let's see and you'll see that there are six parts running and all these six parts belong to app A and just have a look at this IP addresses, right? These are IP addresses of each of these parts. Now let me run a loop here. The reason I am doing it from command line is because if I do it from the browser, there will be enough. There will be session affinity to a particular part. So every request will end up going to the same part. So that's the reason why I'm doing it from command line. If I learn run this for loop and call this URL. It's going to say the same thing. I am version one but look at the part IPs here. 162 63 226 227 247 246 and then again 162 163, right? So the six of them are round robin and then they repeat again and those are the six of these right and these all the six are version one which is app A. Now what we'll do is we'll scale up app B and we'll scale down app A. So so I'm scaling app B down up and I'm right now that a zero. I'm making it one and I am reducing the number of instances of app A to five once I make this change. Let's come and have a look at it. It's creating a new version of app B. We'll see that there was zero here and now it's spinning up a new part. It's already done and also the number of parts of app A have been reduced from 6 to 5 right now under this service a B now you'll see that there were one two three four five parts of app A and one part of app B got added here because this guy is servicing both app A and at B at the same time now. Let's have a look at whatever loop does for us. You see that there are five parts of app A and there is one of app B so five of them are five of the requests are coming back with version one and one of them is coming back with version two. So you just introduced version two of the application just with one part and that's being hit and then it's around Robin right now. Let's go further and make them equal right. So three of happy and three of happy now you'll see that app B scaling up and app A has already scaled down. Let's wait for it to come up. Yep, all the three are up right now. Let's go back and check what the results are. Oops now the load is distributed right between one and two now let's say I completely scaled on scale up app B to six and scaled on app A to zero app is completely gone now. We are happy with app B. So all six are at B. So now everything is version two. So this is how the gradual increase of your newer version of application and gradual reduction of the older version of application can be done now. Remember that there are two versions of applications available at any point of time if I'm not happy with if with the newer version for any reason I can always scale back the old version to zero and switch over to the new version to zero and switch over to the old version right. So that's that's the power of being able to do things on time that that's what that was about the AB. So I'm done with what I wanted to show. Now I'm ready for questions. So there are a couple of questions have come in Vera and thanks for that it was it was pretty pretty good enlightening for me in the blue green. What happens when someone's got an already open connection and you disconnect to do the to the blue? What happens to that connection? Does it keep is it complete? If someone's running. So there is a yes there is a setting that you can do in terms of once when you are scaling down your parts right or when you're trying to shut down your part. There is some responsibility from the application developer side on how they take the termination signal. So OpenShift will send a termination signal and your app once it receives the termination signal can say that I don't want to accept any more new connections from now on. But I'll service my existing connections and complete it before moving on to the before shutting down. If the app can be coded like that, then there will be graceful termination of the of that part. It's the all OpenShift is doing is sending a termination signal to your part which in turn ends up on the app. So the app should respond to that in order to achieve that effect of a graceful termination. Okay. And yes, the session is being recorded Palmer. That's one of the questions that just pop. I know where you set the labels and selector in the AB and the app AB service. Sure. I run OC get service. There are three services. One is for just app A alone. And the other is for app B alone. And the third is app AB. I created a new service and I set the selector as a big group member equal to true. Let me open this and show. See the selector here. I set it as app AB group member equal to true. Now, how did I do that? Let me it was by when I created both two versions of applications using a template. For example, this app a service was created and ab service was created. But this app AB service I had to create myself. So I used this command to create the service. I said expose DC app a for example that at the time only app was there and name I gave it as app AB and I said use this selector. So this is the key here. So I'm defining a new service. Generator equal to service, right? I'm defining a new service and I'm defining that service with this selector. Right and then it created this additional service for me. Any other questions? Mateus has a question. I'm going to let him unmute himself and ask directly. He can figure out how to unmute himself. No, not quite yet. He's figuring out he's muted himself. Can you talk a little bit more about session affinity? I think that was something you glossed over pretty quick. Not doing it in the console versus doing it in the terminal. And what's happening with that? If I, for example, if I I refresh it, right? And now it is like 3.252. If I continue to refresh my screen, I'm still hitting the 3.252. It's not going to a different part. That's due to session affinity configured in the router. If I if the request is coming from the same client. It's getting directed to the same part due to the session affinity. And that's the reason why I had to do it from as a curl from the command line so that I can see the list of parts that I can show that on Robin stuff. It's hard to do it from the browser because I have to completely clear the cache and do it again if I have to show a different part here. But by default, the session affinity is already there. And that's the reason why you are always hitting the same part. Okay. So I think Matthias may have. Hello, can you hear me? Yes, we can thank you. Hi. Go ahead. First of all, it's an honor to be here watching this working technology. We have a couple of customers that often they do a deploy that breaks everything. And I guess the program deployment is it the best way for them to go. So what is your recommendation recommendation when the new deployment would let's say bring some compact BTR or manage with the database. What was the best way to move when change will break. So there was a code so I could not hear it clearly but let me reiterate what I understood. What is the recommended approach when there are application dependencies on let's say the data with the back end database. Is that the question? It's close. I think what Matthias is concerned with and I apologize for the echo if you're getting it too is that if you have a customer who deploying in blue-green that it breaks the application and maybe a database dependency or something else but it's what's the best you know what he's asking is what's the best way to go when a new deploy breaks the current deploy. Must be some interdependency between the applications. So when you're doing blue-green the first type that we discussed about right you have the blue version that is currently running let's say and when you're introducing the green version it's a flip between blue and green at one instance right so either the blue is active or green is active not not both at the same time now in such a case if the blue was running okay and it was it was fine you have as long as the blue is running you have you can take your own time to test the green version as long as you want before doing the flip or right so now in spite of like after the deployment of the blue after switching over to the green version if you don't like it you can always switch back to blue. I didn't understand the one version affecting the other version that that part I didn't understand maybe perhaps you are saying that once I switch over to the green version if if the green version involves some changes to the database and I have to switch over to the blue version that didn't have those database changes right is that is that the kind of question yes I think so okay if that is the case I mean this which pattern that you would apply depends on what the changes can I can I do a blue-green in case where there is a database change is something that you have to think through and then apply so if you are when you are moving from blue to green if there is you can you can still move from blue to green by using this pattern but the being able to roll back from green to blue will depend on what kind of change you made so you need to tell your operations in advance on this kind of a change you cannot flip back because of because of the kind of change it is because the the database is completely different and if I try to switch back there are dependencies that will not allow me to do it so better trust that application fully before before you switch over kind of thing so the the the rollback is still possible by using the procedures that you already use if but the ability of this this kind of pattern only goes only up to a certain extent you cannot it's not a these are not silver bullets that apply in every situations you mean you have to decide on what works and what doesn't for your particular situation. So Mateus is saying that blue-green is is like an atomic switch. You have to go all the way there's one other question there's a couple other questions that are coming in in the chat and Peter is asking or saying in the situation where the app is stateful he's still missing how open shift determines there are no more sessions on a and now a can be stopped. He's a little right. So the as I said before right the open shift will send a termination signal to a part when you are asking it to shut the pod down and the termination signal is can be listened to by the application. Now if you want a graceful termination your application should not accept any new connections but service the or the existing connections before it it shuts down. So there is it's it's not a complete open shifts responsibility it's like open shift will enable you to like there is a termination signal and there is a kill signal and there is a gap in between those two and you can I believe you can configure that but between this time before the kill comes in there is a chance for your application to gracefully shut down. So I think I have to probably go back and research more on the topic and probably come out with an example at some point of time in the future but that yeah I don't have a example as an answer to that question but we need to work on it more to come back with an example for future that I think that'll be another little video blog post shortly I can I can see that one coming the distinction between termination and kill yeah how to handle that gracefully Randy is asking what exactly needs to be done to an application to enable the servicing of existing systems but not allow new sessions new sessions are you suggesting that application should now completely control session management and I think this is more to the same topic yeah it's the same topic yeah I think I'll I'll get into more details and probably answer this question as a blog post or yeah yeah so I think that's that's definitely a topic for the next one yeah and that people are going back and forth giving each other suggestions now on how to do this gracefully but some of the burden here is on the application to be built gracefully so that it shuts down gracefully I think that's it's pretty clear to everyone in the chat as well would be nice if we could do everything on open shift for everyone but there's a little bit that you have to architect into your applications as well I'm looking and there we go are there any other questions at the moment that people have or anything else that you'd like to share? No I'm I'm done with what I wanted to say but thanks for thanks for attending though yeah I thank you all for all for joining us today I'm really pleased to have gotten this topic aired and you'll well Mateus is asking one more question and I'm not sure Vera knows the answer to this one but will we see this on the console? See what? You could clarify that yeah maybe it has I have to read the other things before that question to understand the AB deployment blue green deployments or something or you know adding this kind of feature that onto the console web console itself like a button or something to AB deploy my app okay yeah yeah I don't know that answer to that question I'll have to check with the powers that you on the road yeah the road map and the Trello board I haven't seen that request come in but there you go so there's a couple of things the link will be on for this recording will be on the briefings page on the commons.openshift.org site should be up probably tomorrow sometime and you can also if you've logged in through Bleachings you can log back in the same way and you'll be able to see the video as soon as I it gets compiled and reposted back up to Bleachings today but if you look back tomorrow it will be on the briefings page so there you go thank you again Vera I'm sure we'll have you back again soon always informative and thank you everybody for joining us today thank you Diane thanks everyone