 Should be in this conference. Yeah. No, I haven't watched it. I think it was on Friday It was mainly for the open ship the new platform of the 7th day was not Maybe the similar Yeah Because the recording is automated, so we need to start Here So George you talked about deployments on Friday Cool, so that's a complete Cooperation of the subject so we can start what we will come repel and Learn something So welcome everyone Since Friday, I was talking here in I've talked about microservices. I talked about the 12 factors methodology And in each one of these talks I ask the same question Who here is not a red hat employee? Oh, welcome you to this conference Well, this title seems a little bit complicated But in fact what we will see together here now is about the deployment strategies how we can perform Zero downtime deployments using different strategies And what tools we can use tools from red hat to improve this This deployment Uh, these slides are available in this url. I've told that that even the Slides are open source are freely available. You can use it You can copy you can get these slides that you that you want That's something that you like you can access that in your mobile phone Present that in your company present present that in your local jugs any other communities. Okay Well, my name is Rafael Benevides. I've worked for a red hat for more than seven years. I started as a consultant I was a software engineer helping to produce the red hat developer materials and now i'm helping developers worldwide to Be successful using red hat technologies red hat tools. Okay Uh and as part of this initiative to help developers I like to invite everyone here to visit and join and join the red hat developers program If you go to this url developers dot red hat dot com you will find This portal which has a lot of resources for developers. You will find free access to our products You will find free a books You will find blog posts and and and many things that we That we think that might help your development process Okay And since that we are talking about development I like to start this presentation with this phrase from from forbs Now every company is a software company. Have you thought about that? If you if you think about that you will notice that the most The biggest companies in the world are now software company Uber for example doesn't own any car. It's a it's a app. It's a It's a software product and it's now the most the most the biggest Transportation company in the world. The same is true for example for air bnb. It doesn't have any houses And they are the largest accommodation company in the world We can think also about It doesn't sell it doesn't have any stock But it's one of the largest selling company in the world What else facebook Facebook's the largest content provider of the world sometimes we can question the kind of content that they are providing but it's still this is Facebook's the largest Content provider Uh company in the world But in common to every company including the the company that you work because if you think about that you Also, you don't work for a bank. You don't work for a hospital. You work for a software company And when you receive a request when these companies receive a request to to implement to create a new feature Usually it takes like three months To plan develop start the qa and finally deploy that in production, right? But something that you might consider is the kind of learning that you receive when you are developing a software because When you put a software in production you will Verify its errors you will verify if it's behaving it's it's if it's accomplishing the requirements Just when it reach the the production, right? For example Uh, suppose that you implemented a sailing a selling feature that will increase the sales in 10 percent Only when it reach the production you will find out if it's increased 5 percent or 10 percent or 15 percent You need the software to be in production to learn with the with the result of the deployment And this is part of what's called feedback loop. I've talked about that when I was talking about microservices because Everything in our life as a human being depends of the feedback loop When you we are learning how to walk we depend of the gravity To know about the the feedback to to have a feedback from the gravity if we are walking, right? When you are learning how to speak We depend of uh, we depend of the feedback of our parents to say this is the the right pronunciation No, no my son. This is the right one. You are talking wrongly So we always depend of the feedback to learn something But when you Take like three months to deploy a software in production You are sending a lot of push a lot of commits imagined in three months. How many commits Git commits or Subversion commits you have in three months Probably you will have Hundreds of changes in your code and the problem with that is how do you identify? If something is not wrong, what was the commit? That caused that failure. What was the change that made your software that was stable To not perform okay to be unstable to crash or something else The problem with with that is the size of the batch As many things we tend to send to the production more things we can try to Understand which we can try to debug to see if that was the causing of the problem or not So every time that we reduce the size of our batch We can also easily identify what uh, what was the cause and have more fine grained for fine grained Look up for For problem or issues or our feedback So for example, if you reduce the size of your commit if for example, suppose that instead of sending to the production the production A hundred of commits and you send just the commits of one week It will be easy for you to identify any problems and that allows you to learn faster Because if you send something in production in January and send another one In april you will have a certain a certain amount of learning if you Deploy frequently if you deploy very often You will increase your cycle of learning your cycle of feedback And possibly in very possible very likely you will also increase the innovation of your software But the problem is How can you Deploy many times a week many times a day? Well, that's why we'll talk about the deployment patterns And before talking about that we have to understand that even microsoft Uh, I have a problem with uh, the the learning curve or the feedback About the the the software because this phrase became very famous This guy ronnie, uh, he worked for amazon. He is now working with microsoft and he said that even In microsoft one third of the ideas improve The the result the other two thirds are just discarded because it was not good It was implemented then they figure out about uh using the feedback loop That it was not a good idea and has to be discarded Okay, so when when we want to deploy We usually have the maintenance windows So how many of you have ever worked Friday midnight to deploy a software? A couple of people right sometimes saturday Saturday 4am because this is the time that people are not using software How many of you have to face this? This goes here the maintenance windows It's it's true that we don't like to do that. We would like to have Uh, a kind of mechanism a kind of strategy that we could place your our software in the production with zero downtime with this uh with the And being certain being certain that that that software Will work the way that we want So one of the keys that we will Cover in this talk is there zero downtime deployment And to perform zero downtime deployments We will talk about different strategies and I will show to you how we can use those strategies The first one is called the rolling update and it's called enrolling because we will deploy we will suppose that we have three replicas Of this uh version number one in the production. Let's call this version as version version blue and we have several clients several customers accessing Those versions of course to be able to balance the the request we have we have approximately in the middle and To deploy the version 2.0 In this case, it's called the I'm calling it as a green version We need to replace each one of them at the time so for example We can remove the version one from from the the proxy introduce the version two and Something that we should consider is the time that it takes the application to be available Right because for example some application servers take like two minutes to boot some others take one minute others can take 10 minutes to warm up the application to connect to the database Make sure that the the cache is Is warm it's set up everything in its place So sometimes the application takes between one minute to 10 minutes to be ready to receive requests So when the application is ready is the moment that we can start Sending requests to that version, right? We cannot send requests Until the application is ready because if it's deploying Probably those cost customers those clients that are accessing that version will receive a 404 error so an important thing to do Everything related to deployment is know when your application is live and ready okay so once that The version two Is live is ready We can send requests to another to to to to that version and then we can start moving Rolling this this deployment to another version and gravely these customers will receive the new version So let's see how we can perform that You saw that something That is Extremely necessary in this kind of strategy is to use a proxy in the middle What I will do to to demo this Kind of strategy is use open shift I have here prepared a demo version. It's a Just a wrestling point That I can access on a url here. For example, I can call api hello bernel And that will reply what I use here as a parameter. Okay. It's a very simple Application I have here three replicas And I have also this The result the result of the invocation running and In my console so each one of these from Hello the parameter that I'm using and the from is the number of the Hosts that I have a available here Okay So what I will do now is Let's do a modification in my source code and deploy that and the idea something that I really like about this talk is because it's It makes me feel like doing an an advantage because I will deploy at the software And hopefully with zero downtime So let's do it Let's do a modification here In the source code Here is the phrase So you don't need to worry about that, but I will write here version two And now I will build This demo it's just like a Let me increase the font size It's just a normal maven build, but at the end of the maven build I will call a build inside open shift So that there you can see The rolling deployment because it creates the environment. You can see that it's not live. It's not ready yet And once it becomes red it starts to shut down and create a new one So in a couple of seconds you will see the new version become becoming available During the deployment So you can see that the requests that I'm doing are not Going to the version two until it becomes alive So now that the three versions are available It shut down the it shut down the version the previous version and now all requests are being made in the version two What I've used here is that the full strategy for Deployment deploying applications inside open shift because open shift already have Approximately in this address That balances the requests to these three nodes three these three pods here Okay So what else we can do as a strategy for a deployment? We can do what's called blue green deployment the difference Of the green deployment is because you usually have one version which is called blue again but To be able to change that version to another Again, you need a proxy when you need something in the middle That allows you to deploy a new version called green And in this new version you will select between the blue And the green version if something goes wrong with the green version you can just return The the route back You can point the proxy back to the version blue if something goes wrong So let's see how we can perform the deployment using blue green As you can see here, I have this version that I've deployed this the That was my green version and I have another one here called demo blue So I have the two parallels two similar Uh applications. It's just different between them related to the version This one I deployed five minutes ago and this one have been deployed like 16 hours ago when I was preparing this environment So what I will do now Is since the the The application is already there. I don't need to deploy the new version. I can use The open shift to change the route from one Application to another so let me change that let me Place the window here You can see that I have the route pointing to this version And when I ask to select the blue version the route click Will change from here to here and the result of my application Have changed immediately So that's how I can also perform a zero downtime deployment I create another version of the environment and just switch the route From one version to another But suppose that I didn't like this version. It has a problem. It hasn't it's not performing well It has a bug and I want to return to the previous version it's easy as Changing the route again to the version green Zero downtime deployments the route moved from here to here Something something really really easy another strategy That we can do Is called canary deployments Something that that I like about canary deployments. It's why it has it this name because When I heard about the story it was really interesting because It came from the mine workers When people went through the mines the gas map mines to to work They always take They were always carrying Canary With them if the canary was happy If the canary was singing it means that there was no gas leaking they would survive But if the canary stopped singing That would means that they have a problem They have a gas gas leaking and they should run out of the Of the cave of the mine as quick as possible So the idea of the canary deployment is more or less the same because we will introduce This new version as a canary It's a Just a small amount of requests will arrive in the canary version And then If the canary is still sing if the your application is still Being good you can increase They request to this version and as as more you become comfortable as more you become You as more you trust on that version you can keep increasing and if something goes wrong for example if you try to To add like 200 requests on that application if that application is not Uh capable of receiving 200 requests you can just Reduce the number and it will Work again until you deploy a new version. So let's see how we can Perform canary deployments Using open shift You can see here that these versions are were are splitted. I have I I could choose using the route Using blue-green deployments between Are that this version or that version? But what I will do now is join both of them under the same route Canary unify when I do that Both will receive the request And something interesting is that since I have three versions of this and one version of the Old one than the canary one Each three requests On this version one will be made in the canary So I can check and I can verify how my canary application is behaving If I'm comfortable with that I can for example increase the number of canary to To to reduce this one to From three to two And you will see the number of the requests in the canary Increasing and decreasing now I have 50 50 Two in the in this one two in this new in the previous one And I can keep selecting how many requests Will be made On the canary version I can even shut down totally the previous version and now all requests Will be made only In the version that I've selected And if something goes wrong, I can return again this to three Shut down the previous version or even remove it from I've shut down it very quickly. So that will probably give me an error because I don't have any version Responding yet But I can even split Split the remove the canary from From the project and now my route only points to the x the existing version This one this over here was because I I have shut down The version completely Okay with this three main kinds of Of deployment it allows us to understand. What is ab testing because many people think that ab deployments Are ab testing there is no ab deployment. There is ab testing because The idea of ab testing is just like a canary, but canary is the technique To deploy a small portion of your new version Uh, and then you can try to see if it behaves well if there is no bug But the idea behind the ab testing is to test unipothesis So suppose that you have an unipothesis that if you change Your button From the bottom of the screen to the top of the screen you think your hypothesis Uh Suggests you that you have you will have The increase of the sales you can test that you can Use the canary deployment and test From the business perspective if the sales have increased or not Suppose that you have another hypothesis if you change the The logos of the recommendations or suppose that you change The logo of the product or even instead of showing the recommendation you show The location of the nearest store all of these are business Hypothesis it's not related with the deployment, but you can use canary deployments to analyze the the behavior of Of your application using ab testing And of course what I did here was directly in the production right But it's not good to deploy in the production So we should always think in deploying using a continuous integration in a con and a continuous delivery pipeline because just humble with uh Who is the author of the continuous integration book said that continuous delivery is the ability to get the Of all types including new features configuration changes, but fixes and experiments into production or into the hands of the users because The software only has value to the users wants it to reaches the production safely quickly and in a sustainable way And of course the purpose of the pipeline Is to prove that the release candidate is from unreleasable it means that means that Your pipeline has to have a lot of good and automated tests That you prove that the software is not releaseable if it passes through the the tests then you are good to go automatically almost Any human interaction of course you can configure the the pipeline to be completely automated Or you can request a new man interaction to deploy it in the production So let's see how that looks the idea is that everything starts with git commit or a git commit and a git push Once that your source code in is in the repository Then any continuous integration tool like for example Jenkins. I will use Jenkins here in the demo We'll get the the source code perform a build That will be released in the development stage In the development stage you can run the unit test and if it's okay to in the Running the unit test you can deploy that in the quality assurance And probably there you will run the integration test if If the software is okay in the The qa area you can then move it automatically to the staging area And for the station probably probably the qa team will test it some some other guy will test it and If it approves it can go to the production and once it reads in production then you will use one of the Stretches that we talked about you can you could use the rolling deployment. That's there is no problem on that you can deploy this new version using a b Sorry blue ring or you can deploy it using canaries So let's see how we can perform Deployments using a pipeline I have here another application that I presented during the the microservice talk That has several microservices Each microservice use different technology and And responds hello in different languages. So One of them is hawaiian. I don't know how many of you speaks hawaiian But it should be aloha instead of aloka Okay So this is wrong. I have here a window that's consuming this this Uh resource You can see that it it replies aloha. So we need to fix that And what I have here also deployed in this in open shift Is a Jenkins instance Oops, let's authenticate again. I have a Jenkins instance and by the way Look here the image name. It's called open shift slash Jenkins Which means that the image provided by open shift. It's a supported image So let's open Hopefully it I will not have problem with the The With the internet. Oh no It seems that I have problem with the With the internet or something. So let's try to shut down this instance and start a new one just to make sure And did People say that it usually happens when you don't sacrifice windows millennium CDs to the demo demo gods When you are about to prepare a demo so I forgot to bring some windows millennium CDs to this presentation But here's a good example for for what I told about redness and liveness probe because If I try to access this application you will see the service available Right because I don't have any instant any instance here And I also have not configured the redness and liveness probe. So Open shift doesn't know for this particular service that this the service is ready to receive requests But let's try to access this one. Now I can access I have here already configured A pipeline For this service And to build this service. I need a kind of interaction with With open shift. So I need to provide the credentials that that that I will use So let me update these credentials This was supposed to be already set when I started the The talk but we need to do it again. So let me Let me show you something to you Here in Jenkins, I have configured The pipeline to read from this location here to read the source code from github and also inside the The github it will read a script file called Jenkins file So inside the This service I have a file called Jenkins file that defines my my My pipeline. What are the existing stages the stages? So I have a stage that will perform a gith checkout So it will check up check out from the source code that I defined inside Jenkins So you can see that I have defined it Here Yeah Yeah, I have defined it here the repository Then it will build my project using maven Then it will build it and deploy it in my development stage Run the automated tests Deploy in the qa in the qa area And wait for an approval if it's approved then it will deploy in the production using the canary deployment So let's see if the the what's happening during the build that I have Already requested let me change the credentials again Once that Jenkins start you can see that the the pipeline will use what I've defined in the Jenkins file Just make sure that the console is running. Yes It's running. So it will deploy it in the development stage. You can see that My build is running inside Jenkins, but Jenkins inner have interacted with Open shift created and deployed the image in the development stage You can see that in the development stage it will perform a rolling deployment And something that you can notice is that the image that have been deployed Is 2 a 5 5 9 1 8 This is the hash code of the the image The next step of the pipeline Here was To deploy the run the automated test and deploying the quality assurance area So the quality assurance area in this demo is an it's another namespace. So if you check here It has the same Hash of the image so it deployed The it it promoted the image from the development To this namespace called quality assurance So now suppose that i'm a quality assurance guy. I can open the application And verify if it's working if it's not aloha, but it's not aloha, but aloha So I can invoke the the service myself And note that this is another route. So it's hello road and There and dash msa dash q8 of quality assurance I can see that it's now replying What I want to And Look here the hash code is 2 a But if we return to production The aloha The aloha hash is d 39 because It's waiting for an approval from someone else. So since I have tested I'm sure that I can proceed the deployment in the production. So let's see what will happen. I said to perform a canary deployment I had a problem here as I told you I forgot to Sacrifice windows milanian millenian cds But it deployed that version d 39 2f In a canary deployment with the 2a 5 5 9 1 8 so now I have both versions running because I've selected in the Jenkins file to to perform a canary deployment So now that I mean I'm comfortable with that with the canary. I can for example reduce that I increased that version reduced this version And finally have just one running in production so these are the Steps and strategies that you could use to perform a deployment We have now Some minutes to talk and discuss and if you have any questions to do that at this moment That's a good point The question was isn't it A problem when you are deploying several versions Of the same application responding in the same same url. Isn't that a problem? Well, what kind of problems we could have the first one would be People usually ask what about this the The state of the application This is a very common question and the answer for that is to avoid using state Because on friday, I've talked about the 12 factor And one of the factors is Let me find it Process is acute the app as one or more stateless process If it's stateless, there's no problem to access a different version But we could also have another problem, which is I've changed my payload. I've changed my contract I'm using the version one And now I my my new version expects the version One and two and this is the very important because your application during the the update it should Allow a backward compatibility with some previous version how you can do that Using for example in the payload the version that you are using This is very common. For example in open even open shift so suppose that I'm I'm do here OC edit DC demo, let me see if that Yeah, if you notice here the payload I have defined here that I'm sending this data as the api version one So if I decide to release the version two My application should be able to understand v1 And v2 at the same time at least for a certain amount of time And if it's and the client has to say I'm sending V1 to you So the server will identify and treat that as v1 And if the the client Sends v2 Then the the the server itself will treat it as a v2 that's a kind of evolutionary design Plus Something that I didn't show here, but I can do it Is the kind of route that we are using One of the things that I did Was to specify Let me see here It's in the service Yeah, it's uh, no Let me find I don't think I Have said that I was looking you can specify session affinities In in the router. So for example Suppose that one client access one One server One version of the server The route of open shift will always Send the same client to the same version new clients Will receive the new version because you are tracking The session affinity to the to the client So you can have round robin for example open shift does not control that but you can also have A session affinity defined So I thought that I have defined that here, but I I thought it wrongly But it's an option to define a session affinity Yeah, but that's a that's a purpose for evolutionary design It's not really related to software deployment And the second way Is use a session affinity to avoid having the the client Uh Sending requests to one version and another and another because it's not stuck with a particular version Cool. So that the question was How to handle schemas update in the database? And the best answer that I can think for that is the abstraction because You should not talk directly to our database instead you have a service That expects a payload for example, you have a rest service that expects a j zone in a certain format Suppose that you are let's talk about the customer database the customer table In the version one you stored uh, the name And the address and the version and in the version two You decided to restore the name the address and the phone number You changed the database But uh So you expect in the database to have the the telephone number available But since your since your application needs to be backward compatible What you will do is the version two will We'll verify if the telephone number is uh, was informed if it's not informed It will give an error because that's part of the use case version two You have updated your use case, but the use case number one still needs to work. So your Your tests during the pipeline should test also the version one and for version one if you do not send the The telephone number You can For example, insert a default the full value in the database It's one of these one of the strategies because because you are not breaking the compatibility To version one clients. It was it was not expected on that time so just just Inserted the full Uh, the full value in the telephone number. It can be a default valid called update me later Something like that just for Inserted by version one something that later you can track down And uh, and for business reasons say, okay, let's let's call Let's send a let's send a letter to the customer and ask him to provide the phone number By logging in again the application just as use case that I imagine in my mind You're welcome Questions Okay, so I really appreciate your presence here Once more don't forget to register and subscribe to the developers portal Uh, and also if you like to continue this conversation, you can add me in the twitter. This is my twitter handle I'll be very pleased to answer your questions Or receive any feedback from this presentation Using twitter or also sending an email. By the way The demo that I've used Of the hello all the msa the the one that use microservice are Is here and the the scripts that I've used To perform canary deployment It's here and it uses different versions. I have one demo demo using eap another one using vertex Another one using wildflies swarm for each one of them. You'll have uh The comments here on how to do the build How to select How to change the route How to consume the service And one thing interesting is that on this eap demo it uses session replication Uh, just to show how we can store states and during the deployment the state will be moved From the session of this one version to another version Of course, this is not a recommended way in In cloud native in cloud applications, but just to show that it's possible to do to do it using eap So thank you so much Autograph But at the last time something went wrong as always Yeah Nothing that I shut down and start again even though the setting the password twice I'll prepare Thank you I forgot three years ago The timer the other time happens to me all the time Now, let's try to have a good opportunity to to go grab some food and go back to another great I go the barrier, but I have to wait for it like 15 minutes Long time Let's see how this one goes We'll talk about our killing cube now Sorry, you will be talking about our killing cube. Yes I see that there's our killing cube and cube in your presentation So cube is for Controlling and orchestrating doble containers for your tests whereas cube cube is Small extension on top of it, which adds chaos containers to your test So might I mean it's just using cube, but it's a different different extension So it's about Dockers Or no docker components docker in general docker And then cube uses cube to have docker containers, but what it does it in between injects proxies or Controls the containers so that you can introduce chaos to your system So you can like tear tear down containers or Introduce some latency, which I'll be showing and this kind of chaos stuff Okay, okay, so it doesn't have any Relationship with the cdr from the cloud enablement team, right? Sorry. It doesn't have any relationship with that extension of the cloud enablement team was working well actually We do work with cloud enablement guys and fabricate To bring all the common bits they have to cube so we have kubernetes control now, which is Brought by fabricate and then open shift from cloud enablement So we tried to make it one because there are three different ones at the moment Whereas the cloud enablement and fabricate is using cube Yeah, it's driving us crazy because of so you're working there. Yeah, I'll work with the With the guy who was doing that that work to Uh, John Van der Lever. Okay. Okay. See you run man. See you. Yeah Okay, yeah, so we we were working with you guys for so I know it's like a sidetrack activity So there is no like no commitment to do it within certain amount of time So it's like small fixes here small progress there, but the main goal would be to Bring it all together so that we have one and then your team would be probably using some small extensions Yeah, okay. Let me just plug in everything Thanks. See you around. Let's see how are you? Not yet. I mean now it's gonna be a bit different way of Presenting because I don't have a mirror ring that I'll be doing live demos. So I will have only this screen But that's gonna be No, I cannot that thing is my uh fedora doesn't support it. I cannot enable it. So I'll be just doing it I'm asking the technical whether this screen Okay, right. So so that we will be showing the same here. That would be super handy Where's the HDMI I