 Well, hello and welcome and it's time for another Dev Nation Live. I'm super excited today to have Etzin Yanaga with us. Say hello, Etzin So Etzin Yanaga is with us today. He's specifically going to be drilling down on zero downtime database migrations So for the world we're living in now with cloud native computing and people working in containers and microservices We tend to forget about that little old monolithic database that we all have and I got to tell you The presentations I deliver I tell people the database will the date database administrator will hunt you down and kill you Right, that's a common problem but Etzin's been traveling the globe for the last several months and studying this bigger space around data and Data within the context of this new world of DevOps new world of microservices and cloud native computing And he has a lot of great tips and tricks and techniques to talk to us about and Etzin I know you also wrote a book on this topic also, so I imagine you'll talk about that as you get to your session So at this point, Etzin take it away Okay. Thank you very much Burr Well, oh, it's a great pleasure for me to be here talking to you at that national live I hope you can enjoy the content I'm presenting today and let me share my screen I Hope that you all can see now my presentation And as Burr said today, we're going to talk about Zero don't tie migrations and particularly how can you update your database schema with zero don't tie migrations? Which means that you'll be able to like roll out new versions of your applications without disrupting your users into production My name is Ed Sonja Naga. I'm a director of developer experience at Red Hat Just in case you want to follow me. My Twitter handle is at Yanaga And I talk a lot about DevOps microservices Particularly like data in a microservices world Java and software craftsmanship and other other subjects to I I'm also a Java champion and a Microsoft MVP and the book that Burr was talking about is this one Migrating to microservice databases in February this year. I was able to release this book from Riley and just in case you want to electronic version of the book Red Hat developers has bought the royalties of the book. So it's available for free. If you just go to this URL You'll be able to to download electronic version Or if you don't want to type every all of this you can just go to my Twitter profile at Yanaga The peanut treat has a link to the book. So if you think it's easier, you can go there and check the URL too so All of the things that I'm going to discuss today They're available in in chapter three of the book. So I'll be talking about zero downtime migrations But today I'm going to show you something that I've It's the first time I'm presenting to to an audience How can how can you do that in prets? So today we're going to demo how these zero downtime migrations work in prets so just keep tuned so be able to check that and If you want to start this discussion about zero downtime So I have to tell you that the single requirement for Sing the for for zero downtime Deployments in your production environment is for you to have a blue green environment and when you have a blue green environment to perform blue green deployments It means that you're going to have like two equal production environments and you you will be Very likely having two different versions of our application running at the same time in production. So we're talking about The code it might sound like easy Because this kind of blue green deployment scenario is the things that we've been doing for the past five or six years Today we might have like much more advanced deployment scenarios If you're using platforms like kubernetes and open shift you'll be doing you'll be doing like canary deployments or rolling upgrades But it's sticking to the simplest blue green deployments This is the the the single requirement for you to be able to achieve zero downtime But if you think about that having two different versions of your application running into production Regarding to code you might think we'll be thinking it's not that hard But if you think about state state is definitely much harder. That's why I like to say that state We've stated with code is easy, but the state is definitely much harder And when we're talking about states you can have like two different kinds of states You can have a femur states Which is kind of state that you usually store in http sessions and you also have a persistent state Which is kind of state that you usually store in a relational database system and We are going to do specifically with relational databases on the stock so because After traveling and talking a lot with different teams and people worldwide Of how are the chat how were was the challenges that they had trying to deal with with data In a microservice world number question that always receives is this How do I give my relational database and since we're talking about zero downtime migrations? I'll have to tell you that you're not allowed to do like migrations manually anymore And to give you a technical term Migrations is the term that we use in database words To say that is the set of cco statements and possibly code statements that you need to apply To transition your database schema from one version to the other. Okay, this is the the concept of database migration So if we're used to like emailing your cco updates script To your dba through the night or if you're used to attaching that to a geo issue You're not allowed to be doing that anymore You have to automate everything And just to show you two tools these both flyway And liquid base. These are the most popular database migration automation tools in the java space Not just because they're in java But I just learned that they're they're very popular in other for other kinds of developers too except from the guys from the ruby on rails world because ruby on rails has their own beauty migration to What if you talk to php developers or Python developers? Or even dotnet developers you have many people using flyway and liquid base on the on their deployment pipelines to update their database my ischemas automatically as they They release new versions of software into production So how can we use these tools to achieve what we call zero downtime migrations? So if you want to achieve zero downtime migrations, I have to tell you that If you have two different versions Of your application running at the same time into production, they must be back and forward compatible Which means that your database schema must be must be compatible with the next version and the previous version of your your application And it also means that we're not allowed to have migrations with like a hundred seco statements at a given moment You need to break that into big stats, which means the smallest possible batch size So for many of if not all of the cases you're not allowed to issue like two or three or a hundred or a thousand secret statements into a migration Each one of your migrations will have a single secret statements It also means that If you're using like like an update statement to your database and these update statements take like five minutes to run You're not allowed to have a five minutes lock time in your database because it means downtime into production now. We'll have to deal with We've we've this long locks by sharding our database updates So if you're updating like a billion rows at a time, maybe you're not allowed to do to do that Into a single secret statement Maybe you need to shard your statements to to update your database like a thousand rows at a time Or a million rows at a time There's no specific amount that you can do for that You need to rehearse your migrations in a test environment. For example for just for you to access How much time does it take to update your your database then you'll be able to measure? Well, maybe one second long time Uh downtime is something I can bear into production. Then you just stick to the size. Okay So sharding is a very fancy word in the in the data world just to tell to you that well You need to update your updates. You need to to split your update statements So if you're used to do something like this Outer table customers rename column wrong to other thing Then you're not allowed to do that anymore. Maybe you have to split your secret statements into something like this Maybe first you have to add a column. Then then you have to To update your your database using shards. Then maybe after some time you you'll be able to delete the column so After talking as I said to many many different teams world wine I was able to collect some of the scenarios that the these teams are using successfully into production To update their database schemas with zero downtime So the four main scenarios that I was able to collect were these ones Add a column rename a column change the type and format of a column and delete in the column And to be very honest with you and most of the teams that I've talked to that were able to do that in production Were from startups or very small teams very engaged developers Because they were able to deploy like multiple versions of their applications in a week or even in a day But in the past year I was very happy to see that after publishing my book. I was able very happy to see that some enterprise teams We're also trying to engage with some of these practices and I've received some some some very Very good news from some of these teams that were they were doing these zero downtime migrations into production So I hope that after seeing this session you will be able to apply some of these techniques into your production environment to achieve this zero downtime migrations And I'll try to explain Some of the scenarios very quickly because we have a demo for one of them Today so the first scenario that we have we'll be able to see Is add a column if you want to add a column into production with zero downtime You can see we have like four different steps. We have step one two three and four Each one of these steps is going to be a new release of your software into production So it's going to be a different version of your your application into production So in the first version, you're going to alter table at column the second version Your code computes a read value and writes to the new column third version you update the data using shards And fourth version It might be the last one code reads and writes from the new column And I have to tell you of course between each one of those releases into production You have to open your database cli and issue some cico Select statements to check if your data is being written in the wrong In the right column in the appropriate format and something else So this this is the simplest possible scenario and again all of this information is described in all my books I'll be able to get like much more details if you read the chapter of my book and Next scenario, which is the most complicated one renaming a column with zero downtime Yeah, you can see like we have at least three defense six different steps for you to be able to to achieve zero downtime migrations First version alter table at column second version your code reads from the old column and writes to both Third version you copy the data using small shards Fourth version your code reads from the new column and writes to both Fifth version code reads and writes from the new column and six step Sixth version you delete the old column much later not right now because You know deleted a column is a destructive statement You lose the data into production and some of the secret ingredient from all of these zero downtime migrations Is that is that I never issue a destructive statement into production your Your database schema is always back and forward compatible because the previous version of the of the data is always there That's why you have much more confidence when releasing the software version into production in fact These migrations are not the only way for you to be achieving zero downtime migration to production They are also the safest way for you to be Updating your database schema into production because you never lose any data And this is the scenario that I want to show you today in prets Because I feel many people after reading these steps and thinking about that I don't know if it works into production because you know I'm suspicious about that And I wanted to show you a very nice demo. How can you achieve that because I'm not the only person saying that it works because as I said before I've collected the spreads from many different teams worldwide and they are using these steps They're using these scenarios Successful into production. So let's see how can we achieve that into production too? so now I'm going to show here my project. I created a very simple wild fly swarm project Using some domain driven design techniques. So which will probably the subject Of an another session that we have later. So here I just created a customer class And that's a jp. That's an ejb jpa jsf application, which is basically a crud For a customer entity and this customer entity It has some fields. It has an id. It has a name and has a surname Okay, you might be thinking well, why the hell did he put like this name and surname class name surname? Well, in the end I'm storing this information as varshar columns in my database But I'm using custom types in the domain driven design world. We call this value objects. So We you also have to code something to be able to to to store and retrieve These things from your database if you're using value objects, but again, this is the subject for another day Just bear having in mind that this is our Playing Java classes and they are being just think of them as the strings because I'm storing them to To my relational database as varshar columns So what do I need to do? What do I want to do? I want to rename this field surname Maybe somebody asked me to rename that to last name So I want to show you that this application is running successfully today right now And I have my application here I want to run that with my wildfly swarm plugin So while I'm running my maven build. So it's a wildfly swarm swarm plugin. Basically in the way it's configured in my application It's packing everything inside a fat jar, which in this case will not be a very fat jar. It will be a finger version of my jar and it's already running my application Okay, and since I told you that you need to automate everything your application. Yes wildfly swarm already has a plugin beauty for for flyway so flyway Is a database migration automation 2 and if I just create a special sql files in this Um a folder db.migration and I use this special naming for my sql statements. It's automatically will create an and An execute these statements, uh as soon as I'm my application boots up So that's what happened here. I had an empty my sql database in my machine So they just created say boat and now I have inserted two different customers myself and work so if I open here my Application which is already running you can see that I have had this this data already populated And if I go back here to my terminal I can see yes, the data is there if I want to show you the tables Basically flyway created the customer table for me and also created this table schema version Which is uh the table that flyway uses to keep track of which migrations were successfully applied already to my database schema So if I get here, select my schema version You can see that I have like my initial uh Migration successfully applied on this sql from this sql file. Okay, so I have the data First version is running. So if I want to check its application if I want to check here It's an aga. I'll put the number one here. You can see what's successfully updated if I want to create another um Customer here. I'll put the names of the other people from our team. So I added a reference working successfully. This is the first version of My application. So if I want to go back to the heat to talk to To the steps first there. I need to add a column that I want to to rename So we're in a column. I want to rename for a surname to less than first the step that I have to do is to add a column So what do I do? I'm going to get here. I'm going to keep this application running. So keep you have keep in mind that's my My qn version is running and I'm going to update the source code There so I'm going to alter table and column and you know, I wanted to be very cautious about that So all everything that I'm doing right now Already committed this this application to git and zabela mongit hub. I'll share the link later Or maybe we're already shared the link on the chat so Here I'm going to I tagged some of the versions. So I had version one. So I'm going to check out version two And let's see what changed here in my application code. You can see If I come back here to my customer class, I didn't have a change anything yet But I just added a second migration to my application, which is at less than column So first I'm going to modify the surname column because currently has a not no constraint I'm going to to remove the not no constraints and I'm going to add this second Column called last name to my database Schema. So that's the only change that I've performed here so far And let's run this application into production. Let's see Now I want to do the same I swam run So now we still have like fly. I'm missing an F and that Swarm http port. I want to change the port because I want both versions running into production at the same time So I have version one running here and now let's run version two Which will run on port 8081 So let's just wait for It to finish the deployment If you want to know usually warm flies form like starts much faster But since I have a lot of things running in my machine and I also have this hangout running on the background It takes a lot of cpu So if I get this new version of my application You can see they have two different versions of my application running and they are holding the same data If I want to change that I change the key here to version two And it's working successfully so on version two Yes, I just added a column no big deal. If I add a new thing I add another person in my team I'm going to add don You see it's working properly. Both versions are working properly into production. Okay, so I just did a blue green deployment I rolled everything to my new version So Maybe now that I've seen that everything is running. Okay, I can stop The first version of my application because I only need now the second version of my application to be running successful into production and second step Now I've added a column ready. Now I need to my code reads from the old column and writes to both. How do I do that? So I want to get here in my code So I'm going to open another terminal. I'm going to Out the third version So if I go back here to my id if I go to customer class What did I do? I just created The surname few because I'm not messing with the surname yet. I'm just adding a new field called last name I'm using this jpa tags called prep persistent prep day to be to To make sure that my data my last name column is going to receive the surname column whenever I save or whenever I merge my my objects And here in the center too I'm getting like this dot surname and this that last name because I don't want to be correct only on the database layer I want to be correct here on the code layer too. That's why I've just changed the center to set the last name And here I'm changing the the prep resists and prep dates Notations to be updated in my last name column. So this is the second step Let's run my application to productions to check if it works so odd fly swarm run Swarm htp port and that's 8081. This one is going to run on 8082 So I already have a column in my database now. I'm going to release a new version of my application Into production. I have version On port 8081 and I have another version now on port 8082 Okay, now it's running. So you can see here. I'll check this This version is still running successfully. Now, let's go to 8082 You can see this application is running at least at least it's reading from the database But let's see how it works when I try to update that I'll try to update myself And you can see data successfully updated if I check that here. It's running too Let's see what what's happening inside my database So if I select everything from customer You can see that whenever I update or whenever I create a new New one in my application. It's going to write both surname and last name. So update it at some So now I have the the data value in both surname and last name columns. Let's try to create a new A new customer here I'll add kamesh So you can see I'll add a kamesh on my new version It works successfully if I get back here to my old versions working properly, too If I go to my database, you can see that all the updated records and all the new records. They are already right into both Columns, but the old columns the old rows They still have a no value in the last name because nobody's updating that so that should be our next step Everything is running properly So well, I just rolled out a new version of my application. Now I can stop the previous version from running I'll stop this version from application and I'm at version three. Let's allow check out version four Um, now it seems that I'm in version four. Let's see what changes here I'm going to update my database using shards since I only have like five rows It's not big enough for me to be charting But just have in mind if I have the billion rows, I will be able to have to split everything That I'll be doing right now. So in the this version of my my application I'm going to update the customer set last name equals to surname. I'm going to roll this migration into production. Okay, so Let's run wildfly swarm And now I'm going to run Orcs 8083 Let's wait for that And you know, they always takes long as we're looking that at and doing something for a youtube live transmission Okay, now it's ready. Let's go to our application. So it is not working more and now let's go to localhost 883 My application is running successfully. Let's do previous one previous one is curing Successfully two and let's see the data on my database. If I got my my sequel terminal You can see that all of the columns are populated Right now if I decide to change here Before you can see that my new version is running Is updating successfully everything if I want to add a new Let's add a new guy here at pole, which is watching us If I had pole it adds successfully on both versions and if I go to my database both of my versions Are reading and writing the same thing from both columns. So that's the version that I wanted to show to you So now let's get back here I did a success for deployments. I can stop the previous version and let's go On version five, let's go to the next version If I did v5, I'll go back to my presentation here and just copy the data using small charts Now I can change my code to read from the new column But keep writing to both because I need my previous version to be working successfully So that's what I did right now if I go here to my customer class you can see That's now Instead of being updating the last name with the surname value. I'm doing the opposite I'm updating the surname which is the old column with the column from last name Which is the current property that I'll be using in my code and also here In my setter I'm setting the uh, just change it here because I'm setting both variables at the same time You can see later. I'll be renaming this get on the setters to surname too. So it will be a successful migration So so now updating surname I'm just using last name into production because you can see here get surname is returning the value from last name column And surname is not being used anymore, but it's getting updated on the database So let's deploy this into production and check if it's going to run this one It's going to run on port 884. It's ready. So now let's open So 884 New version is running previous version is running too. So let's try to Create a new customer. It's going to be jessica And sorry jessica. I don't know how to have your name. So it's going to be jessica crow And if I get let me get back here to the previous version is running I can update here my data too in my previous version Go here to my new version. It's working successfully so Let's select everything from customer and you can see My application is working and writing the correct values to both columns At the same time and just in case you will do you're thinking well am I not cheating with wild flies form or something like that So let's stop the previous version for running. I just have the current version running and I'm gonna get the current version running here and I'll get Version running version five. So I'll copy my artifact, which is zero time immigration swarm dot jar to another folder And I'll call that v5 dot jar and I'll run this Current version With java dash jar. You can see wild fly swarming bootstrap. Let's successfully again There's a question came. It's been really I'm sorry I threw a question at you while we wait for that start and For sequel update command and a separate version file because of clarity Or is there some real use case behind it into a single sequel file? Now which one the update statement Yeah in version four you had the sequel update command and a separate file Yes, is there any is there a reason for that? Yes, it is because first I was I want to use the fly way for updating my database And if I use this is as a fly update statement I'm committing this migration into my source code and I have to release a new version into production So this way this this new version will be compatible with the previous one And it doesn't happen all the time if you just decide to use this update thing using your database cli Okay, so I want to keep track of all the changes and it's a very good prep If you do that everything all of your database migrations fool you through your source code repository That's why I'm issuing this thing inside my my Flyway folder here into my project Okay, so it's a requirement. You need to release a new version of your application production for you to be able to Next version to be compatible with the previous one. You can't just update that using your cli Okay, so have a new version. So let's get to the last release which is My code reads and writes from the new column, which means everything is working perfectly fine. So let's release the last version Let's get right ahead zero down time As version v6 If I go here to my project you'll be able to see that Version six. I don't have the surname field anymore I renamed my garrison status to last name. I don't have any other reference for that too, I don't have any other new migration happening here too Uh resources Okay, so that's Dave steps really so now now let's just run that into production Remember 85. Oh this one. I didn't uh give the parameters or it's running on 8080 So previous version running on 8080 and latest version running on 8085 Let's wait for you to deploy We're almost done with the demo And it's ready. So if I go here Give us version That's my application. I'll get to my previous version on htp for 84 Let's go to the latest version Just 85 You can see my latest version is working properly Here up already is ready. So if I go here to 84 Previous version running successfully too, but you can see if you go to my previous version Can I create another person from the team? I'll put andre If I have select assist uh from customer I get it's writing for both But if I get to my new customer and I get to update and create a new customer here I'm going to add Sarah It's both are running Successfully into production. You can see everything. It's fine. But if you go to latest version It's not writing to the previous version to the previous column any more certainly because it's not needed but both versions previous one a new version are working properly into production and I was trying to use Everything is working successfully into production now Now that my code's breathing rats from the new column only I can much later delete the old column But I never delete now because you know, maybe I need to roll back my application to the previous version So i'm just going to mark my version for deletion because maybe the dba on the maintenance window He will he or she will be able to delete the column because after performing a backup. Okay, so That's the most complicated scenario Renaming a column and I hope I have shown you to you that I've done that successfully running different versions of my application into production and I have other scenario here change type and format of a column and if you can see a pattern Change type and format of a column and rename a column. These are exactly the same step So everything that I did right now you can do that if you want to change the type of format for And less scenario delete a column. No, you never delete a column into production Because that's a destructive statement you lose the data So you just stop using the read value but keep writing to the column because you know, your previous version might be needing that In a third release you maybe if everything went well You might be able to stop writing to the column and again if everything went well again Maybe you can delete the column later. You can just mark the column for deletion and your dba will be able to do that Okay And another question that frequently arises that what about my referential integrity constraints and not no constraints too There's no easy answer for that. You have to drop everything and recreate These constraints when the migrations are done So just as I removed the not no constraint before if I wanted to have another step to my migrations I would just add like a modify column and add the not no constraint back again to my application And if you want to check what I did Step by step in all of my source codes I tagged each one of these versions of my applications on git and I've uploaded this The source code on this git repository a github.com yanagas zeroed on time migrations So we'll be able to check everything that I did right now And That's what I wanted to show you today. I hope it has been like Very like productive 39 minutes of your life And I'm ready to answer any questions that you might have for me Well edson, we're totally out of time as far as the session goes But I do want to leave you with one quote a fantastic quote that we got from reverto It's in fact the main challenge when moving to microservices Which answers the question of what to do with this monolithic database? How can this be done etc read the entire book it is excellent So reverto obviously loves your book and again I also highly recommend it and people can grab it from the link that we provided in the chat And of course we can all tweet it out as we can But I thank everyone so much for your time today We try to keep these relatively short and sweet. That's an awesome job on the demo I really love it and then this recording will be available to the masses at large on youtube Almost as soon as we hang up the call today. Thank you Thank you. See you in our session