 All right, so hello everyone again, it's thanks for introducing myself. It's pleasure to be there on the EuroPython 23, I cannot really see and meet some of you as this time I'm online, but thanks everyone for coming and hope you will enjoy it. So let's start from the plan of the presentation. So today we'll talk about like five points. So we'll start from benefits from zero downtime deployments. I will present you a few strategies how to achieve it. I will go through best practices. I will also try to highlight the challenges related to zero downtime deployments. And in the end, I will try to answer the question or help you answer the question. Is it actually worth the effort? Right. So yeah, I'm enough for Novitski. So as you hear, I enjoy web development. I'm mostly like full stack web developer. So even though my last couple of years of experience is front end based related to the JavaScript and the React. I decided recently to go back to some topics related to DevOps. As like for the first time, I started DevOps work at 2015 when when Kubernetes was were just crawling and everyone was on the theme of microservices. So I was also writing some some of them. And the only one choice was was the cloud infrastructure provider, which was the AWS so no one no one really think about using other cloud providers, or they were like really minor minorities. So for in the meantime, I also found the upgrade online startup that that helps small and meat size companies processing they like daily custom orders, and it goes with par with the sale and the settlements. So basically, this says that it's like software for business. We run it on for now only in Poland. But that's like if you're if you're interested, just just just give me a shot. And I'm also like contributor to the one maps software company. This is the house full full of fintech and blockchain experts. So here we provide the complete complete solutions for the web development, including the audits, product design and custom software development. So it's like really, really nice team. I recommended the recommended to you. So let's start from from the benefits. So what are really the benefits of zero downtime deployments. You already hear about that. One of you already use the application that are really are deployed in like fluency fluently. So first thing that came to our mind is definitely 100% of the application up time. So this is especially dedicated by some specific industries like fintech, a banking. So the industries were where the customer interest matters when the end users really matters. So for example, as a user of exchange platform, so I don't want to really miss the opportunity to sell or buy in some critical market situation, right. Also the social medias emails office tools. So everywhere every like every tools so we use it so so so often that that that it needs to be on and we cannot really stop our work because of we have some like schedule for like our office tools. It needs to be like upgraded. Okay, so in desktop application it's quite recently like also happens that you go to work and and you need to wait for for installing the software but but this is not the future where we want to leave. So probably like the Facebook or Google products won't be that popular if they would show some like prompts. Oh, which the system is unavailable. Please try again later. Right. This also builds trust in the relationship with with our customers right the end users. So especially when a number of user growth chance that that no one is using our application in release window go down go down. Right. So if if you're like small small company, you don't have many many customers you can always deploy in the after the office hours right but but if you have number of number of users, it's a bit harder to to schedule. So what if your deployment process is complicated and long and he would benefit from doing that deployment under the food. So I think this is good question. The system efficiency improved. I mean is that like very often zero down the deployment. It's not everything and application are built in the mindset of lean software. Development choose. So not only to deliver as fast as possible, but also money to monitor to measure and quite important to decide on on that data. Right. So how would you that you don't have that. So overall benefit is that complete system efficiency got improved and this is this this is what what what we are aiming also what this is zero down the deployments. So let's talk about a few strategies. So, I will elaborate on three of them that that that I think it's is the most popular. At least what I, in what I faced. So starting with blue green deployments. So this strategy involves maintaining to identical production environments. So the blue, the one on the on the left side and the green and the one on on on the right side. And in between like our user traffic and and and application there is always some kind of load balancer. So this is like always something that that manage our traffic. And at the beginning, once we have version one of the product, it's using some set of servers some some stock. And like the new version is deployed to the green deployment to the green environment and and test and through through this. So once it's like stable once we decide that it's stable or like our automation tools decide that traffic is really suited from the blue to the green. So now you can see that that the traffic is is on the new version, making the green environment, the new life version. If any issue arise like after after that. So if our deployment don't fail during, during the build, if it won't failed on our like checks, or maybe even some manual tests. Then it can always fail later, right. So we should have some kind of tools that will allow us to switch back to the to the blue environment quickly. So blue green deployment is also known as a red block. And maybe I should move more like talking about the red block, which because this this new term is like used quite, maybe not. I don't know if I quite recently but it was introduced like by Netflix, if I recall correctly. And it's also described on the cuba cuba cuba not is documentation. It's, it's, it's all the same. So we just just the name, right. Another strategy is the rolling deployments. So this involves like granularity updating the application instances or services in some sequential matter. So here you can see like four states of the application. So, again, blue is is our live application and the green is something that we want to to upgrade. So we are upgrading like granularly granularly service by service. And instead of like deploying the new version to all all of the servers. The servers are allowed to some like subset the servers. So the previous servers is remaining to serve the live version. I hope that you can see at this point, some difficulties there. I will talk about them in a in a moment. This approach ensures that portion of the application remains available throughout the deployment process. And by this we do reduce the risk of the of the downtime or we eliminated another one that is maybe not that popular, but it's quite quite easy to do easy to implement and in number of application that I was working on is that hot patching and this one involves applying patches or updates is to the running application without restarting it. Right. So, imagine, like, like you could just fix the bug without really like go through the whole deployment rotors in in bigger companies. It's sometimes like procedures procedures that are that involves many people and involves many like many many small tasks to be done. So imagine if if instead of rolling back the new version, you could easily patch it and an example might be just when you have JavaScript application on a search served on on a street, you can easily swap the bundle and and almost immediately allow to to get that. So, hot patching like enables updates to be applied, while still application reminds life and accessible to users. Right. So, this also is done in a in a matter that that no one really recognize that the change has made. Let's go through some best practices now. So it's not actually the best practices of the like like implementation details about the about the zero down time deployment because this this presentation is more more more from the high high level. So we will talk about like set of practices that make zero down time deployments make more sense when they are applied together. And, like, honestly, it's, it's like almost always go together, at least some of some of it. So, starting from the canary releases. So, sometimes they talk that this is a, this is like actually the strategy. I put it like here more in in in the practices section. So what is actually the canary release. So this is something that involves like deploying new feature or update a small small subset of users. Yeah, and I think this is quite critical why it may be noticed as a strategy. So, it makes deploying new features updating to small subset of users or servers before rolling them out to the entire user base. So, yeah, you might already see, or maybe hear about, about similar experiments made by Facebook for, for example, right. So, you have different UI you have different different application than your friends and let's who is who is in your in the same room. So, like, like, run a lary increasing and exposing. Sorry, by, like, run a lary increasing the text. So, like, the developers can can monitor new version, new version behavior collect feedback and detect any issues before a full deployment. So we can like, if we focus on that, the small subset of users is, is our is is like group of our customers that are actually testing our application. So they are actually testing our application because maybe we we just they they've all voluntary accepted to get better like features. So we are on the same stock. That is like, like, where where you deploy the changes first. So for, for example, in an update we have something quite like this. It's like multi tenant application and we have like kind of isolated Tennessee. So we have the whole the whole stock where where there is a customer that that that knows that that is like very like first like like adopter of the application and and gets the most updates more frequently. So once this works. So once the application works not only on the level of like, like CI CCD jobs, like manual tests and so on. We have green light that based on our our best knowledge it's working fine product can be released. So we first release it with with some some some small subset of the users. And then, like after a few days of testing, like monitoring, like, like tracking the user behavior, we decide on deploy to to bigger range to smaller customers. So canary releases are also, yeah, considered as as as the zero downtime strategies. So, as I said, it's like more, more, more implementation like detail for me. And another one, another practice is that like using feature flags. So, so also like the feature flags. I think that that deployment process is is is also like very close to the deploy the development process. So there is also there is some like relation between how would you how do you manage the version of the application and feature toggles or feature flags. This is this is something that's that that's helps us to that allows us to to to to run specific features or or or or hide specific features while deploying new code. So this is something that you like make the end users don't see that the deployment results from the beginning. But but you can just go to the admin panel and tear on the feature later. So this like help you, for example, to to merge and to like partially deploy not complete features maybe like MVP features or or maybe not yet not not not finished one right so you can constantly working on it and once it's ready you can turn it on or you can again turn it on for some specific group of users right so this this provides granular control over the future releases and allows for easy rollbacks if necessary right so it's it's primarily used to review so a be testing review of the effect effectiveness of a change and how the market reacts on the change. So this can be connected with canary releases or future flags for better experience. And that are like gathered there can be important drivers on how application is deployed. So this is something that I observe more in a commerce sectors. So it's like like they very often care about how the sale process should should look like like how fast you should buy the product and how efficient you should do that. And in this case, the small like improvements like like checking different interfaces. I know that the companies that is saying that I can, you can buy item in like three clicks, like with the payments right, I can, I can buy the product with like like four clicks. It's something that that that allows you like there are some researchers that that showed that like if the process of buying an item is long, then you think okay it's it's not something that that I really want to want to do it's not so this boots this book this like computer is not worth the effort I will find another shop where I can buy it. Yeah, so so this this is this is like, like most often used in the sectors where where we really care about like like performance of our user. So implementing automation test processes and utilize utilizing continuous integration continuous deployment pipelines are also crucial for for the successful of their down to deployments. So you probably want to be sure as soon as possible that the new version is ready for deploy that the new version is like stable. So you want to achieve it by just manual testing and making regression tests with with manual regression test with all of the changes. So, like, you need to automate the building you need to automate the testing you need to like automate deployment of the application by this you actually reduce that the risk of the human error and and allows rapid and reliable deployments. So that's that's that's our our goal there. And then monitoring was also something that I mentioned a couple of time during this presentation, and it's still essential to have to have like monitoring systems in place during the deployments. So real time monitoring like had had the text anomalies performance issues or errors promptly. In case of the problem Robux that is should be prepared to revert to the previous version quickly quickly and minimize the impact on the user. And yeah, and this is like the monitoring is very important while the deployment itself, but it's also important. So, after all, when when deployment is done, and maybe you just turning on the future, like, or, yeah, or you or you're making or you're providing different view to to some set of users, like with the AP testing. So, basically, the zero downtime deployments it's not something that you really get for for free. It's also like some downtime downsides and costs obviously. So, since you decide you will always need to take care and ensure to prevent problems that may occur. So, yeah, in the developments, like creating new software. The biggest problem is always maintenance of the of the code. So in this, this time, this time with zero downtime deployments you have a bit more to maintenance and take care. So it's, it's kind of loan that you make make for yourself. So first thing is that the orchestration actually. So the container orchestration platforms, like Kubernetes provides features for zero downtime deployments, not only the Kubernetes but but maybe more specifically the tools built on on on top of it. And these platforms allows you to define deployment strategies and perform rolling updates call scaling and health checks. So by leveraging the container orchestration capabilities, you can deploy new versions while maintaining service availability. So, in addition, you can automate, like automatically handle failures or rollbacks. So depends on the teams, many, many new products starts with Kubernetes. But if you're don't. Yeah, questions questions why Yep, so keep in mind that the biggest problem with the custom solutions is is the maintenance and and the scale. Right. So, if the scale grow up so you need to be to be prepared for it. That's why the DevOps work and the whole architecture application architecture work is is quite difficult because we usually start from small things, small set of requirements, and then it grows and grows in a way that that you would that you would never predict that. So it's always good to start from them from the from the fresh fresh mind. Another big problem, and maybe it's even the biggest problem that makes that, like, like that that's that long on on our on our team is that the database schema changes, because these can be significant like like like challenge. And, like, actually, for for this topic, there could be like dedicated different, like, like completely separate separate presentation. And what I've seen there were even one of of that for zero zero time immigration is in SAS applications today. And I encourage you to take a look at this, but just like quickly going through the possible strategies that you can achieve. Is that like performing the backward compatible data database changes, like using database migration tools. Or like providing some kind of like database sharing and and and replication. So, um, so basically the biggest problem with immigration is still that your end users, like, cannot really, you cannot really stop, like prevents them from using the the certain table once you decide to rename it or or move some of the data. So, so that's why on every deployment, your code needs to be your code and like your then your database needs to be prepared. So actually, like, like nothing, nothing like affects using the database. Um, there are the real challenge is also team attitudes. So when you choose a team that's already built system like this, and you build product project from from from scratch. So we are quite of like on the winning position, but migrating existing system without making like cutting edge decisions, like highlighting the cutting edge decisions. Very often rewriting some set of the application from the from the from the scratch on or the whole application. And then the migration that never as and never ending story and I'm a I'm the witness witness of, of many attempts of migrating to cloud, like migrating, like, micro, like to micro services. Like, like optimizing the deployment process. And like, without really, like, I've seen, I've seen, I've seen it work on the ones. Once we, when the case was like completely rewriting the couple, couple of the services. So it's really hard to rewrite the base of the application. Right. And fortunately, in the Ulan labs, they're like great teams that that that manage to, to, like, set up new projects with with with with everything that is that that that could be needed for zero. So the question is, is it worth the effort so I can see that we are getting out of the time, I will try to finish next, like, like three to four minutes. So obviously, there is no simple answer to that question, or, or obviously it's, it sounds it depends. So it depends on manufacturer that are specific to your product, or company. So, like, what your product does, what are your customers, how valuable is is for my company to have zero down time deployments. How often do you need to deploy right so they're like, very, like correlated a question that you need to ask to yourself, like to your company. Like, all of the like challenges have a price, you need to pay to pay right so start from investigation if it's actually something that that that you benefit. So, if you start to rewrite the application infrastructure, anyway, so if you're in that point that you might building new foundations anyway, just like, try Kubernetes technologies like like something something something that that that would would help you to start, like to achieve zero down time deployments if it's needed, really. So, yeah. So let's let's let's quickly recap on what we discussed today. So we went through strategies. So there are like three main strategies blue green deployments, which is about deploying like second identical production environment, and then we were swapping that the traffic. And then rolling deployments where we deploy services in sequential manner, hot patching was about deploying challenges that don't need to require said that not don't need to need the service to be restarted. Right. And then practices so we went through cannot releases. So this is releasing changes to the subset of users. Right. So, yeah, feature flags like that the future toggles with no deployment needed so you can deploy and later you can turn on the feature the flag. So what I think is for better product deployment decisions making see automation tests to speed up the process and reduce the human error. So this is this is really critical, as well as monitoring, which is for safe rollbacks and as well for a bit testing and So is it worth the effort and the answer is hard and not very clear. So, like just just like you would need to go through that question that I give you. Do you really need that. Maybe like you can just automate a lot of stuff already and this this will be will be satisfying for you and you don't need to care about migrations for example, and like, like 20 seconds 30 seconds off of deployment, like outages is fine for your users. Like, the advice is that small starts small. Because they're like their plenty best practices in that area that can be followed to build better products and mindset as well. So you don't need to do everything now, you know, go to your products and and say, Oh, we are implementing ZD now, we are getting rid of our infrastructure we're writing it because in in in most of the companies, it would take years, right? Unless unless you do something really small. So thank you everyone. So, yeah, there's time for question you can find me on like no virile username or you can use this query code to find me on LinkedIn. And that's that's it for the thanks a lot. Wait for the questions. Thank you for fall. So as always, we have the microphones if anybody has we have like 10 minutes of questions so that's plenty. Yes. If anybody has any questions on discord online, you can put them in the chat and I'll spell them out for you. Does it seem that I was pretty clear this time. Okay, so good. So I have a question. There is a pattern in when you want to do zero downtime, which is to always have a version, a superior version which is backward compatible with the previous one. You mentioned about database migrations. When you do this on the database level, is it enough to prevent any issues on production, or is it too simple or on the real case in real production use cases. It's always more complicated than this. And so you're asking if if like having that pattern with major minor and patch versus versioning, right. Yeah, yeah, just just just if you could shorten the question. So about database migrations, you mentioned it was something hard to achieve with the period on time. My question is, if you ensure that each release is backward compatible that your shimas are always backward compatible with the previous one. Is it something that you do first. So so so this is something like like like without so some releases don't I don't actually need to like to touch the database and it's fine. So sometimes like like you provide new feature and and and like ideally is that once you want to deploy new feature. Or that new database structure new database schema is already there. So this this this is this is the best like like like approach, because like, during the deployment, we don't actually deploy everything in one so you don't like, migrate the schema, make the database migration maybe, and also change the code. Right. So, so this is this is something that that that that you don't do, you usually do that, like, especially if you use for example Django with that codes to migration, like, like, like, like it's quite, you know, close. So, you need to like change a bit a bit like the way you're thinking to, to like, always write the so it's not only about, you know, writing the migration that that will just if you remove the fields. It will just create the field, but it's more it's more more about, like, if you if you create create a new field. You need to like, like, that the current version version of application cannot you use that that that the new field that you want to migrate. You need to like, like, like met that make that jump so so first you, you create that field, and then in another deployment, you, you use it the same for for for removing, removing the fields renaming. Actually, remain renaming the table is almost something not not not something worth to do. So you need to have some some way of of don't don't using the the structure from from this deployment you just you just need to have the database structure from the previous one and then and then just like deploy the features on top of it. Thank you. Thanks. Hi. Thanks for talking to actually I have two questions. One is created. I'm actually one is kind of similar to the question that you just answered, but so you mentioned about the best practices and the future flags and my question is about data data migration. So basically, he said that, you know, you can use the future flux and roll back the changes that you just, you know, deploy. And I wonder about the best practices when you actually have to migrate. I mean, you have to proceed migration of the database, we had a feature flux, and then if you want to get back this feature. So what's the strategy for that. So, again, like feature flux is something that you turn it on after the deployment, which means that at least so so so in this case, what when you when you use the feature flux, you could actually deploy the new feature with the with the flux turn on so imagine that that this is this is some some some some like completely new, like something created to complete a new table. Right. So so like if you use the feature flux. Or maybe let's start if you don't use the use the feature flux, you would need to first deploy the creation of this of the schema that migration. And then in another deployment release the code for it. Or if you use the feature flux, you could start with the future turn off. While deployment created the the table, like me like like run that that that that migration schema. And after is done, like after sometime just just turn and turn the feature on. So, yeah, so so so this is the this is this is what like like how how the future flux could help. Obviously, you could you could have some, some cases like there is, yeah, a bit more more complicated scenarios where, where you where you swap to new new field type. Then, how to make it back but backward compatible if you turn on the switch maybe some of the data will be will appear in the in the one field in the one column. And if you turn off the feature, it will appear in a in a different field. So, so here as well you need to write kind of like like a port adapter. A pattern maybe to to make that, like your API is reading sometimes from one field sometimes from from from another field. So you don't really lose lose that data, or you don't make it, you know, make unexpected behavior on the user. Okay, thank you. And another question about future flux was the best strategy for clean up future flux. So, yeah, this is that's true this this is very often mess, especially if, if you don't have process for creating the future flux and the developers do it on your own. And one thing that that work work from from from the team where I'm working now is something that we, we have a script that manage the manager the future flux. So, we force like people to write down what, what is the name of the feature feature flag, what is the description description what it actually does. And the script also, not only like create, because like like we keep the future flags in the database for like for for doing that from from the from the admin panel. So, the script, like, and ensures that things that are written on that script on that say the dictionary whatever reminds the same so it it would removes you removes for you and like like some old feature for flux, or creates the new one creates ones that that that came with the new deployment. Thank you. So, for everyone purpose, if you're interested, the way how we deal with feature flags in our code is that, well, on the code bill, we don't store them in database. So, we basically put it to do comment next to the use of the feature flag with the date of the release when we, when this feature flag is expected to be removed and become a permanent part of the code base. And when the build runs people get notifications and a list of things which are actually to be need to be removed. Thanks for that. So, quite, quite related to this is I think it's like like that, like, like in general like practicing off of like deprecating some some pieces of the code so it's like really I think it's it's worth to write it in a way that that that at the end of the release you have you have a you have a list that your users are notified about what will disappear soon or or what kind of immigration they need to change to to be up to date with with your future releases. So, I think that was it. We have one more minute if anybody has any other quick question. Otherwise, I guess we're done. Thank you very much for your talk. It was nice listening to you. And have a great day. Thanks. Thanks a lot. Have a great day of the conference and the rest of the week. See you.