 me. Welcome him. Hello everyone. I'm Wojtek. I'm from Poland from company S8 Snakes and today I would like to tell you about how we implemented continuous deployment in one of our projects. If you have any questions, just interrupt and ask me. I will be happy to answer. Just an agenda. In the beginning, some theoretical introduction, then more about workflow, so more about process, how to work. In the third part, about tools, what tools we use to implement continuous deployment. And the fourth part, I could say the most interesting, our failures and how we manage it. With some tips for you, maybe useful for you. And just a little, during the third part, tools, there will be a small contest. So listen carefully and I have small prizes who like prizes, awards. Okay, so let's start with it. Almost, almost. The prizes, Polish Sweety, you came from Millcant and a lot of Sweety. Okay, theoretical introduction. Let's start with, what's the most typical way to deploy things, especially when you work with scrum and with sprints. You work for a whole sprint, a week, two or three. You develop things and when the time comes, after the two weeks, for example, you deploy. And then you start to work on next sprint, next things and then you wait. If you infer today of the sprint, you made some change. Then you wait for two weeks to see it on production. This is okay, but there is another way, much more nice for the developer, the continuous deployment. So you deploy a feature. When it's ready, somebody is testing it, QA testing it. And when it's tested, you deploy it. So with this approach, you don't have to wait two weeks sometimes to see the users that they use your feature, but you can see it just a few minutes later when it's finished. And you continuously do the same. So new feature, closing it, finishing it, testing, deploying and bam, you can see the results. This is really nice for a developer because he feels that his work is for users. It's not somewhere in the repository for demo for two weeks, but users use it. So today I will tell you about how we implemented it for our internal project, RMS in the left upper corners name of it. It's application for managing candidates to our company. Our company is about right now 160 people. And as you know, the rotation of developers is quite big, bigger than in other jobs. In SCX-Nex is not that bad, but still we have thousands of candidates that we have to manage and to do it efficiently. So we created an application for it. It's built with Django, Postgres on the front-end sites, Angular. There are other nice libraries like Factory Boy. It's for managing of fixtures for tests. Really nice library, for example. Yeah, and some other things. This is a team. It's 25 right now. I think even 26 developers work on it. Only a few days, some of those for many months. So it's a quite big thing. Yeah, and workflow, how it works. Starting with GitHub code is there. We have just one main branch, the master. And each feature is developed on new branch. Yeah, this is quite normal. We call it feature branch. And when the feature is ready, we develop or create a pull request. So in here it's nothing new. This is more visualization of it. So we have a new branch created by a developer. He creates a feature. He adds a unit for it. Creates a pull request. It goes through a code review. And there is continuous integration, which checks the tests. And the change is in the last part on this slide. So tester checks this feature branch. So developer don't have to in here merge it when it's okay. But tester switch to this branch and test the application on this branch. Yeah, this is manual testing. And when the tester see that, yeah, it's working correctly from the business point of view, yeah, and there is no errors, et cetera. So the tester merge the branch, goes to the GitHub and clicks the green button merge. If it's gray button, so it has to be rubase, then ask developer to make rubase. When it's merged, automatically we back up the production database. We deploy. Tester clicks just one button to deploy. And then we have a feature on a production. Okay, the tools. So the contest. Contest is about the tools. So there will be a bunch of tools. And if you know in which programming the tool is written, just raise your hand and say loudly what's the name. Okay. No, this one is, yeah, raise your hand. This was the requirements. Jenkins. We have Jenkins, yeah, it's written in Java. And here we have four builds. So we have PR testing. PR testing is a build where tester can build his own instance on the branch that he wants. Yeah, so he goes to the GitHub, he copy the branch ID, then he goes to the Jenkins to the configuration, fill in to one of the fields, the branch ID and hits the build, build now. And after a few seconds, few minutes, he have his own instance on the branch and he can test it. We have also staging. This is an instance which is not updated automatically. This is a more instance when somebody from, for example, stockholder wants to see something. Then we have just one instance for some kind of demos, et cetera. We have testing. This is a copy of staging, but it's automatically rebuilding when the branch is merged. And we have live. We have also built in the Jenkins for the live, but it's not on the same server that Jenkins, but it's on the remote server. What's more about Jenkins? Some tips, what you can use to make it more user-friendly for Django. You can use plugins for Jenkins, like a violation plugin to check Pepe, Pylint. There is a nice plugin for tasks that will be run after the build was successfully built. Also, matrix authorization is for better managing of what user can do. For example, if we have a live instance, the build for live instance, we can assure that this live will be updated only for some kind of users, some part of users. A GitHub plugin has a few things there, but we use the automatic creation of tags. We can have also packages on Django site, the nose and nose X cover, which creates tests that are understandable by Jenkins in an X unit, Java, XML unit test format. Build out. Great. Of course, Python. In the project, we use build out, not the virtual length, because we wanted to have just the same environment wherever we build it on whatever server. In this case, build out is better. It's more heavy, I could say, than the virtual length, but you can build many different sandboxes, many different environments on the same server, because the Jenkins that we have there, it has 30 or 40 different applications there. And also in here, you can use receipts. So these are some kinds of plugins for build out. Django receipts, this one creates Django instance. Receipt template is for creating configuration. So you can create, for example, local configuration, local pi for Django from the build out. We have received for UISG for creating, compiling the UISG. And Mr. Developer is a very nice plugin for downloading source codes from repositories, like GitHub, Mercurial, and many different. Yeah, that's right. This is more, almost, almost. This is about frontends, yeah? So the JavaScript is it's the main thing there, so Bower and Granta or JavaScript. We use it for managing the whole front and side. So in here, you have also many useful plugins that you can use. Fabric, of course, Python. Great tool for managing remote tasks. Yeah, so in this case, we use it from Jenkins in the post-process build action. So the live build in Jenkins, yeah, runs the tests, try other things like try to migrate data with South. If everything is okay, yeah, then we have the post-build plugin in which we have just one comment line with which runs Fabric, which updates the instance on the remote server of the live server of the application. And this one is, anybody knows? Yeah, raise your hand. Yeah, that's right. This is not so well-known application, but it's really nice. We use it not in this project that I'm talking about, but in another project that we implemented the same workflow, but in a little bit different way. Gallery is written in Python, and it combines some things from the Jenkins, so you can build, you have a nice web interface to click things without making comments. So even testers which don't know how to develop can do that, and it's also used remote tasks. So you can execute also the same task on remote servers. Okay, this was the tools, and now the challenges. The first thing that we encountered that we implemented the continuous deployment was the restarts of Apache. So on the server that we had Jenkins, we had, our Django application was on modWizgi on the same Apache that was serving the Jenkins. So to restart Django from the Jenkins, we had to restart Jenkins as well. This was stupid, so we changed it. We don't use the system Apache, but we compile uizgi for each instance, and we restart only this one instance that we want. This is the pro tip is a little bit tricky, because if you run build out, in some conditions, you can lose the PID file. So without the PID file, you will not be able to kill the previous instance. So to avoid it, we used masterfifo, so diskfifo, and this is a tricky comment to restart it if the fifo exists, and if there is an instance which listen to it. Next one. We had a problem with migration and data in the instances. So when the tester changed the branch for the feature branch, Jenkins ran the migration if there is an end in this feature branch, but okay, this is fine. But when tester says no, this should work a different way. So he have to revert to previous state, so he will be able to run another to test another feature branch. So we had two choices, or support big migrations, which are quite fine with south, but not in all cases. When it's automatically generated migration, then fine. If it's written by a developer, then developer also have to write back migration. The solution in here, when we are creating new instance on a new feature branch, we remove the database, and we copy data from the staging. Pro tip in here. Postgres don't have a comment to drop all tables. You have to drop one by one. This is a comment that you can run to remove all tables in a loop. Next one. We had a problem with migrations, the south migrations on different branches with the same number. And when the tester merged the branches, he don't see anything that can happen incorrectly. If there are two migrations with the same number after the merge. So we added just a new file, version TXT, with a current number. So if two branches have the same, changes the same file at the same time, then the second one will, yeah, there will be a conflict, and the tester will see that he can't merge it, developer have to rebase it and check the migration as well. The south migration works quite well, but only in cases when two different migrations changes something in different places, in different tables. If two migration works on the same table, it doesn't work that well. We had problems in these cases. Last one. If developer thinks that something can go wrong, then it will definitely go wrong. So you have to be prepared for it, make backups, create tags, so you'll be able to revert to previous tag. And we also had to adjust another rule for testers, upgrades only in the office hours, so there will be somebody in the office who will help you. And of course, monitoring, for example, New Relic Nagils. Okay, this is the last one. You could wondering why Jenkins, why we want Java there, why power grant with JavaScript, why not use everything with Python, because we wanted to make it easy and low-cost. What doesn't mean? We already had Jenkins there for continuous integration. We already had power grants. We already have Fabric. So it was more about reconfiguration of everything and changing the workflow, the procedures for developing code, not to rewrite all the things around the project. So we use these kind of tools. And it works quite well for our non-critical application. So this is, as I said, internal application for managing candidates. And if we had some problems, we had to downtime for a few minutes, and it wasn't a problem for us. What the benefits that we felt very well, new features faster in production. So users, in this case, the recruitment team could use it after hours, not after weeks. And also developers had faster feedback from them. Also less boring stuff for developers, more things for testers. And so we also felt that the feedback is good. And just last reminder, remember about Murphy's loss. Everything can break. So be aware of it and try to think one step forward. Thank you very much. This is it. Thank you for some questions. So please raise your hand. Thank you for talking. You were saying about PR testing. I'm not familiar with PR testing. What's PR testing? PR testing. This is a name for pull request testing. So in this case, the instance works on the same branch that is in the pull request. Thank you for the talk. And in continuous deployment, the idea is that they automate the whole process and just putting away all the human interventions. And from what I see, you have deliberately put humans there, like someone is clicking on the deploy button or something like that. Is this a problem that you have faced when we are fully automating the process? We have seen lots of problems and we required the human intervention or maybe it was that we are not really there for the continuous deployment and we have some things to do still. So in this case, we are talking about this one. So this step is not automatic. The tester have to click, just click one button, build life. It's because we want the life builds in the office hours. So why? Because application is not that used out of office hours. The main users are in a recruitment team, which is also in the office hours. This is one thing. And the second thing, if something will break, there will be admin or other developer which can take a look on the logs or run the revert script. But yes, it could be automated in here. How many time happens that you've got a broken production? Because as I can see, there is only one level of human testing. And for example, in projects where I develop, we have, I think, at least three levels of testing by humans. Yes. So I don't know exactly how many deploys we had. I could say about 100 deploys. Which only few, maybe five, were broken. In only one case, we had to remove, drop the database and, yeah, create it from the backup. So it's not that often, but still it's, yeah, it's better to have some kind of procedure to get it back. Any more question? Sorry, just following on from gentlemen's question there, did you actually automate that process of the rolling back process? Or did you just do that by hand? Yeah, so this is more bash script, which checkouts the, not the newest stack, but the previous stack, runs a build out, restarts the instance. And this is it. So only few comments in the bash script, but you don't have to remember about it. You have just one script to run it. But only some admin or developer have to go with SSH and do it. Thanks for the talk. I would like to know how are you managing your infrastructure? Are you using configuration management? The application, the application, life application, the Jenkins, other instances are on our own servers in a basement of our office in this project. So we, in here we had full access, full, we could do whatever we wanted on the servers. Any more question? Okay, so maybe I will just, this is a contact for me, if you want to talk with me. And just one thing, this is my company. We have many projects interesting. If you, if you want to move to Poland and work with us, just let me know. Thank you.