 Thank you very much So hey, welcome to the lecture to the talk about upgrading Django from legacy to latest My name is coming culture, I am the VP of engineering at hacksoft I've been a programmer for more than 10 years and have about Eighth or more years of experience with Django. I'm also from Bulgaria, which is a country Above Greece north of Greece And these are my handles if you want to reach out online The original presenter couldn't make it but fear not the talk is still happening and That being said this was supposed to be a talk from Rato from hacksoft, but it won't be This is just a tribute So just bear with me for a little while And without further ado, let's get into the pragmatic Stuff so the goal of this talk is to provide the value to be pragmatic and to Think about how could we upgrade Django from a an old version to the newest and latest and greatest version And the context is that we've done several upgrades of Django throughout the years and We have some experience with that. We've done upgrades from Django 11 to Django 42 And we've done other upgrades of other libraries like react on other languages So we've got a lot of experience and we want to share it with you And we want to do that because we want more Django projects to be Running the latest versions rather than just some old versions because people don't have the time or energy to do that So this is the quick agenda that we're gonna go to we're gonna Look at what's the problem? We're gonna look at strategies in different ways just ways to solve the problem and we're going to Provide you with an algorithm to solve this problem on your own And we're gonna talk about how to make our lives easier with this algorithm and with some tools And we're gonna talk about how to not end up in the same situation ever again So let's look at what's the Example problem that we're dealing with here. Well, let's say that we have Django version 2.2 7 It's a very old version from a couple of years ago and let's say you've inherited this project or A client has come to you and he's asked that you upgrade the Django You want to get to the latest and greatest version, which I think it's 2.4.4 4.2 4.4 right so There are some benefits of running the latest Django and some of those are the developer satisfaction Some of us are developers. We love working with latest and greatest software Not because when we open the documentation goes to the latest by default But also because there are performance improvement security upgrades and we can use more third-party packages along with new features so it's great and The all of these give us the ability to do faster development, which means we can provide more business value to our clients awesome, but Upgrading Django should not be treated as something that goes really quickly And it's just bumping the numbers in the requirements file and it's over with it's a lengthy process especially if you're doing it for the first time you should be very careful and it depends on a Couple of things so we can look at it From a different perspective. It's like we have the room on the left Which is a kind of messy room and we want to tidy it up It's gonna take a lot of effort to do that and a lot of time too But if we're working with the room on the right keeping it tidy is tidy is going to take much less effort to do So it's better to work with the room on the right Then the room on the left and you can think about that as your Django project if you've let your Django project go and you've Kept an old version for a long time Then you're gonna have a bad time upgrading it, but if you're keeping up with the latest and greatest versions It's gonna be much easier So some strategies to solve the problem There are many ways you can approach this and I mean many ways lots of people have tried lots of people have Completed this and lots of people have gotten to a dead end and had to retry again so let's Think about our approach and to do that we have to answer a very important question, which is How reckless can we be? So how much can we break how much can our project? Be offline if at all is it in production is it currently being used by users Is it from front-facing or just the back office? Can it be offline for some time? Can it endure some bugs or downtime? Should we be backwards compatible when upgrading? This is very important and the most important thing do we have tests and do we have a process? To see if we have broken something with our upgrade So let's look at the first one is it in production if it's in production It's like we need to be very careful because this code is probably used by users If it's not well, we can basically scrap it and start over not recommended, but you can And the second important question is is this mission critical? So is this project front-facing used 24 seven by Lots of users or is it a back-facing system used by the clients stuff to do their work? There is some variance and room to move here so we can think about that in our project and Last but not least do we have tests for this project in the worst-case scenario? We won't have any tests and We've dealt with that before we've upgraded projects that were that we've inherited without a single test without a Single piece of documentation and mind you without a requirements txt file So we've been to the worst. I think I don't know if there's worse than that But if you don't have a QA process, we have to start having one Because we want to make sure that we haven't broken anything which means we might have to write some documentation about it We might have to write some tests about it, and we might have to gain some The main knowledge about the project so we know what to test about we know when we've broken something and we know how to fix it so It's all about having a feedback loop and knowing If the project is working correctly So when you do an upgrade a patch upgrade for Django or any other library You need to have a way to test if the project is still working correctly And that's what a feedback loop is the more tests we have the more reckless We can be the faster we'll move the less tests we have the slower we'll move the more the more the less the less so Let's look at the algorithm the algorithm is we're gonna define it in a couple of steps So first we're gonna know our top level dependencies. These are the dependencies usually in our requirements txt file these are the The things that we've added to the project and they have their own Subdependencies that we have to worry about at some point Then we have to look through the code since we've inherited this Sorry since we've inherited this project we There might be some libraries that are not used anymore and we don't want to care about those We want to just remove those and not bump those and not spend time on them and Then we want to figure out which packages require Django because we're gonna be bumping Django and we care Only about Django in the packages that require Django and nothing else if there are 200 packages in this project We don't want to upgrade all of them just the ones that are Django because we want to do it in one in a single step in a single Task because it's not a single step and then we're gonna do a yellow upgrade Which is just go bump the versions and see what breaks and then we're going to start with the actual Algorithm because everything before that is just a prerequisite. So we are aware where are we with this project and what are we doing? So before we start we need to know our top-level dependencies and how are we gonna do that? The top-level dependencies are that important and used in the projects. Well, we can use big freeze, right? it's a great tool, but it often doesn't give us the exact information that we want and Like I said, we've also dealt with projects that don't even have a requirements file They're running in a production server somewhere And we're giving access to this production server and the client says, can you please upgrade this to latest? So what can we do we can do peep freeze requirements and look through that? This is somewhat tedious for everyone that's used peep freeze It gives us a list of all of the requirements the top-level requirements and the sub-requirements and all of that And it does it's not very useful to know which are the top-level requirements. So to know that we can use package code pip depth tree It gives us a pip dependency tree For the packages that were used and we can figure out which are the top-level dependencies from there if we If you want to move forward carefully if it's your first time grading a large project in Django Or if it's in production if it's mission critical I suggest we do a new virtual environment for the project and start the clean installation of dependencies there It's way better that way in my opinion So what can we do we can install peep depth tree and we can run it and we get a list of packages This slide is not meant to be read from the back role I think but it's a long list and it gives you a tree and the top level of the tree are the top-level dependencies, which is awesome Then we can get those But it's a there's a faster way to do it. There's a command in peep depth tree We can use that lists only the top-level dependencies and the Recommendation here is to add those to a single requirements file so you can start iterating over that So What's the suggestion here always use exact versions? This is very important because if you use one of the other operands for versions Different versions of packages could be installed on different environments, and that's not great that can give you all kinds of and Unpredictability in your builds so you want to go with exact versions and And let's go to step number two remove unnecessary dependencies and and use dependencies So upgrading the Django will lead to upgrading other Various libraries that depend on Django that Django is a sub dependency to like for example Django REST framework Django is a sub dependency on that So if there's one of those libraries that's not used that where Django is a dependency We don't want to care about that. We want to remove it and How can we do that for every top-level dependency? find it on by PI which is this amazing Website which has a lot of information and after that you can Check how it's supposed to be used in this website. You can find it in the project You can search for it. And if it's not used just scratch it out. It's great So the next step figure out which packages require Django, this is very important So how can you do that? we can try for example with Django REST framework, let's say we have a Django REST framework and We want to upgrade all of the packages where Django is a Subdependency because Django as a framework is an ecosystem of a sort and everything works with everything else Django is not a single package It has a lot of packages below that and some libraries can use Subdependencies on Django without caring much about the version of Django. So we want to be safe So when upgrading Django, we want to upgrade everything that a Django is a sub dependency to this is important And we can use pip-depth tree to do that If we call pip-depth tree and we want to see the reverse for the package Django It's gonna give us all of the packages where Django is a sub dependency. This is great because these are the packages that we care about This is the usual tree of packages in a very large project. So hopefully we don't have this large project We're gonna look into a Case with little less a smaller amount of packages so We'll come back to that and for now. Let's go back to the algorithms So we know our top level dependencies. We remove our unused code. We figure out which packages should we care about And then we get to the fourth step, which is yellow. Let's do the upgrade Going to the requirements txt file in a new environment bump Django to latest bump everything else to latest and run the projects run the tests Run the qi process and look what's broken at this point You might be finished, but I doubt it because it's not usually that best case scenario Most likely we won't be finished So but during this process we can identify which packages could cause problems in the long run in which packages Should we care about when we upgrade because the upgrade process is gonna get done in an iterative process by checking versions for each package and upgrading those So let's go back to the algorithm. We've done the first four steps, which is the prerequisite before we start And now we can start we can start doing what we call ladder climbing This is what mid-journey thinks is a lot of climbing. So that's great So let's talk about jungle versioning jungle versions are usually specified by three Numbers a b and c and according to Django's own documentation these a b represent the feature Release number and the c is just a patch version For example in the latest version for point two point four four point two is the feature and we have a patch and The first two letters which are the feature version are mostly backwards compatible which should be read as non backwards compatible and The patch version is guaranteed to be backwards compatible, which means we can count on it not breaking stuff which is great and And Because we have backwards compatibility issues. We need to take a step-by-step approach into upgrading Django So I highly recommend reading Django's release process Everything that I talked about about the versions is over line outlined there. So you can go ahead and read that All right, let's say we're running the following. We have a Django and we have a single sub dependency We're a single package to where Django is a sub dependency, which is Django REST framework We could be talking about something like this, but then we won't finish in time So let's talk about a simpler version where we have just Django REST framework And let's first go into the Django documentation and orient ourselves so What do we see here? We see the Django release is listed And we are at 2.1 and we want to go to latest and greatest whatever that is currently at the time of the stock It's 4.2 so What are we gonna do? We look at the sub releases of 2.1 and we look at where are we and we're currently at 2.1.8 in our example and When we get to 2.2, we can say that we've climbed a single step in that layer. We've done a single upgrade and It's a single complete iteration of the upgrade process which we can repeat until we get to the latest version So first of all, let's get the easy things out of the way and the patch versions We learned are backwards compatible So the easiest thing to do before we even start upgrading and before we even start ladder climbing We can bump the patch version up to the latest available And this is a quick win because once we do that we need to go and release whatever we have on all possible environments This shouldn't break anything and we should test that with all of the tests and all the entire QA process that we have And this is a single step. We can wrap up and we can say this is like the first easiest Low hanging tree we can do And then The feature release versions we said are not backwards compatible So if you want to bump to 2.2 We're gonna have to read their release notes and we're gonna have to see what could be broken and Depending on how large our project is there could be a lot of things that are broken So this is a single step that should be taken into account But keep in mind the release notes are not listed on every patch version the release notes that are breaking are only listed between versions 2.1 point 15, which is the Bottom version and the next one, which is the major release 2.2 point zero So we should go into 2.2 point zero and read the release notes Which usually looks something like this and we can see here what's being changed. What's not backwards compatible There's a whole section in there just for backwards incompatible changes So we can know what can we look for and what can we upgrade when patching? So we can Read this we can go into our project. We can find all of the occurrences of this and we can replace them But we shouldn't really go to version 2.2 point zero because that's I Mean we can go directly to the highest version of 2.2 point two which is the patch version So we can upgrade to 2.2 point 28, which is the patch version, which is great By just fixing the release Uncompatible changes from 2.2 point zero That's good And once we've done that we should really have a look at all of the other libraries that depend on Django because Django is like I said an ecosystem and all of the other packages that are using Django are using different Parts of Django and those different parts may be backwards incompatible. You cannot count on it completely if you leave a big enough gap between the versions of Django REST framework and Django you Can have some strange issues and strange bugs that you have to fix Usually updating the other line the other libraries like Django REST framework is a lot easier So you should probably do it anyway and you should probably keep your all of your dependencies on latest and greatest versions anyway, so We need to find the next Version of the Django REST framework package that satisfies the our Django version as a Lowest requirements and what do I mean by that? we're going to use a tool called pip grip which is searching in the internet in the Documentation for Django for all of the packages and we can ask it So what are the requirements for Django REST framework the version that we currently have and it's gonna spit out some Version requirements so this current version requires Django to be above version 1.11 But we are at version 2.2. So we should look around and see which version of Django REST framework Requires Django to be at that version at least so We can look at the versions of Django REST framework and Django REST framework has used only mostly patch versions throughout this point and we can use We can use them binary search along this to find the version that we want So what this means is since we've already tried the bottom most one We can try the top most one and you can see if that works. So let's try that. So we say hey grip what requirements does Django REST framework 3.14 has and What we get is that it requires Django above 3.0 which is more than what we have so we go back a step and we try that for a while until we find our winner which is Django REST framework version 3.13.1 which requires at least Django 2.2 Which is great because that's what we have right now So this is the version that we should use to avoid any kind of unforeseen and potential problems So we can bump Django REST framework and we need to check that those release notes for something then Django REST framework will break So you see it's not like a process that can be done like in a single step It's some iterative process that you should go into and Depending on your project depending on how much tests you have depending on your QA process and depending on how reckless Can you be you can either do it quicker with having a faster feedback loop? Or you have to be slower and take your time when doing it because you don't want to have issues on production because of the upgrade so you go into Checking Django REST framework What's great about this is everything we use is open source So the source code is in GitHub so you can go in there and check it and check the requirements file for Django REST framework and any other dependency that you want to upgrade Which is awesome and you also have tooling because we're using Python And it has great tools and it can help you along the process just like pip-grip or pip-depth tree If you don't have those in your upgrading some other Language, which is also open source like for example a package in npm. You're gonna have a bad time It has other tools, but it doesn't have these exact ones So you might have to go into GitHub and read all of the requirements of all of the packages that you want to upgrade So here's a quick tip Since pip version 20.3 pip is pretty much Awesome at finding these versions that we talked about If we write in a fresh environment that we are using this version of Django And we leave the version of Django REST framework without We leave the package Django REST framework without a version number and we do just a fresh install Pip is going to find the best suited version of Django REST framework for that Django Version which is great. It's amazing. I hope you get to work with that version of pip above 20.3 In the project that you're updating because sometimes in these projects Python is also an old version and you have to deal with that as well Don't worry about it So if you have this use it, it's great And Once we've done this we can safely say that we've just climbed a single step Of course, we should repeat this process with every Package where Django is a sub dependency and here we've only looked at one and it's tedious already So bear with me After we've climbed a single step we have to make sure everything is okay We have to run our QA process. We have to run our tests hoping we have tests and we have to deploy to all Environments and test there as well hoping everything goes well This is why We have to use exact versions the step number two because if we don't use exact versions We won't we shouldn't count on everything being the same on our environments So use exact versions in your requirements text make sure everything's okay Make sure everything's deployed in all environments and that's a single step in the ladder And then repeat the climb until you get to the version that you want So How many versions should you upgrade? Can you just do version Django 3.0 and stop there? Absolutely, you can do it. Yeah, you can if it satisfies your needs But remember the messy room picture you are leaving your room messy and it's gonna require some more energy next time You want to upgrade and some more code is gonna get written on this project and you're gonna have to maintain that as well so my recommendation is Repeat this iterative process until the latest and greatest version because then you're gonna have the tidy room And it's gonna be much easier to make it even tidier if we want to So How can we make our lives easier? Well, first of all, there are various tools you can use There is a Package from Adam, which is called Django upgrade. It's great I'm gonna leave just links in this presentation. It's gonna this presentation is gonna get shared on Twitter by at hacksoft and on the bottom, it's a It's an article from the same person Outlying how to delete unused code from your application, which is great And then you can give PIP tools a try. I mean PIP is great PIP tools is Awesome it builds on top of that it use it uses a different file for top-level requirements, which is called requirements.in and it's good that you have that outlined and Then it it uses the requirements txt file more like a lock file if you are familiar with lock files from other Ecosystems and then you can generate that lock file with by compiling the requirements in and you can Install all of the requirements with a different command called PIP sync. This is great so All the mess investments that you do in your project pay off the worst that you can have is a project without any tests without a QA process without documentation and Without versions in the requirements file. That's running only on a single server in production somewhere and you have to upgrade that So make sure you have tests make sure you have a QA process make sure you have documentation so you can make your life easier and Probably the lives of the people that are going to maintain this project in the future and Then make sure you have more than one environment. It's really great to have more than one round environment There's a saying that Everyone has a test environment, but some people are lucky enough to have a totally separate Production environment so make sure that you have that and do a proper local development setup This helps a lot because when you're testing in upgrading you're usually not testing in production or even on staging or even on dev You're testing locally on your computer. This gives you a very quite faster feedback loop so you can find mistakes easier so do a proper local development setup and Help your teammates on the process as well And let's go back to the same thing. So If we can summarize the talk so far Cleaning a very dirty room or maintaining a very old project with all dependencies without tests and so on and so forth is very Hard it requires a lot of energy. It requires a lot of time So you don't want to go there and if you're there already just clean it up and make it the best that you can Whereas on the right we have the clean room, which is much easier to get tidier so I Think we should aspire to having the room on the right in terms of our projects Instead of the one on the left so How can we not end up in the same situation again? First of all, we have to realize that we have to keep our dependencies up to date all of the dependencies not just Django and even Python even the language that we use and Even in other projects like front-end which are in entirely different languages and stuff. Just keep them up to date It's not it's not that time-consuming if you're doing it on time because then there's a lot less things to get fixed And your project is up to date and Then this has to get done as a part of the development project process as a part of the engineering process and I am aware that sometimes Clients have a need for business value to be delivered faster and quicker and we have to We have to know that and we have to keep up with that but we can also make our life easier by having the proper tooling for our projects and Saving the time to do those upgrades and keep our code up to date So here are some of those tools We mostly use the panda bot in hacksoft. It's a great tool We use it because we are in our all our projects are in github. So the panda bot has a great integration with github it checks the versions of newly released It checks the newly released versions of all the libraries and it does pull requests for you It opens for requests with bumping the version describing the Critical changes and I think sometimes it tries to even patch the versions though. I'm not sure about that If you're not a fan of the panda bot there are other libraries you can use and you can also make your own library for that You can also have some tool that you Create to keep up with the versions and track that for you and tell you when something's not Latest and greatest and you should think about migrating to the latest and greatest version This helps us a lot We try to keep most of our projects at the latest version because it's much easier to maintain those and it's Much easier to tell the client that we need to have a day to upgrade rather than have three weeks to upgrade the version Yeah, so reduce the cost of upgrading your dependencies by having a good software development and engineering process and By keeping all those things in your in your head and reduce the cost by staying up to date So if it happens so that you don't have the time to do it or you need help to do it We can always help you at hacksoft. We get those requests a lot We've updated versions of Django from one point one to latest. We've also graded Python even typescript projects and all of that So if you need a hand to reach out we are a team of very passionate people that love programming and We can help you out Thank you for the fullest thing So I said at the beginning that there wouldn't be time for any question Q&A, but there's actually plenty of time You were very good on time Do you still one? Yeah, absolutely. I had More than 90 slides and 45 minutes. So I thought that it's gonna be hard But I guess I was speeding a lot. So if you have any questions, please So I'll put it back on the on the stand for sure there Hi, thank you for talking. I wonder if you had any recommendation or best practice about Deprecations like when you when you are using for instance in one one or two two a Specific, I don't know like a subclass or method and then it has been deprecated in 3.2 And then eventually move. Can you can you talk closer? I'm sorry So I was wondering if you had any recommendation or best practices regarding deprecations Deprecations. Yeah, so things that will eventually get out or already out of 4.2 for instance like features or Specific traits that you want to work what you are using right if you were recommending for instance to say as soon as you see a deprecation warning in your step The letter that are stepping or if you wait until the last moment just before it's removed Or in between, I don't know Okay, thank you. So talking about deprecation warnings I think it's a good idea to upgrade to whenever you see the warning But most of the time you don't have the time to do it exactly at that point So one of my points is that you should plan that and you should plan upgrading I don't see deprecation warnings as something worrisome because when you go to the next version you're gonna upgrade it anyway, so you're gonna Think about that whether you want it or not So I don't have any exact Process or steps that we follow with deprecation warnings if we have time to fix those and if they're rather quick We do it if it's just use a different version and you get the same result We just do it and it's okay, but if it's something harder, you really have to plan for it before you do it. I Don't know if that answers your question So somehow thank you Alright as I don't Please Thank you for your talk. I would like to ask is it possible to update The Django version without this iterative process like for example for version two point something to four point something You can try Like I said, there are different approaches you can take on that So this is the one that we've seen works for us Because most of our projects that we get are in production and they are running in they're being used 24-7 So we want to make sure that when we upgrade We don't break anything for the end user and this happens by just having a lot of testing on every step because there are a lot of Deprecations between versions and there are a lot of breaking changes between versions So if you try to do an upgrade from version two to version four straight out It's possible that you don't get any errors in your tests and any errors in your career process But you have broken something somewhere because sometimes these things don't just raise errors Sometimes they just break silently and only the users hit these things when they use the program so since we want to make sure that the software that we are Maintaining is live 24-7. We want to do an iterative process But if you can be reckless enough and you can do it faster That's great. I mean, it's it's awesome if you have good test if you have a good QA process to Assert that the system's still working. Just go ahead and do it. It's it's not a problem You can also try bumping the version and fixing the errors on the spot But then these things that I just mentioned You won't be able to account for all of those so something might break silently and you won't know about it Until your users hit it Thanks. All right. Oh nice Hi, thank you One question should we update it to the newest version or just to LTS? What's your opinion? I think you should keep up with the newest version LTS is longtime support. It's great to have it there. It's a safe place to have it there But I think the newest version is also Something that gets maintained really carefully by the Django developers So if there are any issues there, they Run a patch version and you can count on that So I believe that if you keep up with the patch versions and you go to latest and greatest versions You have a much easier time patching your project the next time something gets released. Thank you Thank you for your talk. This is not a question. I just wanted to really thank you for Django's style guide It helped me tidy up my room in my project and I recommend it to everyone It's basically a pattern of how to Clean up your code. I added a service layer because of it and selectors. Thank you very much Thank you If there are any questions anymore, I also don't see anything online Thanks again come in for the great presentation. Thanks again for your attention Please take any litter with you out of the room and also We will be posting the Twitter of of hex often in the in the group So you can finally you get the links in the presentation absolutely So another warm applause for coming for the great presentation