 Okay. Hello everyone. We're going to get started. This is moving to Python 3 slowly. So what we're going to talk about today is the new hotness, Python 3, right? It's brand new. Everybody should be using it. So not really. Python 3.0 was way back in 2008. Even what most people consider to be kind of like the bare minimum Python 3 you should be using, which would be Python 3.3, dates back to 2012. So this is not a new thing we're talking about. But there is this kind of huge amount of Python 2 code that's in use. And I'm here to kind of present some ideas about how you might start to think about migrating code that you're responsible to from Python 2 to Python 3. So a little bit about me. My name is Roger Lopez. I'm a principal engineer on the tech operations team at Warby Parker here in New York. So Warby is, this is a New York crowd. So is anybody not familiar with Warby Parker? Okay, cool. So at Warby Parker we make affordable eyewear and sell them directly to consumers for every pair of glasses that we sell. We distribute through our charity partners a pair of glasses. We've done this over a million times already. So a fantastic company we're working for. At Warby, how do we use Python? We have applications for our e-commerce website, our retail stores, which we have, I think, 32 across the country now. Our retail stores run custom point of sale application that we've written, all of our customer experience tools, all of our supply chain, all of our order fulfillment is all Python code that we've written. And it runs everything from the front to the back and everywhere in between. We have around 60 engineers. We do the slides a little old. I think on last count we're doing about 400 deploys a month. And this is all in one giant repository. I just ran this before this slide. We have 173,000 lines of Python 2 across around 850 files. So this is not like a trivial migration to move to Python 3. So, and this is all, you know, the very definition of business critical code. Everybody, I just a note about like our velocity right now. I gave this same talk in the Caribbean in February. And at that point in time we had 129,000 lines of Python code. So it's growing rapidly. But on our team, everybody would like to be writing Python 3. We all know the cool things that you can do in Python 3. Some of the issues we run into over and over again are kind of a symptom of the way Python 2 treated things like strings and Unicode and just some of the things that, you know, that would be easier in Python 3. So there's not really any dissent on our team about whether we would like to run Python 3. It's just how do we get there. And this is kind of, it's kind of the goal is that we would like to migrate our application to Python 3. But we have to absolutely find a way to minimize the risk. And almost just as importantly, the disruption. We have 60 engineers working in the same repository all the time. So the last thing we want to do is cause like a huge amount of disruption to the other engineers by, you know, doing some sort of like major switch. So that's what we're kind of setting out to do. And the goal of what we'd like to achieve with all of this, just in preparing for this talk, I looked at the word migrate. I was actually kind of surprised to find that there's a like a variation of the definition specifically for computing. So to change or cause change from, or to change or to cause to change from using one system to another. That's simple enough. We got this system over here, now we're going to use this system over there. That reminds me of what I call the Indiana Jones swap. And we've all been part of projects that do this. It's like how do we move from this thing to this other thing? I'm going to make a branch of this repo and I'm going to go way into like, you know, just going to go into my cave and maybe take my team with me and we're going to come back six months later and then we're just going to swap it all, right? I don't care what's been going on in the last six months. We're just going to show up and what could go wrong, right? Have you guys seen this movie? What could go wrong, right? Everything. This approach is just fraught with peril and the merge conflicts and there's all this new code that you don't even know about and so you haven't made the same changes to the code that's been happening in the meantime. We want to avoid this, right? Especially with all the developers. We're talking about minimizing disruption, minimizing risk and this is just a terrible way to go about it. So about a year ago, I came across this. This was actually in a tweet and Martin Fowler actually posted this and it came from a quite a long time ago. It came from a quote from Jessica Kerr talking on a podcast about refactoring and he took that quote and made this image out of it but it's like if I want to go 100 miles east but instead just traipsing through the woods, I'm going to backtrack 20 miles to get to the highway and once I'm on the highway I'm going to go 100 miles east at three times the speed that I could before. Sometimes you just need to say wait, I need to check the map and I'll find the quickest route and this is how she was explaining something called preparatory refactoring. So sometimes you just need to take a step back and figure out the direct route as the crow flies, the most direct route is maybe not always the fastest or the safest or whatever your goals are in getting there. So sometimes you just need to take a step back before you actually move forward. And so this kind of like struck a chord with me, like maybe this is kind of the way we need to do this refactoring. So just digging deeper into the discipline of refactoring, like you hear the term refactoring thrown around a lot as in like oh I'm going to go refactor this code when really what we mean is like oh we're going to go change everything about this code including the features and what it is. But Martin Vowler's written a lot about refactoring and this is kind of his like the definition of refactoring as it exists on his webpage is that it's a discipline technique for restructuring code and at its heart it's a series of small behavior preserving transformations. That last line is like really critical to what refactoring is, is that it preserves all the existing behavior. We're not adding features, we're not changing features ever so slightly. Everything, all of the existing features remain exactly the same but we're changing something about the underlying nature of the code. You could change from one, it could be as drastic as changing from one programming language to another as long as you kept all the existing features and existing behaviors exactly the same you could call that refactoring. And so like this again really got to the heart of what we're trying to do. We want to keep all of our code, all of our behaviors, everything that our customers and our business users and our retail associates depend on, we want to keep all of that exactly the same but we want to be able to move to running our application on Python 3 rather than just staying with Python 2. Kent Beck said it like this, for each desired change, first make the change easy and then make the easy change. So if you think about what the actual change is here, it's that we want to use Python 3. So how do we make that change easy? And then once we've made that change easy, then it's just a matter of changing which interpreter you use, right? So first thing we're going to do, and this is the plan going forward, is we're going to refactor our code base for Python 3 compatibility. We're now not talking about migrating our code base from one thing to another, we're talking about refactoring it for compatibility. And then once it's fully compatible with Python 2 and Python 3, like then we just like change our deployment scripts and run it on Python 3 instead. But back to this, Kent Beck says right here, this may be hard. Making the change easy is the hard part. So we're going to focus a lot on like what do we do to get to the point where we can be fully compatible with Python 3. So the first step is focusing on preserving behavior. How do we make sure we haven't changed anything in the behavior of our application when we go from Python 2 to 3? What I'm going to talk about right now assumes you have some sort of method for testing out your existing behaviors. You probably have a test suite of some sort. And we're going to use a tool called TOX that's going to allow us to test these things across environments. So TOX is an automation framework that kind of runs a step or like a level more abstract than something like PyTest or NoseTest or even UnitTest. And this will run your automation commands. But the way it does is it runs it in an isolated virtual environment for any number of different environments. It manages the setup of these so that you can run them on the same machine. It's kind of like having a whole like CI testing tool just right in your terminal. So for every command that you set up to run in TOX, it will first create a virtual environment. It will install the dependencies that you specify either through like your setup py if you're running like a proper package or you can give it like a requirements TXT if you're doing something like a Django app. It's going to build and install your application in whatever way you specify for it to install it into that virtual environment. And then it runs whatever command you've given it. And so this becomes incredibly powerful in being able to test both in Python 2 and Python 3. So here's a basic setup. In this setup we've told TOX that we're going to run two environments. We're going to run Py27 and Py35. And it knows by convention that if you create an environment named Py27 that it's going to look for a Python interpreter on that machine called Python 2.7. The same it's going to look for a Python 3.5. And then this test in stanza is kind of the default for each environment. So we're saying for each of the two environments that we've listed, create an environment with the respective interpreter, install py test. That's the dependencies that we see right here. And then after you've installed py test and installed the application, by default it will look for a setup.py to install. After you've done that, then run py.test in each of those environments. If you run something without a setuppy, so something like a Django application, you can tell it to skip, like to skip the regular setuppy method of doing things. Instead, run depths minus r with your dev requirements so it installs from a requirements file. The skip sdist is the part that tells it don't look for a setup.py file. And then again at the bottom we can give it any command we want. So the commands we're using here is like Python manage py test. Every time you invoke this we'll run each of these environments and give you the test output from each one. So if you have a healthy project, your Python 2 version should be passing. And if your Python 3 version passes, hey, you're done. Call it quits. Walk out. So this is how we're going to make sure that we maintain the existing behaviors and make sure that we're doing refactoring and not, you know, some other software engineering practice. The next thing we have to look at is dependencies. And this is, for a long time, has been the hardest part about actually making this happen, right? Just by show of hands, does anybody know if, you know, the project that you work on on a databases is, if all the dependencies are compatible with Python 3? Does anybody know how to check that or no for sure? No? Yeah. Yeah. Yeah. So there's a couple of them. There's, I like this one, Python 3 wall of superpowers. I started as a Python 3 wall of shame. They, in order to be a little more, a little less antagonistic, I guess, they changed it to the Python 3 wall of superpowers. So this uses Py, this uses the metadata from PyPI to figure out which of the top 200 Python packages are compatible with Python 3. We're at a very high percentage right now. We're at about, this is an old screenshot. This is 174 of the top 200. Currently, we're at 184 of 200. So just a couple of stragglers. This is another very similar thing. Python 3 readiness.org, 319 of 360. So same thing, just the top 360 instead of 200. These are great for the ecosystem as a whole, like Python, how ready are we as a community? But this tells me nothing about my particular project. This one's even better. It's called, can I use Python 3? You paste in your requirements, TXT, and it tells you exactly which of your requirements are not compatible. For any of them that are not compatible, it will even go one step deeper and say, does this dependency itself have any dependencies that block it from being upgraded? So if it doesn't, maybe that's a good place for you to jump in and go help out and get it upgraded. So now we've checked all of our dependencies. This is like, certainly you're going to come against some that are incompatible, so what do you do about it? I don't know, maybe just ignore it. Sometimes things aren't listed as being compatible with Python 3, even though they are. That's becoming less and less common, but the flag for whether something's compatible is just like a piece of metadata on PyPI. We can remove it. That's always one of my favorite things to do, is to remove dependencies. Maybe it's something you don't really need. Maybe it's a library that you're using maybe one function out of. Maybe just copy that function into your project. Maybe replace it. Maybe there's a thing that does something very similar, but is more modern and is compatible. Or maybe be a hero, go fix it, help the entire community. So that's dependencies. Once you get your dependencies all worked out, the next step to actually making your own code compatible with Python 3 is something that's part of the standard Python interpreter. So Dunder Future is a library, or it's an interpreter feature, but it's also a module that can be imported. So with this type of statement, we can import things that the core Python has decided will be we're going to backport future implementations into the existing code base. And so this is a way that we can start writing more compatible code and start using features that were introduced in later pythons in our existing code. So if we want to use the new division operator, the way it works in Python 3 instead of Python 2, we just put this at the top of our module from future import division. There's one for absolute imports. So imports and relative imports are kind of a mess in Python 2. I highly recommend you just throw this at the top of every one of your Python 2 modules. Print function instead of print statement. This is one of the first things that your test cases will yell about because it's absolutely changed in Python 3. And then also Unicode literals, a little less useful because you still got to keep track of the specifics of when to encode and decode or whatnot, but if you want to use it, it's there. The next library I'm going to talk about is a community library called 6. It's kind of the nice compatibility library for writing Python 2 and 3 code. So here's a couple examples. So if you're going to iterate over the items in a dictionary in Python 2, you do dictionary.iter items. This gives you an iterator instead of a list of things back. To do the same thing in Python 3, they've decided iterators is the way to go. It should be the default rather than actually getting a list back. So Python 3 looks like this, but instead of doing a bunch of checking, the 6th library gives us a way to just write it one way that works in both of them. And so you can change that and just say 6.iter items, and it's going to work in both Python 2 and 3. Similar with metaclasses, Python 2 and Python 3 have very different ways of doing metaclasses, and so 6 gives us one single way to do it that's compatible with both. It also, in reorganization of some of the standard lib modules, so in Python 2, HTML parser was part of this HTML parser module, and Python 3, it's been moved to HTML.parse. And so 6 gives us one way to import it that works with both. There's a number of modules that it does this for, and it's really, really helpful. So that's 6. You can start using 6 immediately. It's one tiny, tiny little package. It's all in one file. You can copy it into your code. You don't even have to make it part of your requirements, and it's built to be imported that way. So this gets us much more compatible. What's the next thing we want to do? The Python 2 binaries come with a tool called 2-3. This sounds fantastic, right? It's like, why didn't I lead with this? This is exactly what we want, but this is that Indiana Jones swap that we talked about, right? We've run this on our giant code base a number of times, and it's pretty terrifying. But 2-3 does some interesting things, and it'll change the code right in the files, like right in your source code for you. And it'll take something like this with our iteritems example again, and after you run it, it'll give you a diff of what it would change. You can have it change it immediately if you want. And it, so it'll show you, like, it changed this iteritems to .items down here. And so that's great for Python 3, but again, we're trying to write compatible code. What we really want is something that's like, you know, how can we go from Python 2 to, like, 6, right? 2 to 6 is really what we would want. And so what we started down the road of doing was, like, using 2-3 to see the changes that we needed to do, and then we would swap in the 6-compatible thing there. But this became tedious, like I said, like 850,000, or 850 files, you know, 175,000 lines of code. This is not something that we could do a lot of. So we really wish there was kind of like a 2 to 6 package. And there it is. It's called Modernize. So Python Modernize is a library. It's, it uses lib 2 to 3, which is the same thing that the 2 to 3 executable uses, but it adds its own rules that says, hey, when you find this thing, swap it out with 6 instead. So when we run it, when we run Python Modernize on our example, we get exactly what we wanted before. It even does some additional things like throwing in the appropriate future imports. So it's fantastic. If you want to do, run Python Modernize on your whole code base, you can do that. But again, I would suggest you do it in little pieces, iterate over it, refactor as much as possible, continue to run your code base, or your TOX test on your code base. So just in conclusion, we want to practice preparatory refactoring, not the Indiana Jones swap, to preserve current behavior. TOX is a great tool to do this. It's not the only way to do it, but it's by far the easiest one I found to do it. Look at your dependencies for compatibility, future and 6, help us write forward compatible code. And then if you want to use Python Modernize, it's an easy way to get to that level of compatibility. That's it. Thank you. I'm at ZRoger on Twitter and GitHub, and my slides will be here at Bitly Python 3 slowly. And thank you. Any questions? The shorter answer is we haven't. This kind of falls into kind of like tech debt and just like general refactoring for us. And we give our project teams and our project managers... What's that? Oh, yeah. He's showing the Python 27 countdown clock. Yeah. But we just... We give our teams a lot of freedom to choose the work that they're going to do and balance these kind of like tech issues with kind of like business features. Yeah. Yeah. I have no idea. This is, like as we saw from those dates, like Python 3.0 was 2008. It was, you know, well, well before I was involved with Python. I mean, this is the sole reason the 6th library exists. And so it's, you know, it does everything it can and to remain compatible across versions. And like I said, it's a really tiny module and it basically just provides a framework for saying like if Python 2.0 do this, else do this. And a lot of it is just it uses Python's dynamic nature of being able to rename and remap important names and function names to different names. And so it just gives you that like nice compatibility layer. Yeah. I wish I could talk more as to why, but I have not been involved in those conversations ever. Yeah. Yeah. And then as, you know, if you're in a place where once you migrate to Python 3.0, you can forget about Python 2.0. Say this is just like an internal application, not like a library that you're supporting for, you know, the entire community. Then you can go through and replace all your 6th code with the Python 3.0 variant of it and stop using 6.0, which would be like the final transition. If it's a project that you truly do want to make Python 3.0 only. Is there a question over here? This is the worst, right? I don't. You either try to write code for the parts that, or write tests for the interfaces, the parts that must remain the same. But there have been times where I've had those same types of, like, legacy issues. And one of the best practices I've found is to write kind of like a, almost like a really thin interface wrapper over the top of it, and really use it as an external system. So that, like, your actual code that you're responsible for doesn't directly, like, import it or use the functions directly, but instead interacts with it as, like, a web service or something, so that you can progress your code without that as a, like, hard dependency. I don't know if that's feasible in your case, but that's something that I've had success with in the past. Yeah, that's much more difficult. Now, I'm sorry, I don't have any suggestions. Anyone else? We actually do, we use that as part of our CI linting, is we'll run Pi 2 to 3 just to see if, just to see what comes up. We don't actually change the source, but the default functionality for that is just for it to, like, spit out the diff. And so, like, if you want to just, like, throw that into, you could, like, run that as part of your talks suite. You could add a 2 to 3, like, a stats. I don't know if that is built into it or not, but the library that it runs on, it makes it really easy to implement your own, like, plugins for it, which is, that's all Python modernizes is a set of plugins. And you say, like, when you come across this thing, react in this way. And so you could, if it doesn't exist already, I'm sure you could build a pretty trivial reporting tool on it. Anyone else? All right, thank you.