 Okay, this seems to work. Hello and welcome to the first day of regular conference talks. And now that we have split in all these different tracks, you should make sure that you check the schedule and choose the one that you want to watch later in the day. Here in the Optiver room, I'm really happy to welcome an experienced speaker who has given other talks at previous EuroPython, and I had the pleasure to introduce him myself during one of the real live ones. Let's welcome Sebastian Wytowski. Thank you very much, Martin. I'm glad to be back online at EuroPython. Yeah, online. It's more fun to meet in person, but I think this will have to do. You're calling in from Poland? Yes, that's right. Okay, and from my home office? Yes, from the home office. For those who don't know Sebastian, he's a Python trainer, and you should check out his own page, which you can find a link to in the EuroPython schedule. What are you going to talk to us about today? So I'm going to talk about pretty basic but fundamental topic, so how to manage Python versions and how to manage dependencies of your projects. Okay, so are you ready to go? Yes. I'll be collecting questions. If anybody has any questions, please type them in the matrix, and if we have time, we'll do a Q&A. But now let's give the stage to Sebastian. Thank you. All right. Hi. My name is Sebastian Wytowski, and I'm a Python consultant, freelancer, and trainer. As part of my job, I join different teams and work on different projects. And as I work on those different projects, I notice that they often suffer from one of two problems. They are either over-engineered or stuck in the 90s. The over-engineered ones involves jumping on every shiny new toy or tool that you find on the first page of Hacker News. And then everyone gets excited because it's exciting to work with something new. What is not fun is when this tool gets deprecated. Maybe its creator got bored and they stopped working on it. So now you have to either fork it and maintain it yourself, or you have to replace it. On the other hand, we have projects that are still stuck in the 90s. Some people think that because it was fine to manage everything using bar scripts back then, it's fine to keep using bar scripts today. It's fun if you're a hardcore Linux user and you love writing bar scripts, but to be honest, when was the last time you actually wrote one? Okay, writing them from scratch is not that bad. Maintaining a bar script that someone else wrote and that person is no longer at the company is much worse. And I don't mean that bar scripts are bad. They are actually pretty good for simple things. It's just you should use tools that are more suitable for the task at hand. So neither of this situation is good. You should update your tools, but at the same time you should choose those tools that are proven and you know that they will be around for the next few years. So that's why I've decided to make this presentation. I want to show you some useful Python tools that will make your life easier, tools that I recommend to my colleagues and to my clients. We all talk about things that a lot of beginner programmers struggle with at some point. So first, I will show you how to install different Python versions on your computer using a tool called PyEnf. Next, I will talk about Python dependencies. I will explain you what are virtual environments, how they work and why we need them if you're working on multiple Python projects. I will also compare the built-in VEnf module with some external tools like virtual and wrapper. And finally, we will talk about tool called PIPX. This is a great tool if you want to install some package globally, but at the same time you want to isolate it from other packages on your computer. And before we move on, I really want to highlight one thing. Those are all just tools. And there are a lot of other tools out there that do similar things. If you're using something else, that's great. My goal here is not to convince you that those are the best tools. No, I just want to show you how to solve some basic problems when you're configuring your development environment. If, for example, you are already using Konda and you're happy with it, then the first two bullet points on this list are already solved for you. And that's perfectly fine. But if you don't know what to use, let me show you some recommendations. So, first, let's talk about installing Python on your computer. Depending on your operating system, your computer might already come with some version of Python. If you're using macOS, then it comes with Python 2.7. If you're using Linux, then the Python version depends on the distribution that you're using. And in Windows 10, you don't have any Python installed. Let's say that your operating system comes with some Python version. This Python version already installed on your computer is often called a system Python. And no matter what system version of Python you have, I strongly suggest that you don't use it. In many cases, as we saw, for example, with macOS, it's terribly outdated. Python 2.7 is no longer supported by the core developers since a long time, so hopefully you're also not using it. You might be tempted to upgrade the system Python to Python 3, but you probably have some programs on your computer that require Python 2.7. You would be surprised how many programs or even parts of the operating system still require Python 2.7, at least on macOS. If you upgrade system Python, those programs will stop working. So, if you change the Python version that comes pre-installed on your computer, you risk that your computer will stop working. And that's not fun. I have done this in the past when I didn't know much about programming or operating systems and I had to reinstall the whole operating system from the scratch. So, my advice here is to leave the system Python alone and pretend that it doesn't exist. No matter what operating system you have, you will need to install Python. And there are many different ways how you can do this. You can go to the python.org website and download the installer for your operating system. You can use the package manager like homebrew on mac or apt-get on linux. And you can even compile Python from the source files. However, my favorite way of installing Python that I have been using for already a few years is to use a tool called pyenv. Pyenv is a tool for managing Python versions. You can use it to easily install new Python version on your computer and to switch between those versions. It might not be a big deal if you're working with only one Python version all the time, but, for example, if you're working with multiple projects, then if they require different Python versions, this tool is extremely useful. For example, when I do some Python tutorials or courses, I try to use the latest version of Python available, but when I work with clients, usually they don't use the latest Python version, so I need to have the old versions available as well. I've been using Pyenv for years and it works very well. If you work with some other programming languages, you might recognize similar tools that work in exactly the same way. So, for example, in Ruby there is rbnv, in Node.js there is Node.nv, there is also go.nv for go and so on. While pyenv will work for macOS and Linux, if you're using Windows, check out the pyenv win. It's a port of pyenv to Windows. It might not have all the features that the standard pyenv has, but it has all the essential ones that I will talk about. But again, if you're a Windows user and you're happy with conda, then there is really no point in switching from conda to pyenv. One of the things that conda can do is to create virtual environments with different Python versions, which covers two out of three tools from the stock. So, you can install pyenv with homebrew, you can clone it with Git and then manually set up a few things, or you can use pyenv installer tool. And while this last option requires you to download a bar script from the internet and run it in your terminal, which is a big security no-no, it's actually very convenient. It will install not only pyenv, but also some additional plugins. For example, pyenv doctor that you can use to verify that your pyenv installation is working fine. Once you have pyenv installed, you can use it to install new Python versions. To see a list of all the versions that you can install, just run pyenv install dash dash list. And as you can see, there's a huge list of different versions. At the beginning, we have the standard C Python versions that start with a number. And then we have anaconda, mini-conda, or pypy, and so on. So, those are all the different types of Python that you can install with pyenv. If you ever wanted to install, for example, pypy, now you can very easily do this. Once you select a version from this list, type pyenv install in the version number and go make yourself a coffee. I mean it. The installation usually takes a few minutes. There are some additional libraries like OpenSSL that you have to pre-install to make the installation process faster. So, if you don't have them, pyenv will try to download them each time you install a new Python version. So, I suggest you take a look at the GitHub repo to see what prerequisites are recommended. So, once this is done, you can run pyenv versions to see if this new version is installed correctly. We can see it on the list, so that means we are all set. Now we have to tell pyenv to use that version. And for that, we have to choose one of three different levels at which we want to change Python. We have global, local, and shell. So, what do they mean? Well, nine out of ten times you will probably want to change the global Python version. And to do that, you just have to run pyenv global 3.9.0. And from now on, you are using Python 3.9. Then we have pyenv local. So, imagine that most of the time you're working with Python 3.9, but you have this one project on your computer that still requires Python 3.7, and for some reason it won't work with any other version. So, instead of changing the global Python version to 3.7 each time you want to work on this project and then changing it back to 3.9 when you stop working on this project, you can set up a local Python version. This is where the pyenv local command comes in. You can run pyenv local 3.7 and it will set the version for the current folder and all its subfolders. So, whenever you go inside this folder or any of the subfolders of this folder, pyenv will automatically detect that you have a local version and it will change the Python version. And when you go out of this folder, it will change the Python version back. That's also very handy when you work with multiple projects and each of them requires a different Python version. So, instead of changing the global Python version back and forth, you just have to run pyenv local in each of those folders. And when you go inside of any of them, pyenv will automatically change the Python version. And finally, we have pyenv shell that changes the Python version for the current shell session. You might use it in a situation where you want to temporarily change which Python version you are using. For example, maybe you want to run some Python 2 code. So, you just run pyenv shell 2.7 and now you can use Python 2. But once you open a new session in a terminal, you are back to using the previous version of Python. So, that's how you can easily manage Python versions on your computer. Now, let's talk about dependencies of your project. Let's start with peep. Peep is a package manager for Python. Whenever you want to install a new package, all you have to do is to run peep install and package name. However, peep has one big problem. Whenever you ask it to install a specific version of some package, it will uninstall the previous versions from your computer and install the one that you asked for. Let me show you an example of what I mean. Imagine you are a web developer and you want to build a Django website. You start by installing the latest Django version by running peep install Django and this installs Django 3. Everything works fine. You build an awesome website. So, awesome that a client or a boss or a colleague comes to you and says, hey, you're making a really cool Django website. So, maybe you can give me a hand with my Django website. As it often happens with clients, they are not using the latest Django. They are still running, let's say, Django 2.2. So, you install that specific version by running peep install Django equals equals 2.2. And peep does what you ask for and after a few seconds you can start working on your client's website. So far, so good. But later this day, you discover that there is a bug on your personal website, the one written in Django 3. So, you quickly fix this bug, but when you try to test it, you get an error message saying that Django 3 is not installed. I mean, we just installed it yesterday. Where did it go? Well, when we told peep to install Django 2.2, peep first checked if we already have Django installed and we did. It's just it wasn't the 2.2 version. So, peep uninstalled the version that was on our computer and installed the correct one. So, we just run into a problem with dependencies management. We have this problem because peep installs all Python packages inside the side packages folder and puts each package into a separate folder named after that package. So, Django 3 is placed inside side packages slash Django, but when you want to install Django 2, it will also be placed inside the side packages slash Django. So, peep has to first remove what's inside the Django folder and then install a different version of Django. If you only work with one Python project on your computer, then you're probably not affected by this problem. But sooner or later, you will need to install different versions of some packages and you are going to run into issues with peep uninstalling some previous dependencies. The problem with peep is that because we had the problem with peep because it installs packages in the same folder. So, how about we tell peep to temporarily install packages into a different folder and then we tell our Python interpreter to use packages from that other folder? Well, that's exactly what virtual environments does. Virtual environment is a folder that contains a Python installation and any additional packages that you install. When you activate a virtual environment, two things happen. First, you tell peep to install any new package in that new folder and then you tell Python interpreter to use packages from that folder. Now, if we try to install some Python packages, peep won't uninstall the previous versions from your computer because it no longer knows about these global site packages. It's using the site packages folder of this specific virtual environment. So, how does it work in practice? Well, first we need to create a virtual environment. Python has a built-in module called VNV that you can use to manage virtual environments and have to install anything. We create a new virtual environment with Python minus MVNV and then the name of the virtual environment. In this case, I'm creating a folder called .vnv inside the current directory. This VNV or .vnv is a common convention for naming folders with your virtual environment. So, whenever you see a project that has a folder like that, then you know that inside you have the files for the virtual environment and you can have to check what's there. But it also has another benefit. Some code editors, like PyCharm or VS Code, will automatically detect this folder in your project and recognize it as a virtual environment. So, when you open a project like that in PyCharm, it will automatically start using this virtual environment. Okay, so we have created a virtual environment. Now, we have to activate it. Inside our VN folder, we have the virtual environment directory. And there we have the activate script. Since this is a bar script, we have to run it with the source command. Of course, if you're using a different shell, there are other files that you can use. If, for example, you're using Windows, there is also activate.bat. Once you activate it, you should see a difference in your terminals prompt. It should now display the name of the virtual environments in parentheses. If you see something like this, it means that a given virtual environment is active. And now you do everything as you would normally do when building a Python project. We can install a package with PEEP and it will be installed inside this virtual environment. Or we can run a Python script and it will have access to Python packages from this virtual environment. And if you want to stop using a virtual environment, you just have to run the activate command. When you call the activates, you will see all the changes that activate did. So you will be back to using the global Python version and global PEEP packages. A typical workflow with virtual environments involves creating one virtual environment for each of your projects. So if you're working on two Django projects, as in our example, you first create one virtual environment, you activate it, you install Django 3 and you work in it. And then you create a different virtual environment, you install Django 2.2 and you work on that other project. And VNF module is perfectly fine for managing virtual environments, but I want to show you another tool that I have been using for a very long time that is called virtual and wrapper. And it comes with a lot of cool features that makes working with virtual environments even easier. You can install it with PEEP and it works on Linux and macOS. For Windows users, again, you have to use a slightly different tool called virtual and wrapper that works pretty much the same, but as I said before, if you're happy with KONDA, then you can stick with KONDA for managing virtual environments. So one main difference between VNF and virtual and wrapper is that virtual and wrapper stores all your virtual environments inside your home folder in a .virtualNFS folder. And it also provides you with some convenient methods for managing virtual environments. So you can create a new virtual environment using a command mkvirtualNFS. So we can just type mkvirtualNFS.jango2up and this will create a virtual environment inside this .virtualNFS folder in the home directory. And if you want to activate a virtual environment, you just have to type workon.jango2up You don't have to type the whole path to this activation script. VirtualNFS will figure out where is the activation script for specific virtual environments when you just give it the name of the environment. And if you forgot the name of the environment, you can just run lsvirtualNFS and this will print you the list of all the available virtual environments. With VNF, it's impossible to get the list of all the virtual environments because, well, VNF doesn't keep a track. With virtualNVrapper, since all of them live in the same folder, you can easily get that list. And finally, to remove a virtual environment, you just have to run rmvirtualNFS and the name of the environment. So you're probably wondering which one is better. Well, both tools are great and it really boils down to how you like to use your virtual environment. I like virtualNVrapper mostly because I don't want to type this whole source vn slash bin slash activate and then, of course, in half of the time I mean like some subfolder of my project and I got an error saying like vnfolder doesn't exist so I have to figure out what's the path to the activation script. Now, I just want to type work on name of the virtual environment and be all set. I also like to create virtual environments that are not connected to any specific folder any specific project. For example, I have one virtual environment for a bunch of data science libraries that I use each time to start Jupyter with Pandas or NumPy. Also, virtualNVrapper has some interesting functions. For example, you can create a temporary virtual environment that will be deleted when you deactivate it which is quite handy when you want to, for example, test some Python package. But the vnf module also have some benefits. For example, if you store the virtual environment in the same folder as your project, it will be detected by your code editor. And when you delete this project the virtual environment will be deleted as well, so since all the files are stored together. And vnf is actually built in module so you don't have to install anything. All right, so we covered how to manage Python packages in your project. How about global packages on your computer? Some tools are much more convenient to use when you install them globally instead of installing them for each of your projects. For example, we have the code formatter black, the code linterflake8, or even the virtual and wrapper tool that we just saw. We want to use those Python packages outside well, across of all the projects or even outside of the project. For example, we would use virtual and wrapper to actually first create a virtual environment before we start working on a specific project. But then again, if two different tools require different versions of the same dependency, we have a problem. You install one tool and everything works great. And then you install the second tool, it changes some dependencies of this first tool and then the first tool stops working. We could install each of those tools into a separate virtual environment, but that's a lot of hassle to use it like that. Each time you would have to activate it, run some commands and then remember to deactivate it. I mean, you can do this but it's a lot of typing and generally it's a waste of time. We can easily solve this problem using a tool called PIPX. PIPX installs Python packages into separate virtual environments, but at the same time those packages act like if they are installed globally. So you don't have to manually activate any virtual environment to use them. So how does it work in practice? Let's say I want to install Black to format some Python code. I run PIPX install and after a few seconds I get a success message saying that Black has been installed and now I can use those three commands Black, BlackPrimer and BlackD. Now I can use Black as if it was installed with PIP. It doesn't matter if I'm inside a virtual environment or not, I will always use this global Black package. I don't have to activate anything. I just run Black and it works. If I want to install a different version of Black inside a virtual environment I can always do this. And this Black from inside a virtual environment will take precedence over the global Black that as long as I'm inside a virtual environment. What else we can do with PIP? We can list with PIPX. We can list all the packages to see what commands they offer with PIPX list. We can uninstall the package and this will also clean up the virtual environments for it. We have one command by running PIPX upgrade all and we have one very useful command called Inject. It will install a PIP package inside a virtual environment of another package. You might use it, for example, to install some pipest plugins. So let's say you want to add a plugin called pipest.cof to generate the coverage report for your code. We can't just run PIPX install pipest.cof It will install pipest.cof in a separate virtual environment and pipest that is like in a yet another environment won't know about this plugin. So we have to call pipx.inject and this will install pipest.cof inside the same environment where the pipest is installed. And that brings us to the end of my presentation. So let's quickly summarize what we saw. First, use pyenf to install new Python versions. It's really easy to install new Python versions and switch between them. Then use virtual environments when you work on different projects and you want to isolate their dependencies. Always make sure to create a separate virtual environment for each of your projects. Python comes with a built-in vnf module but if you prefer a tool with some additional functionality, virtual and wrapper is a really good choice. And finally, if you want to install some tools globally, you can use pip outside of a virtual environment but you risk that if there is some version conflict, your packages will break. So a much better idea is to use a tool called pipx that will install each package into a separate environment but for you there won't be any difference in how you use them. And that's it. Thank you. Okay, thanks very much, Sebastian. We've had a lot of activity in the conference one up to the hallway. So I'm just going to ask you, we have two more minutes, two of the questions we got. So the first one we had is how would you compare pip virtual events against Kondamamba, especially considering that Kondamamba have a lot of extra features compared to the first ones. Yeah, so I don't know about Mamba but I'm assuming it's like alternative to Kondamamba. So that's a very good question and half of it is answering the question. Kondamamba has a lot of features. So if you're looking for one tool to do all of it, it's a really good choice. I like PyEmp because for me it's more of this Linux style philosophy of a tool that does one thing and does it good. So PyEmp basically just modifies your path to use some different folder where you store different Python versions. So it's very easy for me to understand how it works. It's very easy to uninstall it if I don't like it and so on. Okay. And another question is that's something I ran into myself as well. How do you run a virtual environment without activating it? So I'm assuming the question is about how pipX does it. I think it's there was a longer discussion about this. Just like if you want to run something with this and you don't want to activate it and there's a longer discussion about this, basically you can just go into the directory and start the Python that is in there to avoid the source command. Yeah, basically you can do what the source command does manually by finding the code. By starting Python, yes. And if you for example have to automate something then this is an option instead of writing the source command. Yeah. I remember I had some issues with that because I'm using fish shell and then Python got updated on my computer and then some hard coded paths to Python environments broke. You can do this, I don't recommend it. Okay. Thank you very much for this and if you have time please go over to the matrix where there might be some other questions that you can answer. But here we have to continue with the next talk. So after a brief interruption we will probably start the next one. Thank you so much. Enjoy the rest of the conference. Bye.