 Thanks everyone for coming out to my talk for your final presentation at Python Europe or Python. So this is sight unseen, but hopefully by the end of it, it won't be so hidden. Python is a very customizable language. I think we all know this. It allows us to do all kinds of things. It gives us the power to monkey patch, change the interpreter as it's running. And there's a lot of customization that can be done. Sometimes the customization is hidden from us, like when we use decorators or context managers, what actually happens isn't immediately obvious from reading the screen. And it seems almost magical, but it's not really magical. It's just that the complexity has been shifted somewhere else. And the same thing happens with the interpreter startup. We also have power to change how Python starts up. But a lot of it's done out of sight. And so it's not really magical. It's just that it's happening somewhere where you don't see it. But if you know where to look, it's not so magical and hopefully not so complex. The customization of Python as it starts up is done by module called sight. It's part of the standard library. And when you import sight, it does these things. It extends your module search path that is to stop path. Enable some interactive features for when you're going to use the interactive Python shell. And also executes further user customizations, basically letting you run arbitrary code to further customize Python. So if you import site, you get all these features. And I'm sure all of you have benefited from happening at some point, probably almost every time you're on Python. But I don't know how many people out there have actually run the statement import site. This is because the interpreter does it for you when it starts up before you get to your main script. It's already decided to go ahead and import site for you because really it's what you want to happen. Now Python still gives us the power. So if we really don't want it, we can turn it off with an option on the command line. Or if we decide later we really do want it turned on, we can still be explicit. We don't have to let Python do it behind the scenes. But site is not just a module you can import. It's also one you can't execute. So if you run it as an executable module on the command line, it'll print out some diagnostics that can really help you when you're debugging tricky, especially import issues. If we run site on the command line, it prints out something pretty much just like this. It shows us our entire sys.path as it will appear just as we enter our main script. And also gives us some detailed information about user site, which we'll get back to in a little bit. If we tell Python again to execute site, but we pass the dash s option, which means don't import site for me magically. It still works. It's kind of confusing sites being executed, but we told Python not to import site. But it turns out it's actually smart enough to work. And it just works so we can still debug path problems, even when we don't want to have the implicit import site. And just to prove to you that you don't have to use magic. If we tell Python not to import site for us, but we do it ourselves, we get the same result as the first time. So the first job of site is to extend the module search path. I think everyone probably knows how to extend search path using the environment variable Python path. Probably everyone out there has set Python path in anger at some point, because Python just couldn't find the thing that you knew where it was. I know I do this often, although I try not to. Putting Python path into the sys.path list is not the job of site, but every other addition to sys.path pretty much comes from site. So we have other ways of extending the search path. Specifically the site will add directories called site packages. You might know this name if you've ever been curious, where pit puts things that you actually install. They all go into a directory called site packages that sits alongside the rest of the standard library. So wherever you have your OS and your path plan. There's also a directory called site packages, but it doesn't have a dunder init file in it. So it's not importable, but somehow everything inside of it can still be imported. This is because site goes and finds it and adds it to your sys.path. So otherwise you couldn't import any of those third-party modules. Next path also, or site, excuse me, also adds user site packages, which is a different kind of site packages. It's still site packages, but it's located under your user home directory instead of relative to the install location of Python. And this can be very useful because you always have write permissions under your home directory, and you might not where Python is installed in your system. So if you've ever tried to pip install something and got hit in the face with a big error that you eventually figured out was because you don't have write permissions to wherever Python is on your system. The answer to solving this issue is not to pseudo pip install your package. It just use the pip flag dash dash user, which will tell pip you want to install in the other site packages, the one that you have write permissions to. You might be using this and not even know it because for the last couple of years pip has released versions that will automatically fall back to the user site. So it's not just permission issues, but sites not done yet. It also adds virtual and so there's also a site packages in your virtual ms. I love virtual ms. I hope everyone out there uses them. But you spend all that time and effort isolating your dependencies from all the other environments and your base Python. But if that site packages doesn't make it on your sys dot path, that's kind of it's all for not. Even if you run Python from an activated state, if you have an imported site, you don't get to use any of those packages that are in the virtual environment so you kind of still really need site. And finally, path will read and execute path files. What are path files. They are any file that site found as it was adding these other site packages that ends in a dot pth. So for every one of these files that finds it'll read it and these files can contain two types of lines. They can contain a line that is a path anywhere local on your system, and it will be added also to sys dot path so it can be anywhere. Or it can contain import statements. In which case, site will execute those import statements. And so this provides a mechanism for third party modules that are in site packages to participate in the customization of the Python startup that sites doing. Because site is also extending your path and it's also being imported for you kind of implicitly path is a pretty advanced feature and if you don't know that you need it you probably don't need it but what could we do with it. Well, it's for modules that really need to set themselves up before you get to your first line of code. So if you want code coverage you probably want code coverage from your first line to your last line. And that's pretty tricky to do if you have to first import something and do some boilerplate setup. So if you were perhaps creating a new coverage library. You could drop this path into your site packages, which would add your source code to the system path, letting you import it as sort of an edible install. And you could import the setup so that your code coverage tool could cover its own code. So here's that command that we ran earlier just added comments to show where each directories coming from. The main transistor path is the directory you launch Python from that's always first. Then comes Python path, if it said, it could contain multiple things. And then your standard library. So these things are all set even before we get to site. But the last four are all set by site. So about a half of our search paths are dynamically at least in this case, could be more or less. So I added a virtual and site packages, a sort of random path from a path file. My user site packages and the base system Python the one that created the virtual line is last on the list. So sites done adding or inflating your system path. And now it goes on to enable some interactive features. This is when you're going to enter the Python interactive shell. A lot of what makes it nice to work with is because of site. So it will import and configure the redline module, which is another standard library that comes with Python and it's really just a thin wrapper around the sea redline library, which is a line editing library. There are different languages that have interactive shells, including bash and it gives you things like tab completion in your Python Ripple command history, which lets you do things like up arrow or research your commands. When you close the Python interpreter and reopen it, your old commands are still there because they were saved in history file which is from redline. So you really know that you launched interpreter that didn't configure redline properly if you've ever gone and made a simple spelling mistake and tried to use your arrow keys to get back and fix it and instead just filled your buffer with garbage. Eventually figured out you need to clear a whole line and start over. And for multi line statements and Python this means you have to clear the whole thing and start from the first line again. It's really nice having a redline set up for you. The other thing that site will do for interactive is it'll create new built ins. So, I said at the beginning that we can monkey patch things in Python we can monkey patch built ins. And in fact, site does this monkey patches new built ins every time you launch Python. So as a couple, including the help function and the exit function that you use. You don't exist unless you have imported site somehow. So if you had a script and you use the global exit function. This would work Python would execute and exit cleanly. But if you happen around the same script again without the implicit important site, you get a name error. Because it's not actually defined in the built ins class it's put there by site. So for scripts just always use the start exit. But for the interactive shell it's actually really nice to not have to do an import of sys just to leave the interactive shell. So next is user customizations. These are hooks that site provides for other code to participate in the customization of the startup. So so far everything's been kind of prescriptive but here site is going to give us some opportunities to just run arbitrary Python code before we get to our main script. The first way we can do this is through a module called site customize. So after site has inflated the full sys.path it will then attempt an import of site customized so if this is a module that exists in one of those directories it will be loaded. And because it's loaded all its global statements will be executed. So whatever it does affects Python. It's designed to be for system administrators or whoever was the person who installed Python on that machine. So they can make it work better for the machine or for the users, whatever they need to do to administer the Python. So if there was a really curious system administrator that really wanted to know every error that was happening they could use this site customized to set up a login handler where they got emailed. Every time anyone logged in error above on any Python process probably be very chatty but they'd certainly know what was going on. The next opportunity to extend what site does is user customize. Which is essentially the same thing as an attempted import that's done after sys.path is fully inflated. But this is by the name designed for the user the person who's launching Python to add custom code. So this is really opportunity for you to add whatever you want to change Python to behave differently so it suits you better. And really the sky's the limit because it's Python code but what's one thing you could do well if you maybe like pretty tracebacks. We have some pretty great ones in 311 and better ones coming in 312 but maybe you're on an older version of Python, or maybe you really want colors in your tracebacks and we still don't have that in the core. If you had this as a user customize, then any Python you ran, if it ever crashed, will give you a pretty traceback, even though you didn't necessarily put this important every script and every library that you used. The final opportunity for us to insert some extra customization is the Python startup, which is an environment variable that can point to any file on your system. So unlike the first two that are attempted imports, this one can be anywhere, it doesn't have to end up on sys.path. If it exists, it is exact, not imported, but only if you're about to enter the interactive interpreter. So that's important in a couple reasons. One is that we can customize Python in a way that's maybe slow or not useful in every situation because site customized user customers are going to happen basically every time you use Python for any task. But Python startup is only when you're entering an interactive session. So we can take the hit of even a couple seconds maybe if it really helps us out. And the second is that it's exact. And so whatever happens in Python startup is going to share the same global namespace as the shell you're about to enter. So any name bindings you do and things like that will affect your interactive session. So we can, for instance, always automatically apply future updates in our interactive session. We can, if you want to, are always doing it where it announces every time we open the interactive Python shell. We can just import some stuff ahead of time and they'll be ready as variable names before those first three arrows even appear in our terminal. We can change tunables about Python, things that aren't necessarily available to the command line or that are, but maybe you don't control them entirely. So it's kind of the end of what site does, but it's a lot, right? We've changed where you look for packages. We've monkey patched built-ins and we've executed some arbitrary Python code. So if things are going wrong, there's kind of a lot of opportunity, a lot of places to that might be affecting them. So how do you track that down? Well, we can run Python in verbose mode. If we do, many of the standard libraries that run during the start of the interpreter will pronounce some diagnostic information, including site. I think it's very chatty actually, but it can reveal a lot about how the interpreter is running. And if I'm having a problem specifically with import paths, I also like to use the C Python option dash x import time. This prints the time to import each module, which is not really a concern to me. I've never actually had to track down a long import time before. But what it does is it also by printing out those times tells you the real order that everything was imported. So when you start up or when you're running your own scripts, you know, you might have an import OS statement 15 times might import path live 12 times and encodings twice. But only one of each of those is a real import and all the rest are cached. So if you want to know the order of things, we're actually loaded up disk. This will kind of record that for you. So it gets very chatty several hundred lines just to run the site as an executable, but if we put a little filter on standard error. This is everything that the site module tells us about what it's doing. So it tells us where it's looking what it's adding and the order in which it's looking for things. So maybe we found the path that was getting in our way that was giving us the wrong import or the missing import. We might want to reduce how big sys.path is. And there's a couple options. Again, Python path and the current working directory are not added by site, but we can still turn them off through other mechanisms. So if you just unset Python path, then it won't be added to sys.path. And Python dash capital P does not prepend the current working directory to your sys.path, which is a great way to remove those bugs that of a module shadowing built in because it happened to be in the same directory you launched Python from there was a JSON.py or something. This is a great option, but unfortunately it is only available in 3.11 and up. So the problem is not our import paths or imports, but customization. Some of this arbitrary Python is going wrong or causing bugs. We can turn off the Python startup so we don't get exact code by just unsetting that variable. If we use Python dash lower s, that will turn off specifically just the user customized. So we still get the site customized. We still get our sys.paths. But the user import is not done. So that custom code is not running. If you use Python dash capital s, we saw earlier that turns off all sites. So this will not monkey patch built-ins. This will not add any of the site packages. It will not execute any of the arbitrary code. So you can't turn off just only site customized, but you can turn off all site. And remember if you don't get any site packages, that means that you have essentially disabled your virtual amp. So you kind of can't pick and choose there. Maybe you don't trust the user who is executing the Python code or maybe the user is you and you trust yourself, but you have kind of really mopped up your environment that you are launching Python from. So if you use dash capital E, you will turn off, or Python will ignore all environment variables that are for it. So all your Python path, Python startup, but also things that have nothing to do with site. They are all ignored. So Python break point, Python hash seed. It doesn't consider anything. Just use defaults. And dash is a shortcut for ES, lowercase s. So this means it will ignore your environment variables and specifically ignore your user customized. So you still have a chance to run a site customized. You still trust the administrator who installed the Python. And the packages that they installed because you still add your site packages. But if we add all this up, and we just still really don't trust what Python is doing, we can run a whole safe version of Python, a safe mode in Python SIP. This turns off all of site. So none of that customization, no arbitrary code running. Ignores all your environment. So you only get defaults. Whatever the interpreter was built to consider the default. And you turn off the prepending of the current working directory to the beginning. So this is a pretty boring Python. Your sys.path is just directors that have built-ins. You have no other modules. You haven't customized at all. Your interpreter, your interactive interpreter, if you want into one, is just very plain. And doesn't have any charm to it. But you've also eliminated a lot of possibilities for bugs. So sometimes we need to be safe. Sometimes we need to debug. And this is a great way to do it. But I do encourage you to take these lessons and go and customize code. Or the Python interpreter, because it's really just a tool for us. And we should make it productive for ourselves. And we should be on that. Just make it so that it's a joy to use every day. I've managed to mass a fairly long Python startup file. And I've shared this gist, which I know is a long string of just numbers and letters. But if you go to my GitHub, it's currently the only public gist. So it should be fairly easy to find. Thank you for coming out. And enjoy the end of your Eurobyte. Yeah, thank you very much, Emma. We have, according to the schedule, time for questions if there's any in the room. So let's have a hand sign if there is a question. And I see one. So, Mike, could you please go to the mic? It's in the back. I have a question about if you used virtual amps, if you use condo environments, are there any differences in terms of location for all those things? Do you know anything about this? And I don't know. I don't launch condo environments. If condo is using site packages as its install location, then condo uses condo with site packages in the site in a different for each environment. It creates a site packages, someone nested down there, which is the site packages. There's no global site packages for condo. Right. But if it uses it, I think, I think I don't know condo that well, but I think if it's using a directory called site packages, it's relying on it being found by site. Because site packages are added differently than other directories to stop path. But if it's maintaining its own cache and just adding that cash to Python path or something like that, then it would be going around site. Okay, thanks a lot for the question. I see at the moment no further questions. So we're going to take the opportunity to thank you again for your talk, and hope to see you again next year at your place. So let's have another round of applause for Jeremiah.