 Thank you. So, yeah, today we're gonna learn about virtual environment and I have to make a confession This talk is marked as a beginner and which is true that I'm not expecting from you any kind of previous knowledge But we're gonna dive in quite deep into the Python interpreter and its implementation details. So It's gonna be a bit dense to make it a bit more sweet. I did brought a few nice pictures of My yosha terrier puppy. He is silky. He's just a five-month-old and This is basically he's just asked her he got home his first picture with a tennis ball in his mouth so similar to him will Get on to a course to actually discover how virtual environmental work underneath Okay So just a quick intro. Who am I? Yeah, I'm the maintainer of the virtual tool Which hopefully would make me quite competent to talk about this subject I'm a member of the pi Python packaging authority and I'm quite active in the open schools page and you can find my contributions on there this github handle Gabber burnout and I'm also maintain of the talks tool which heavily uses virtual amp This is one of the main reasons I became Maintain of the virtual amp tool a lot of talks tools talks box are kind of like virtual amp box So might as well fix it there and I'm also a software engineer at Bloomberg working. What's that data in just some pipeline? Inside the company. So what is the virtual environment? Luckily, I don't have to come up with the definition the pep 405 defines it very clearly It's an isolated Python environment that's had access to its own side package shared success to the host building standard library option isolated from the system side packages and this is the point Like whoa, what are all these kind of random definitions that you're throwing out and why what what do we want to do here? And in order to understand this better less step Let's take one step back and actually cover So how do we actually get hold of a Python module Python is this famous language in which you can do pretty much everything by Some package that has implemented and you can get hold of our some module But how do you actually get hold of that module and the answer is basically in this case CD? slide which buildically means if you want to build to use anything all you need to do is basically imported like that so The thing is in the way you get something is by importing it So how does the importing system works? Well, pretty much you just specify import you specify what you want to import and then in the next line You can already magically use it, but the question is what actually happens under the hood when you say that import that says, okay? And Another kind of like complication around these that not more not all modules are equal for example If I look at that says I can see that even if I print the representation. This is a built-in module if I look at the OS Module you can see that that's kind of like has a file inside that oped Python 3.7 blah blah blah dash OS that pie Similarly the .csv this is more interesting because it's actually you can see this implementation is a national file Which is kind of like a DLL for on the windows Then you have Python will be a different location or I can have like here pi 5.7, which is even a different location So as you can see just by the fact they're living different layplaces They are considered different type of modules like this one is a built-in module though S1 is a standard library module. This is the modules that come if you install it If you just Python built into the standard library and then we have also standard library model by dynamic module these are modules that are are Would be slow to actually implemented within Python. So what the Core developer team does they implement the actual underlying implementation inside C and then they just create a A slight front proxy to that that's why in this case you can see this is actually underscore csv I'm not csv because the csv is basically just a slight proxy the actual implementation the performance implementation is the underscore one And that's available under some actual binary code Okay, and then we have this import Python will and this is a global site package And it differs from the user site package in a sense that this is actually available on the system path While this other one is basically available under a user site path Okay, like you can see this user's burnout that dot local which is kind of like users home Okay, so the next section is okay now that we know that we import things How does the importer works and this is something that has evolved throughout the pythons life But this is how it works in 3.7. It was similar on Python 2 But the only difference is that there you could not modify this it's not as easily exposed So we can actually inspect it. But the idea is that Python doesn't know if something is importable or not It just basically goes with the principle that try first as for beginners later so it tries to import all the actual packages and It has a list of importers if knows and you just ask in order each importer. Hey, can you give me this? Sys if yes, it's gonna return that it's not gonna follow our inforter to the further imports The first one can can return it in the order. That's the one that Vincent actually provides that module for you So the we have the built-in importer This is the first one built-in importer imports Element which are literally compiled within the CPyton binary and or like the Python binary and in this case. It's actually a Such example is the sys itself like you're not gonna buy a sys.py anywhere It's literally available already inside the binary itself. Then we have frozen importer By default Something simply like everything can be put straight into your Python binary And the way you do it is Billy Q freeze it and the freezing means I take your Python code I compile it into the something machine code and I literally attach it into the Python binary and Because this is no longer lifts anywhere on the file system It's not so not on the no namespace of the built-in. It has this frozen importer Which couldn't know at which location inside that binary. I'm gonna be able to find that element or my final option At least in the default is the pet finder and the pet finder as its name kind of like in chat. It basically looks at the operating system File system and we'll try to find it somewhere there Okay, so let's investigate later. The pet finder is basically just give me somewhere on the file system and and It doesn't operate it has its own configuration and the main configuration for this is gonna be actually the information that you find in the sys.pet Okay, and the sys.pet is basically a list of locations where this importer will try to look and find if that package is there If in any of and this again, we're gonna do it in the order and the first one is gonna find it That's what's gonna be returned to you. Okay So we have you can have multiple elements But this is like a list of default that I get on my macbook For example, you can see this third one is basically a zip file like we can even import from zips itself This is the first place it's gonna look then it's gonna look in this sender library directory Then we have this lib-dial which basically where we store the c extensions for the sender library Or we have the user size package or global site package. Okay So now that we know all this we can actually interpret this definition So a virtual environment is just a isolated Python environment that had access to its own site package So it must have something else besides what is there Okay But the system one is basically has access while it has access to the sender library pass plus the user and the global site package the Virtual amp may or may not actually have access to the global one So like in this case, you can see it has the first three They are the sender library locations and the fourth element is basically its own site package Or if you configure it to also have access to the global site package It's gonna append to end of this list the user and the global site package Okay, so virtual environment is nothing else just a Python executable that whenever it starts up. It's magically gonna Make available its own site package folder and may or may not disable the global one. Okay quite simple So the question is why do you want to do this and the reason for why is because this will allow you to actually Install packages our system administrator right because we saw everything else other than user side package was available Only modify a ball by the system Python. So if you don't have admin, right, you might not might be installed it the You also have the option that you can create easily and reproduce test environment You don't have to reinstall your entire operating system just to test your package You can just create a virtual environment and it's gonna be exactly the same as you would have done that And it's also easier dependency management and the way this manifest is that in practice you would have multiple dependencies and if Two tools if you would install everything in the system a single global size Site package you may end up in the situation when two of the site packages have conflicting dependency requirements and this would mean that basically by just installing a new packaging to that environment you break another one Which was already there. Okay, and it is also just allows you if for as more like as a testing the ability to easily test against multiple dependencies easily without even have to Go too hard about it. Okay, so Let's see how this actually works. Okay, so in the next we're gonna try to look into I told you already that basically What gonna determine what is a module is gonna be this is dot pet where something gonna be loaded is the sees dot pet Okay, now this is that pet The question one could ask. Okay, but not but who comes up with this is okay Like this is just magically made up or and what more if you start up in a different platform It may well be the case that not real mail be technically for sure gonna be the case that you can have a totally system Totally different type of layout here or you may have more or less element. Okay, so The question to this is before at the moment at least there is different location where the standard library is identified And there's a different Location where the other side packages are inside all the standard library discovery happens straight into the python binary so this is kind of like a built-in logic and then Inside whatever customizes the global side packages is the side of pie This is at least the most of the case you can Delabel the side of pie. We're not gonna cover that case too much But in that case the built-in interpreter gonna fall back into some built-in logic But this is very rare in practice. You're most often what's gonna happen is that on your machine in the standard library They're gonna be the side of pie, which is gonna magically customize and they come up with the global and user side packages okay, so If you can look at it, you see there's some commonality in this and all those other packages which are outside of the Virtual emphasis on side package and the user home side package all these other ones start with this magic prefix called opt python 3.7 and this is a good reason because the way this list is generated is basically By using some prefixes. There is these two Variables that you can change but they go to metal gonna have some same defaults This prefix and sees that exact prefix and this is basically what's gonna determine where the python gonna try to find this thing these packages, okay now obviously Python doesn't Just magically come off it. Yeah, it's right here. It's like a very hard-coded version It's actually dynamic and what ends up happening is that there's a different Algorithm that comes up with this logic and this is actually available Mostly in get-pad.c this has changed quite a few time as the C Python corp team for example started optimizing startup And they needed to move around thing to make things faster But the rule of thumb is that if you set the Python home, we're gonna use that for C's prefix So this is a great thing if you want to break someone's environment You just set the Python home environment for able to something that doesn't exist and everything's gonna be broken So but otherwise if this environment variable doesn't exist We actually are looking for a landmark file and the landmark file chosen. He's actually this OS dot pi so the Python gonna Start from its own executable. We are the Python executable selfies I'm gonna start going up and at each level going to see can I find this Leap Python X dot epsilon OS dot pi file if I can find it that is what C's prefix will be Okay, and another Otherwise it's gonna be like a built-in hard-coded value But this more like a fallback most likely will not work because this is whatever the compile whoever compile your Python set is gonna be exactly that one and There is a different logic for the exact prefix which is basically the C extension modules the only different hill that in this case It's not gonna look for the OS dot pi File instead is gonna look for the leap download element. Okay folder and wherever it finds it folder. That's what the exact prefix is gonna be set to so That's more or less it We only need if you want to create a virtual environment like roll your own or most the only thing you have to do is Modify this C's exact prefix and six prefix If you're done with this you're all your power is in your hand. So how we created This environment in order to have shared access to the host We just need to make sure that whenever Python starts up is going to resolve the standard library element by having available those Two landmark files the file and the folder. Okay, if you want to actually create we want to Have the system side package added or not This is just a conditional logic in the side of pi the side of pi automatically get loaded before any code gets Excluded at the startup of your Folder so if we do any changes here whenever the user will get the interpreter will already have Modified C's that that's gonna work for him and similarly adding our own virtual environment side packages It's building just just some logic that we can tossing into the side of pi. Okay, so What are the changes needed to actually have a virtual environment? So I just take a python executable and I want to not use the system one. Well We just need to meet keep track of the host so I can actually find this global Standard library. We just need to know where that was available So I can set the C's prefix according to that and just some flex into the side of pi for anything Now according to the path I will say this configuration information is actually sorting is this pyvm.cfg And this is something that Python 3.3 or later actually already will parse and read and the way is gonna Set the standard libraries for the standard libraries C's that prefix is gonna be that basically gonna look at whatever that home value is Okay, if that home value points to something from where it can find those landmark files You're gonna be happy. It's gonna resolve the correct system side packages And otherwise just the fact that should we add the include system side packages it's just this second argument that includes system side packages equals false and The fact that should we add the virtual environment side of pi or not is basically just done by is this file exist then yes, if not We're should not bother with that. Okay, so quite Simple and straightforward in that sense Now there's one more thing if you create a virtual environment The it's gonna be not gonna be much of a use if you can't install packages in it So the first thing that you'll have to do once you create a virtual environment. You want to be a want to be able to put Pipe into it now this is kind of catch 22 because usually when I wear someone's I'll install a package You just go yeah, I use peep to install it But in this case we don't actually have at this point any peeping that package yet so how do we install peep without peep and the way we resolve this question 22 is basically We put we have a wheel of the whatever we're going to install or as if file of the peep and we can actually put this onto the c's dot pet and This basically gonna mean that for that invocation People gonna be available. We're gonna be lookable by the importing system So it's gonna be able to start up and then we're gonna ask peep Can you please install yourself by using this wheel that I just pointed to you at and Pretty much. This is what in yourself installing itself and pretty pretty much what ensure peep more or less does it does a bit more Magic, but this is the core gist of it. Okay, so Now we have a virtual environment. It has a lovely Peep in it. We're kind of like done. Yeah Well Now let's have a quick look at the implementations. So the implementations out there available the one that's recommended for use So to say is the vm the vm is a standard library module is built in within the interpreter and It was only Python 3.3 only but hey Python 2 we're gonna be deprecated next year. So who cares It's gonna be also available of the sender library. Well, mostly feels like Debbie and derivatives decide not to install it So and it's only made copies is gonna be able only to target itself So if you want to create a Python 3.4 or 3.6 Virtual amp you must have available Python 3.6 or 3.4 with virtual amp within it to be able to do it Okay Now there's another thing called out here and this is for what I'm the maintainer is This virtual amp thing and how virtual amp differs from that built-in VM should if this project even exists anymore Or can we just drop it and in order to answer this question? Let's quickly go over what virtual amp offers on top of VM Okay, so the first thing out is is that it's a third-party package which means that you can upgrade it out of the band So you no longer have to wait for operating system to operate your Python. You can literally get the latest virtual amp creation By just installing this package. Okay, so in that case, it's a bit more future proof it's a bit more easier to have the latest of it with the latest features it also targets to support both Python 2 and 3 and It supports C Python and PyPy We have some gyton implementation iron Python is something that may be work in progress But if you have interpreter and you want to have a virtual amp for it And maybe you don't necessarily want it within your standard library Just submit a pull request and if it looks good, we're gonna accept it It's also cross Python support meaning that you can actually if you install virtual amp on Python 3.4 It's gonna be able to create for you virtual amp for all supported pythons Okay, so even if the VM package is not there for the 3.6 It's still gonna be able to create a virtual environment for that 3.6. This makes it very Useful for tools such as stocks, PPM, all this tool or Ppix all these tools because now They can actually don't have to check if the virtual amp is correctly set up or not They can actually just have a look at it and They can just use this tool and this is guaranteed to be work for all available Python executables. Okay Another thing is course path from support obviously and it has built and interpreted the discovery meaning that and this is especially handy on the windows You don't even We automatically discover all pythons available or try to do our best all pythons available in your operating system So you don't have to figure out. Hey is my Python 3.4 in on the path Is Python 3.7 on the path especially on windows for example the path file 4 allows on windows tools to actually register Their Python existence within the windows registry and the virtual amp go automatically gonna load this and it will be able to You can just specify to eat that dash P I want the Python 3.4 and we're gonna discover it and create it using that and similarly We're gonna use a path discovery method on we by using PEP 39394 this basically means that we're gonna be able to Create You're gonna have a single virtual amp for creator for all the pythons And you don't have to use have different form for different Python versions Okay, so and it also has its own bundle PIP set are sweep to the bootstrap meaning that it can work in offline Bond but it also available to use index servers So it's possible to install the latest one rather than whatever we distributed the last package with Okay So you may have like a newer PIP setup who's wheel and this in turn causes that you have less time the awkward case when Something doesn't work. You go to stack overflow and they did you upgrade your PIP? Okay. Okay now it works so Let's have a bit of the other thing is though because we need to support Python to you We're trying worse a bit differently than the amp anything I everything I told you on the beginning of the Vm that I Vm the CFG is not a thing in virtual amp So let's go over how virtual amp does actually create a virtual environment So we have to basically do the same on alter the standard library logic discovery and alter the side of pylogic To be able to create a virtual amp, but without the interpreter having any knowledge of any configuration files And the way we do this is basically the first thing is we always copy the Python to at least the Executable and we put it in a folder that looks like kind of like a python virtual environment Okay under like bin in like in this macOS then we created two landmark files. We just seemling this to the operating system one so they actually Contained the same thing as the operating system one Then we had a few configuration. We have a configuration flag if you should add the System Python or not. What is the prefix from where the Python that we created with is available? And we inject our own side of pi now the reason we need to inject our own side of pi because It's because the side of pi that is shipped by operating system is not aware of this magical Oric prefix takes the or no global prefix takes the So the if you would use the side of pi that it shipped with the operating system That's what magically ignore all these configurations and you will not get an actual virtual amp So for this reason we have to like whenever you the virtual we have to like basically light to you and have our own side of pi this kind of like works for all Platforms out there now you can imagine if this works for all platforms and all pythons. It's not too easy to maintain so There's one more thing the side of pi gonna take care of adding everything on the path But there's a caveat it seems the side of pi doesn't work on its own in magic Just it actually works by using Python standard library modules But at this point at the point the side of pi gets loaded those are not yet on the path So, okay This means that in order to be able to load or start up the side of pi file and that's gonna do the modification We first need to be able to modify The to be have available all those packages that side of pi happens to use now this is a short list of what we need basically and Again, this is kind of like platform dependent and each and version dependent So, yeah, you can see that we create a lot more Links or copy them literally over the standard probably just so that our customization logic is able to run And yeah, it's few I meant 21 in this case I don't know quoting a recent show not great not terrible could be 60 or 60 so 25 maybe let's work with it And then we just have to do some fix up basically at some activation scripts link the include Fix up the distutils cuz turns out the distutils uses its own kind of side package discovery So we have to basically ship a different distutils and your operating system And then we just fix all other bugs that are available a long list of it like being able to discover Tlk platforms having python coughing run the documentation run you can see there's a long list of fix that we have to do for this to work and Just yeah, that's more less with now Just to have a bigger thing In this virtual environment the only thing that we actually carry that lonely side of packages everything else is just There to make things work and just for a comparison. This is what VM has Looks a lot slimmer no So and the reason why I can get away with having this last slimmer packet since last Slimmer director is three is because it's able to the interpreter itself is able to know about virtual environments And the interpreter itself is able to basically load these configurations from that pi VM. They don't see FG across it Okay, so yeah next time you create a virtual environment and you see this on the your hard drive don't despair. It's just How it is obviously this is not a good way to have things and because we basically said before we Patch both the side of pi and the disk utils as opposed to what your distribution ships with we have a lot of bug reports around ID doesn't work on this magic random platform because basically that has a customized disk utils of the side of pi and various features start breaking and because this is kind of like hard to maintain I Start doing the rewrite and this is my proposal Basically the idea is that it's gonna still be offered as a third package So this is what I'm gonna differentiate from VM. It's gonna also have all Python implementations welcomed It this means including Python 2 and 3 like pi pi is even to the see Python gonna be deprecated next year Pi pi gonna be supported to version gonna support it for foreseeable future So we still need to be able to create virtual environments for that interpreter So we're still gonna support that we also gonna have a two years grace period meaning that well I put here two years, but maybe I'm gonna make a year and a half the idea is that we're gonna maintain our virtual amp to be compatible with a Python that has drop support for its Implementation a bit earlier and the reason we want to do this is that you're most likely unless you live in a perfect world You're not be able to migrate to the latest fanciest Python immediately And we want to make your life a bit easier You still can use virtual amp to actually create VMs for your favorite deprecated Python So it's also gonna be able to select target pythons It can even do fancy things such as for example request a given subversion or request to give an architecture on Windows Again, this is kind of like maybe not as Interesting for the end user, but it's something that tools definitely want and it's very handy and One thing in order to minimize the bug request We actually gonna what they're gonna do is physically gonna ash If possible, we're gonna delegate the work to the VM in the spirit of why work if others can't work for you But we really just gonna make add a few optimizations on top of it And we're gonna have like a broad shell activation support This is something that differentiates for us for example also from the VM that we actually support a lot of other shells Then whatever the standard library does for example We support the Sunnish Python shell interpreter and also we're able to move much faster For example when power shallow all of a sudden become available on the POSIX operating system, too We immediately added that now we generate that generate that activation script that platform, too Okay, and we're gonna keep doing that and we go ahead and Make sure that it features work similarly And we have a three-layered configuration system Which basically gonna allow you to have some global settings customize how workshops are created on your hard drive And the kind of things that you can this can be useful is for example Whenever peep releases a new version and for some reason is broke broke You can set it at operating system level that I actually want to use an older peep and create all my virtual Environment with an older peep not a lift but still the less known working version or something like that Okay, and we might also have a plug-in system for extension This is mostly something use that people are able to create their virtual environment Implementations for the pythons without necessarily having to get the poor request to us. You can basically load And add test it on your own and then if it's stable we can upstream it within the virtual environment itself. Okay So yeah, that's gonna be all my talk. I think I might have a question left or so. So, yeah, thank you So, thank you very much for this rare insight There is time for one quick question and any other questions you will be around afterwards So feel free to do that after the talk. So is there any quick question? Sorry go on. Yes Thank you for that was really interesting to see the internals You said you were planning on doing a rewrite which would include kind of support to select any Python version Yes, obviously say you're doing front-end You can use like MVM at the minute to kind of practically select any node version right now What is your kind of go-to for downloading and managing Python? versions as opposed to virtual m's For Python versions, I either install it from the upstream or I install it for example using Pi MV Or I Other thing which has something I use conda which has but like this for example would allow us to even you be able to Say that you want a conda environment And if you have conda inside of the system It would be conda would actually create its own specific environment And that could be still managed by the single front-end of the virtual amp instead of having to use multiple tools virtual amp would kind of act like it's unified Interface which is very nice for tools, especially. Yeah, pretty. Thank you One one last. Thank you and welcome for this talk and then that's it. Thank you guys