 So thanks for the introduction. Can you can y'all hear me? Do I need to speak up a bit? Yeah, it's okay. Good Great, so Pyron This is something that I do for my day job. We're not doing your Python So just to give you a little bit more background of what I'm what I'm actually doing all the time During the day, so I have my own consulting company. I'm a CTO of a FinTech company in Cypress I'm a senior software architect and have done lots and lots of things in the Python community You can read up all on that on my blog. I don't want to go into too much detail here But jump directly into the talk. So what is Pyron? Pyron has a long history. It started. I think I don't really remember. It started in the late 1990s. I think it was 1998 is kind of And it started with a completely different project. The project was called MXCGI Python At the time I named all my tools MX something because of a naming conflict I had with the Zopcorp packages for MX Daytime and The idea there was that I wanted to use Python on the typical FTP Website hosters that you had at the time. So in those days you you couldn't just you know upload a script and then Run it if it was Python. You could do that with Perl. They all supported Perl at the time But Python was not really a Thing there. So what I wanted to do is I wanted to get you know sneak Python in on the on the web hosters machines Which was kind of like hacking, you know an executable in there And it worked really well because I found that you can upload it into the CGI bin directory And then you could also upload a shell script, which then turned your uploaded file into an executable So I thought okay, let's let's try this. Let's do this and and take Python Make it really small upload it because FTP space that at those times was expensive And so we we I wanted it to be really easy to do so I just created a single file out of Python and Then uploaded it to FTP to the FTP hoster ran the script And then I had an executable Python interpreter and I could upload my Python script to the CG CGI bin directory and run it and I was not alone with that With that wish to run Python on one of these hosters There were actually quite a few other people so the project sorry about that I Just press here So there were quite a few contributors who then created these single file Python binaries for lots and lots of different systems at That time there was it was not like today where Unix basically means, you know, you have Linux you have maybe free BSD and then maybe you have Unix system like macOS There were lots and lots of different systems solaris HP UX all kinds of variants of that So you you needed to first figure out what system was running at the FTP hoster And then you could upload the correct binary so that lasted a couple of years it then padded out early in the 2000s because then the webhoster started to support Python and Then basically I dropped that project Now in 2009 my company was producing a Product or wanted to produce a product which is one part of the product is a Is a server application that's written in Python needs to run on on Linux And I needed some way to ship this product to clients And so the the problem that I ran into was that if I would were just to use this the OS based Python installation there were so many variants of that OS based Python installation that I couldn't really support it because I was basically a one-man show as a company and So I needed something that had a basically a stock Configuration something that where I knew exactly what kind of Python to expect And so I remembered that I had this MX CGI Python project and I then revived it and then turned it into a bit more than just in the single executable For Windows it the solution for us was very easy. We could just use pi to XE for this so Pyrone curly does not run on Windows because of this because we don't have a need there But on on Unix you there was no appropriate solution for that so I started to do The revive the MX CGI Python and I beefed it up a bit So I had a business requirement No, let's add some more business aspects to this So the business requirement for me was to create a single executable that Has the complete or more or less the complete Python runtime including the standard library in a single file So that installing Python on a machine literally becomes a single copy operation And I wanted it to work on Linux on free BSD on macOS Those were the ones that I was interested in at the time It probably does work on other systems as well Now how does this work? How many of you know the freeze tool in Python? Okay, I Bet you didn't know about this five years ago because Or maybe let's say ten years ago. Python 3 is a bit older now so The the freeze tool what it does is it takes Python modules It compiles them to bytecode and then stores the bytecode in a C struct or array And then it puts everything into into a C files and then compiles everything as Python C extensions and then puts that into Into a module file that you can then link And so this is how you get how you can get a Python code into a an executable or a library And this tool has been around for ages. It was written by Guido himself Later Mark Hammond extended it to also work on Windows I don't exactly know why Guido wrote this but he probably had some some use case for it Nowadays, it's used for the import lip because for the import lip You have this bootstrap problem that you first need to if you want to run Python You first need to get the code from somewhere the Python code and import lip is written in Python So the the issue is that if you want to import something You know that the import has to use import lip and so you need to somehow figure out how to do this and the way It works is that a small Let's say a core part of the import lip is actually frozen into Python as well and and this is why it's Why it started to be used again when I started to write pyran The freeze tool was not maintained anymore. So I had to do some fixes to make it work again So how does it work? Essentially, I wanted to take the standard library, which is mostly Python modules. It's also a few C modules So you had to do two things one was to get all the C modules the Extensions that are that are being built in the standard build process of Python to not be compiled as shared libraries But instead a static library so that you can link them directly into the interpreter And then the second step was taking all the Python modules that you have in the standard library And then convert them to C extensions as well and also link them statically. So you get everything into a single file Now, of course, you can do this for a single application It's a bit tedious to always redo everything for every single application that you want to run or let's say you want to do a Release cycle in your in your product and you always have to run all these things again Which I didn't really like so what I decided to do is to just take The the standard Python Turn that into a single executable and then wanted to ship the application code the Python code as a zip module So that you essentially get two files the the py run and the executable and then the zip file with the packages the Python code now That was relatively easy to do But then I wanted a little bit more because I thought that well, we're almost there We always have almost have something which is more or less identical to Python And it's tiny and it would be really nice to use it pretty much everywhere instead of virtual ends For example, you can just you know, you copy up your py run and you're done You don't have to have a separate installation for a virtual inf So I thought I'd add the the Python command line as well Now the the problem is that the way py run works It cannot use the the C command line parsing that we have in Python But instead it has to use Python code for this. So I had to rewrite Most of the command line parsing that's being done in Python in Python and then you know again do the same thing wrap everything put it into put it through Freeze and then put it into the executable So I managed to do that. Okay, that's very nice I managed to do that It is a bit slower of course because it's Python running to do the command line parsing but there's a trade-off there because When importing things from the C extensions that freeze builds the import is a lot faster than going to the file system Because it doesn't have to go to the file system file. I always always very slow If you load everything into RAM it's much faster. So I Could make it a bit a little bit slower Then gain all the flexibility and then have it work and it even supports interactive use now So you can actually start it and it comes up with a command prompt again So you can use it just as regular Python So this is essentially where I am. So we now I have a py run. It's open source. It's a free more less drop-in For a standard Python runtime. It doesn't use hundreds of megabytes in the file system. You don't have to install it anywhere It runs on or let's say it works with Python 2.7 3.6 and 3.7 now It also supports lots of older versions So 2.4 is the oldest versions version that I still support not in the current version of py run But in previous ones and it runs on all the platform forms that I wanted to have it run on The executable size is between 3.7 megabytes for Python 2.7 and 4.8 megabytes Of course, I'm cheating a bit. I'm using up x compressed for this But anyway, it still works. It's the startup time is a bit slower So this is what it looks like and because I wanted to Not only talk but only show some stuff. So This is the project Let's go here for example for 2.7 In case anyone is still interested in that So here you go 3.7 3.7 Can you read that? Is that should I make it bigger? No, it's okay. Good So just to to demonstrate how this works. Let me just do this and you can actually see it So this is the UPX version Where is it 2.7 UPX? So this is what you get when you when you run it and it works, you know in standard way you can import stuff you can You can do all kinds of things you can you know, basically it's a standard Python You can also run a pip with it. So I can do like this Actually, let me see whether it's already installed It is already installed so I Can install pip I can install setup tools and I can I can then also run pip the pip will then use This is a bit annoying. I can then you know install something. Let's say Let's say this one Okay installed something and then I can go here and I can I can then run this this is a game of life in Python So that was 2.7. Let's go to 3.7 3.7 is a little bit So you can see here down here 4.7 megabytes The original one the uncompressed one is 14.4 megabytes, which also is not that big But it's it's amazing that you can actually compress it down to that size And it works in the same way as the 2.7 when I just showed so let's go back here By the way, if you have questions, just feel free to ask. Yeah, so we I Can just answer them right away These are some use cases of a janks pyron, I'm pretty sure that there are lots more These are mostly the ones that I came up with. I know there are other projects that try to do similar things And they are better in marketing than I am so they come up with more use cases But the ones that with that we really care about is that we're independent of the OS Python installation That's the most important one. We want something small to easily ship to to clients We want to easily make it available as a download without having too much bandwidth use It's extremely good for docker containers because for docker containers you need because it's so small You can just easily put it into a container image and then you the lobe in the image is very fast It's much faster than having a regular Python installation there And it's very easy to to build single app applications out of it. I'm going to go into that detail some more Because what I've added is integrated zip file support for pyron and the way it works is it's very simple You create your your Python application Very important is you have to add a Dunder main module. How many of you know what Dunder main does some of them? Okay, so the way it works is when you when you have a zip module and you have Python run that zip module Then if it finds a Dunder main module at the top level of that zip file, it will execute that so it works basically like what you have with the with the typical The the main execution part that you put into Python scripts And so if you have an application like that if you have a script and you just add this Dunder main module to your zip file You then concatenate pyron that zip file you produce a new file. Let's say hello You make it executable and you're done So you don't need compilation anymore So this this basically made my day because I did not have to send zip files around anymore For like to say application updates. I could just create a new executable and send that around and Just so to show you how that works So I prepared a little something here. So you can see there. There's the the main module It's just a very simple hello world, right and Then you have the I put that into a hello zip file Actually, I don't want to unzip it. I just want to read it It's We just do that again So I put the the Dunder main in there right and Then I concatenate the two so I can I can choose the the slightly bigger one the 40 megabyte one Which is faster to load or I can use the upx one, which is smaller to load So let's use the upx one and I simply put the the hello zip behind that and I Create this one UPX Let me make it executable You run it There you go So As you can see now Thanks, so It's like magic, huh? It's like, you know, you turn a hundred megabytes into four point seven megabytes Yeah So it's it's really really easy to create these single file apps now I'm basically these three steps or two steps that I have here I'm probably going to turn into some show script or something as well You can just run that okay, so Customizing py run of course this this was the like the easy way to do things you just get the py run for your platform Maybe you have to compile it yourself Normally we provide binaries for these We haven't done that in a while because the build from that we had basically crashed and I haven't had time to Refix it and then get it running again So the nice thing about this is that you don't actually have to be a core developer to do this It's not really if you know the right places to fix and the right places to tweak then It's it's not that hard. So of course you you should know a bit about how to freeze works We I have put this I put a special file in here this py run done the X Extras.py in that file you just import whatever Packages you want to have and then freeze will automatically find them for you and then integrate them into the package So it's very easy to to add new modules It's a bit harder to exclude modules So let's say if let's say you have a Python package that has test modules And you don't want to ship the test modules together with your product Then you typically want to exclude that and in order to exclude things You actually have to go into the make file and then into the excludes Variable and put put your particular what you ever would you want to exclude? Let's say your test sub package you exclude that so it doesn't get integrated into the package And then the next thing is if the next step Let's say is if you want to add custom C extensions that you have or maybe you have dependencies that have C extensions For those of course you have to tell Python when it compiles to Add those to the executable that comes out in order to that you You have to use the module setup How many of you know or have ever edited this file in the Python installation? When compiling it Extremely few okay So here's some fun So let's say What's the name again? Let's say we go here So this is a standard implement standard Distribution of Python it's it has a few patches because I need to do a few tweaks for PyRUN, but not really that much and then The way that that Python determines whether to compile any of these modules that you see here Into the into the executable that you're building or into a shared module It looks into the setup file Which looks like this. This is an extremely old file. It's still references the make fire pre Logic that Python used in the very early days to build C extensions So in the very early days you did not have any something anything like this tutorials you you had to basically do everything yourself You used this make fire pre in Concept and then you added your configuration into one of these setup files And and this would then make Python compile your those extension into an executable or a shared library And then you could put it into your package So there's lots of description here You know you have it looks a lot like a make file Then you come down to this section this section down here tells Python which modules to actually integrate into the Into the executable and then there is let me see where they can find it So all these modules that you see here. Those are statically Compiled and put into the right here it is Statically compiled into into Python and everything that comes below this this shared indicator in here This will then be compiled into a shared library And as you can see here, I commented out that shared so everything that comes below is still going to be compiled into the the executable and because that's what I wanted and As you can see here, these are just you know C modules that Python son at lip uses and then I had to make a few fixes because I wanted to add some of the Modules that typically don't get added as a static Version into into Python This file is not being let's say it's not well maintained anymore because Normally nowadays everything gets gets compiled as a shared library And so some parts are missing like the the the various show modules here for example, they were not Not in that file. I had to add them Some things are also removed for example tk inter I don't use so I did not put that in And then if you want to add other stuff, then you can just go down here and Just append it so the way it works is you just have to tell Python where the C code is Whether you need any parameters like for example here This is the these are the things that you have to do for sequel light And then you just add it to the setup file and essentially you can just let the make run and everything works out by itself So That was a short tour through the setup file. I don't think I have time to actually show the complete compilation, but Let me just maybe Go through the changes that I had to make to go from three dot six to three dot seven And this is interesting. I don't know is Victor here No Victor's dinner. He made some changes between those two releases and Because the import logic in in Python sometimes changes from release to release or there are new new ways of Integrated into the Python build process of how to configure certain things you always have to touch the the code base a bit There was a lot to do from three dot five to three dot six the the path from three dot six to three dot seven was Basically just a few hours work This is just to give you an idea of how how you can port py run to new Python versions It's actually quite easy. You just take what's there already for the existing Python version and then you have to tweak the patches a bit You have to sorry You have to then adapt your setup a bit because new models get added, of course Others may need some tweaks in terms of definitions that you have to add and then basically Things just work and then the freeze tool itself also sometimes needs some fixes because what I did like I said I had to tweak freeze to make it work with Python again and and what I did is I basically copied the freeze tool into py run into the the source code package so that I can apply changes to that as well and I had to make some some fixes there as well and that was done. I was done and everything worked so This is where you can get py run Like I said at the source and the released source and binary versions are a bit a bit older. I The release ones only support three dots five as the latest version and then two dots seven What I will do is I will put the the current version that I have already working I will put that up on github so you can download the sources and then compile it yourself Compiler get yourself. It's pretty pretty easy It comes the package comes with a make file it comes with some documentation You just have to run make make distribution and then basically you're done everything is should then be done for you and then you can just pick up the The release package of py run from the distribution directory and then you can use that Yeah, that's it Thank you for your attention We do have time for a couple of questions. There is a microphone there. You can just ask away Thank you for the for the talk. I have a question. Is it possible to use py run to embed Python in Binary's or if not, how much work would that be to Create a static statically linked library to be used by executables You want to take py run and put it into some other executable? I would like to have a Library that I can then use to embed Python for example a C program and So that library doesn't have any dependencies external dependencies That would require some work, but it's possible. Yes, definitely So essentially what you have to do is you have to basically you have to change the the way that the main The main function works in in piracy you have to remove that and turn it into a library But I suppose you could do the same trick with a lip Python. Let's say or lip py run And then get everything integrated You probably have to define some some entry points there for the library some new ones To get everything working, but it should be possible. Yes Okay, thank you time for one more question anybody Well, I have a question Thank you Thanks for the talk I'm just just wondering come with this pattern it looks great and everything but I'm just wondering whether there are there any Limitations some features by some features that you cannot use it with it I mean are there any downsides or is it a silver bullet for for all the problems? Well, there is There's one downside with this whole approach and this is also why for example the the Python test suite does not fully run Which is that some packages they put Extra files, let's say let's say text files or symbol files or whatever into the packages themselves And because of the way though those packages are written They try to actually access the file system to get to that particular extra file that they have put into that package And of course py run doesn't have a file system. It doesn't also doesn't emulate a file system So the the files are not available. I Had that issue with py run because for example the Python grammar is one of those files It gets put into a special file that it gets installed in the in the file system And in order to make that file available I had to basically take the file and in the process of building py run I had to integrate it into py run as well and then you know write some extra code to make it available via the standard API's Inside py run, so I had to do some of those tweaks I only did those tweaks for things that Python itself needed if you want to if you have something like that Then you would have to do those tweaks yourself But that's that's actually pretty much the only limitation. I know of The there I could imagine that some packages that have external C extensions that they might be hard to compile into a static version Because sometimes they the way that those shared libraries are built is very complex and and if you want to turn everything into a static library Then now you can you can run into problems But it's you know, it's just a matter of effort. You can get this working Pretty much in any case. All right. Let's thank Mark again