 Today we want to talk about Stackless. Former name was Stackless Python. I don't know how many people know about this project already. It is a rather old one. And in the last couple of months a lot of changes have been made. A lot of enhancements which really make Stackless very usable and a drop in replacement for C Python. And that's mostly the work of my colleagues. And one of them is Ansem Koiz who is giving the majority of this talk today. I think to do an introduction about what Stackless Python is. And then we get to the new things and... But please interrupt me when I'm quibbling too much. I would do the same for you. Okay, so... Is that good? How is it looks? It was okay. Presentation mode? Okay. So what is Stackless? It has been around since 1998 and had its shadow existence behind regular Python and it is kind of a niche product which tries to do things that normal Python cannot do. So it is called that Stackless is a Python that does not use the C stack. That's a lie because of course it does use the C stack. It does not have problems because of that because the C stack is always built up and removed again. So Stackless means that there is no dependency from the C stack. And that is the main reason to be able to do task switching and other stuff. So actually Stackless is a Python version that does not keep state on the C stack. It has some state sometimes but at certain times where it can perform some context switch, this stack state is removed completely. So it's breathing up and down until something else has to be run. So this is... Yeah, theoretically in an ideal world it would work this way but in the real world in 90% of all cases it works in this cooperative manner but not always. So Stackless is like C Python. Unless you import the Stackless module there's nothing changed at all. Not even the behavior, the creation of recursion level limits and all of that. It's all original C Python. It can do a little bit more than C Python. So there's a Stackless module and you can import the Stackless module and that suddenly creates a lot of new... a few new functions with a lot of possibilities. So it looks a little bit like an extension module but it is no extension module because Stackless Python requires quite a lot of changes to the interpreter. This does not change the way how you use it and it does not change the format of pickled things or anything else but the way the virtual machine is working is different from that from C Python and I think Stackless does it the right way and C Python does not but that's a long fight. I will probably lose in the end. So yeah, what is really... what is it about Stackless? Yeah, you have a little tiny... tiny programs. These tiny programs can be switched forth and back. You can communicate over channels between these tiny programs. The tiny programs are called tasklets. It's similar to so-called micro threads but there are no threads at all. A tasklet is just a tiny piece of program that runs a while and at some point it decides to yield its time slice to someone else and then it continues in another place in the program and that's a difference to C Python. That's something that was invented for C Python in 3.3 or 3.4 I think by using yield from constructions or so Stackless had that since 19 or since 2002 I think. Okay, so actually we have tasklets which are a little like coroutines and this can switch freely between each other and the ability to do so is yeah is the way the interpreter is constructed which is non-recursive but it's only doing small calls and going back before it switches to something else so we have no contention of the C stack which in C Python you have the C stack and if you have three nested calls of a Python function you also have three nested calls of C functions which do this Python function and that's exactly what Stackless does not. Okay, here's a little example we have some cooperative multitasking we import Stackless we define two functions receiving tasklet and a sending tasklet and then we just start them I did some examples just in order to save me from typing because typing is not so well but we can have some examples afterwards so let me skim over this so when the tasklets are communicating you see some protocol where you see that they are jumping from one context to the other and well some people might say why don't you just use the greenlet the greenlet is actually quite famous and very complete and good implementation of the task searching it does that with some technology from Stackless Python which is called hard switching it does that by slicing the C stack into small pieces and really brute force context switching so it is quite perfect because the greenlet has been unmaintained for many years and it's always working because it does not know anything about the things it is switching so it is very stable and very major but it has the drawback that it's slower than Stackless Python and it has some things that it cannot do so the technology of the greenlet is quite close to Stackless 2.0 where we only had this hard switching grabbing pieces of the stack and moving them around the soft switching does that in a cooperative manner so the stack is actively unwound and the context switching is built into the interpreter and that is much more work because it's harder to maintain it's also much more efficient and has some other advantages we come to that later so we might talk about the hard switching but I think maybe not now hard switching and soft switching is some real difference between Stackless and greenlets but the real thing actually is with Stackless Python we can pick a program state so we don't have only the ability to switch around like greenlets can do but we can pick the program state we can run a program to a certain point then take a snapshot of the program and take that snapshot on another machine or tomorrow or whenever in the cloud and continue the program from where it was picked so I have a little demonstration that takes simple recursive functions just to show how that you have a recursive function that calls itself until certain level and then winds back and well if you run that then you see the function prints one to nine then some word and then it goes back comes out of its recursion and the interesting thing is when we restart that program later on then we can see it continues exactly at the point where the program state was saved and continues to run that part of the execution okay yeah okay okay this is the we have that as a running demo as well and I think we should show it right now or later now okay okay so I have well do I need to make this bigger okay I run the program this is the recursive run and it's a little bit enhanced the example I worked on that yesterday the recursive function goes to a certain point comes back and then it says well I was recorded on Darwin so it was recorded on this platform I can now run it it has written its output as a pickle into a tasklet dot pickle and I can run the program again and give it the the pickle as an argument and you can see here at the point where at the high that was the the the program state was written to the pickle and here where we restarted the pickle it was it came out of the recursion so the the stack with with all the recursive calls was pickled and was now revived and continued to run and we can do even more because sorry so I will now it's dangerous because I'm now starting Windows and I have prepared a Windows share let me see if it works okay yeah we are now in the same on the same virtual drive in the virtual parallel windows machine and we have the same program here and I can run that okay so I started the same program on Windows it returned from the curtain and tells us I was recorded on Darwin but continued on win 32 and it's actually really the same tasklet pickle we recorded you look at the timestamp just four minutes from now and actually we are using this technique in production at large important German auto manufacturer is how to make us to control high performance computing jobs where you start on the desktop engineer on the desktop of an engineer running Windows and then move a program to an HPC cluster running Linux so that's working well okay so I will do the same thing the other way around and then close the Windows shell because okay so I did the same thing here no I'm stopping the parallel desktop and run the same thing on the other side and let's see what it tells us now yeah I was recorded on win 32 but continued on Darwin so just so everybody believes that we moved a running program from Mac OS 64 bit to Windows 32 bit and it still works okay well that's just a summary of think about what you already told us green that was a stack less yep okay now she skips this slide yeah I think I said it all right okay so the reason why stackless really makes a difference is oh it's still a bug we have the persistence we can use persistence the cloud computing move running programs around use a different computer use a different operating system do it in a cloud the cloud is the only thing that I didn't show right now we can save snapshots you can also use multiple snapshots and whatever you want so actually you can do with stackless everything you can do with regular python it's fully compatible to regular c python and it's much it's even much more fully compatible now as it was before yes indeed we fixed some corner cases where extension modules that didn't adhere to the coding standards for python extension modules didn't work with the stackless unfortunately there are quite a few and important extension modules of this kind for instance pie side and pie side is a very important thing I'm working with pie side since 2011 and I could not use stackless Python it was crazy in spring this year this problem has been solved now it runs really well with yeah right I actually actually I have another thing to show um what you see here is a little puzzle game and this puzzle game is that visible yeah this possible puzzle game is written in is a demo from pie side and this runs on top of stackless python so the old problem has been solved we can well and so on another little demo is for instance so we have a little painter demo painter demo some different colors which are painted uh with different styles and certain rotations so all great things it's actually stuff for for an extra cute pie cutie or pie side talk I was just so happy that this now really works with stackless so I had to show that the last thing uh a nice embedded dialogues here we go this is um a projection of a dialogue into an open gl perspective and so it's very sophisticated huge stuff here and you can have that all in uh in stackless python now so yes yep I think that was about what I wanted to show and say and I might pass over to you now okay so um there's also a few things that stackless can use for uh Christian didn't mention yet so um you can use the build as a primitive functions stackless provides especially the task is in the channels to build a control structures for higher level scheduling paradigm so you can we can implement for instance a c-sharp style async programming or um we can implement so go less things uh Christian could uh tell something about it I'm not so in this business I'm more in the um pickling and migration of programs so now we have seen what stackless is and we have to talk about the stackless project too um the stackless project well who is it that's all so we are some uh of course the maintainers uh and but also all stackless users so who's using stackless and well there are a few uh few well-known users as a nagari I hope I pronounce is correct a web framework is based on stackless then ccp games the company behind their eave online multi-user online game you probably know it yes as some spacecrafts flying around and so very cool graphics and I can show it here I have it on the machine well actually um they use stackless since many years and have bought a large part of their infrastructure on stackless then there are few less known users um customers of the company I work for science and computing using stackless pisons on a few thousand systems and the systems all together have over a hundred thousand cores so that's quite a bit and well then there are also unknown users so I recently uh looked at the um access log files of www.stackless.com and I was very surprised to find that we have about 220 downloads of the windows installer each day and 85 downloads of the source archive so I don't know who's downloading it but obviously someone does it and we also uh see that 98 percent of the windows installer downloads are for pyson 2.7 and for the source archives are 96 percent for pyson 2 so it looks like the production usage of pysons three is quite low so uh Christian already talked about the history and stackless was formerly known as stackless pyson I will probably say something about it and it's invented by Christian and I think the first version was a 1.5 is this correct? Yes the first was a 5 and 1.5 and it was continuation based and much more complicated so the history is the same Christian knows best but this talk is not about ancient times and the history of stackless pyson but about the recent changes some more information about the stackless project who's working on that well we are just a few uh volunteers Christian and son Christian Willard Johnson Richard Tew the what's the pronunciation of Richard? Richard Tew? Yes that's correct and myself and all in all we have it's less than one full-time equivalent yeah so so we can't do very much we have a few resources um so it's a stackless website stackless.com so you can get stackless pyson and so it's a primary address for this project then we have the documentation it's now hosted on Reeds and Docs and so it's very fortunate because we can now host as a documentation for every version of stackless so that's really a good thing and since a short time we have moved the development environment a bit bucket and we have the project stackless depth stackless and also I have a project for the stackless as a installer that can be used with the installer pip and I have to say we have a really really happy that there's a wonderful infrastructure provided by BitBucket and Reeds and Docs without this infrastructure it wouldn't be possible to maintain such a project so how can you use stackless well you have to install it obviously yes and there are few options the simplest and most common way is probably to get the installer from stackless.com and the installer is available for Windows and for Macintosh OS X so just download it and install it it's a usual MSI installer I don't know the format that Mac is yes and that's something really great because the installer for OS X I used that the first time now it just replaced my standard CPython from python.org it continued to run with all installed extensions because it simply swapped out the interpreter so I was very pleased with that it's a great experience now to have installers I never used installers but but now I'm very keen on that sorry yes that's fine and then we have for the Linux people and also for Windows if you have on Windows and using virtual and for instance and great way to install a stackless we made an installer an umbrella installer that that's a stackless python here and and if you install this umbrella installer with pip or easy install it detects your system and then in a second stage downloads a pre-compiled package that only almost only contains the Python library and executable and three or four library modules and installs them in a way that doesn't hurt your existing python installation you get a new command SL python stackless python and well it's working because stackless is almost CPython so we have no problem to implant this new command into an existing installation of course on linux if you do this for the system python you need appropriate access why it's probably you have to be rude and finally you can get to source code and compile it just as you can do it for the regular CPython and if you need something like a debug build or special installation options then that's the way to go so if something doesn't work that's all final theory but the practice is sometimes different but first stackless is major yes for the biggest part it's just CPython and 2.7 is uh now really solid and it has many users so usually things work stackless 3.3 is also good that's currently the only maintained version of python 3 we don't maintain a 3.2 or we don't release uh for 3.2 anymore because may I add that right now we have stackless python 2.7.8 which is very complete and working we failed to finish the python stackless 3.4 version and we want to do that on the sprint in the next few days maybe it works so it's actually my intent is to support python 3 as much as possible but it's quite some hacking and fixing so yeah so there are a few bugs in the area of pickling because that's where we have stackless specific extensions and also as python 3.4 got an extra and new version of the pickling protocol but we invited to participate in this yeah so if something still doesn't work we have the mailing list and we and you can always open an issue on our big bucket side and sometimes you might need commercial support and well unfortunately there's a that's a bit of a difficult topic you can ask christian he is probably will help you if you can and you could and negotiate an agreement with science and computing but we are more focused on bigger companies so the agreement might be too expensive but we can provide stackless support around the world 24 hours a day and 7 days a week if it is necessary and then I think you had a we had something you wanted to show interactively and we're running out of time oh okay fine yes good I will just go to the reason that one month um we switched I told already told it we improved stackless a bit low level tasks with methods are now much more complete and importance that customer interest scheduling is working more stable we are better have a better compatibility with c-poison and the documentation has been updated and we got a better debugger support recently so that's very important and a few other bug fixes and we also did something that didn't work out the documentation it's now on read the docs and that's the most important point and it is the retro c-poison documentation and a stackless specific at the end of an additional chapter and this chapter is now also complete and correct it wasn't sometimes ago and we also adhere to the pysons software foundation trademark uses policy for the trademark pyson and that's also the reason we changed the name because as a PSF says you should not use the name pyson alone so the source code also contains a well maintained change log file now and our wiki pages still need some love I probably will go over the technical details very quick we have a more complete set of low level methods and you can find that in the documentation it's usually not so interesting but we have here an image and it shows you that we have added a few that's a state chart of the tasklet yes and the tasklet is created it's not alive and then it's alive and scheduled eventually and you see we have added a few red arrows that weren't available previously so you're no much more flexible we have an enhanced functionality in the schedulers it's not long necessary to run the scheduler from the main tasklet you can run it from every other tasklet or from within a library or something like that kristian valio johnson implemented it and he was very excited about it but it's fairly new and there are no I don't know any application of this new functionality yet custom scheduling is no possible using the new method tasklet switch it simply transfers the control from the caller to the argument of the tasklet switch function and that's independent from the scheduler and it's an atomic operation so you can build your own scheduling based on it um there are known applications for instance we have in the stacker slip that's a library collection of functions you could use for your programs with stackers yarn it's not really necessary to use them up but there's a module called async and it is c-sharp style asynchronous programming and it also provides some futures pys and futures based on uh targets can you quickly show your interactive so let's move on uh multi-interstrate scheduling is probably not so interesting uh sub-classing also we have bug fixes already kristian showed you the pysite working and we have better debugger support now because we added some hooks for debuggers and we got uh two new eds working the wingward debugger is support stackers since ever since ever but now uh pydef and uh all the py charm started supporting stackers and hi yes yes yes yeah here we are so i will show the pydef debugger and we have a very very simple program it's an demonstration program from the stacker source and it shows you how to implement a mood tax uh using uh channels and uh targets that's really not important if you run this program i simply um run it yes it has some output yes uh we have here the three named uh targets and say uh the target function is this function f and while it acquires a lot as a mood tax and then it schedules it passes the control to back to the scheduler and then it releases the mood tax again and if you run it you see uh well it's a expected output yeah so run uh interleaved into each other so now let's debug it i already set up breakpoint here and we debug it as a python run um pydef switches to the uh python debug perspective and we are here so we step over that's the question of the task let's debug is that's false yes so now let's step into and here you see the uh actually the sweats or frame stacks that are known to the debugger now i switch here and we see how we got um still the main thread is now f but we got a paused main task that of main sweat and if you look here yeah that's the run that was the main task so now it's a different task that running so let's stop uh this one we will get the lock that's that's not surprisingly because we are the first task that uh so hey we're back here but now if we look oh there's another target so we have no three and you see uh that's the scheduled target ticks that was the target we have seen previously and it's sitting here in the schedule and writing to be reactivated and again but now if we look go into lock here with uh we will see here it blocks yes now we have the third target so you see the debugger perfectly supports uh the switching between the tasks and the switching between the context of the execution and that's important where without this support you wouldn't be able to debug this reasonably so i just continue it and done with it now and i think we have very few things still to say um future goals in the immediate futures we would like to find additional volunteers to help us yes then stack us as 3.4.1 and we also would like to install the pip installer to cover more versions especially pass and 3 and later yes we will probably still need additional ideas and uh i also would like to update the stackless implementation such as contained in pipi yes that's also something we could try on on a sprint to work on that because that's a few changes and it's quite a mixed of stackless programming with the pipi guys maybe interesting and well there are some uh things we also have done that was a thing that didn't work out we had problems with the visual studio install uh compiler with the old visual studio 2008 and um then we made a patch uh to build uh python 2.7 with the visual studio 2010 and if you don't change the name of the python dll then you can get serious problems because if the dll was um compiled with a different compiler it uses different c runtime library and this can cause all sort of problems in the end we uh currently can't publish it because we have no resources and it's so the possible violation of the trademark python and so it's uh still amounts are in our repository but it's uh well that's the question do we want to support a non-existent python 2.8 in the form of a stackless 2.8 which grabs all the things that are missing from c python or do we not want to do that and i know there are a couple of people who think it's a great idea and other people are there who think it's a very bad idea so we are really undecided and wanted to leave that to hear your opinions about should we do something about that yes okay and so surely we can't do it without participation yes and probably it would also need some funding yes so that's all and any questions so far