 Thanks, so yeah, I'm actually happy to have so many people here, so thanks for coming. So a couple of words about myself. I've been a Python developer and also an advocate since the early 2000s, currently working for Scooby, which is an e-book subscription service, so pay a fixed price per month in your unlimited reading, as much as you like. And besides that, I'm a consultant and trainer, so I give Scython trainings, also in-house, and I'm working on a couple of open source projects for which you may know me, the most well-known to maybe Alex and Mel, and Scython, Looper and a couple of others, so you'll find them on the page. Okay, so Scython is used a lot in big data applications, so I would like to give you a bit of background about what we're doing at Scooby. So as I said, it's an e-book subscription service, and we actually have a very large catalog, so there's more than 130,000 publisher titles, and it's actual publisher titles, not like self-publishing or something, and that's all books available to everyone, every subscriber, anywhere, anytime, you can just click on a book and read it, and once you have a catalog of that size, it becomes kind of tricky to keep it navigable, so it becomes difficult for people to find the books they want to read, and so that's the problem we're faced with, and the question is, how do I find my next book, right? So there's a lot of stuff you can do, you can have recommendations, editorial recommendations, but they don't really scale, to a large user basis, personal recommendations, you can crowdsource that, and certainly you're trying to automate a lot of stuff, so you can do a similar research between users, between books, data mining, and so that's a major thing that we're currently working on, and so you come and play it, and you can actually help us make our book lovers happy, because we're hiring, so go to that page, find a job, and come and join us, and talk to me in the recruiting session for example. So for the actual topic of the talk, the topic is Python, which is, what is Python? It's a couple of things, first of all, it's an open source project, you can find it on Python org, it's a Python compiler, it's almost a Python compiler, so it translates Python to C plus C API calls, and features static code optimization for you, it's actually more than that, in that it's an extended Python language, which allows you to write fast extension modules for C Python without, you know, resorting down to C and writing C API calls yourself, and it's a very nice way for interfacing Python with C and C plus plus libraries, and over the years, so Python is actually relatively old project, it was originally for it from Pyrex, who of you knows Pyrex? A couple of people, okay. So Pyrex was a project that was begun by Greg Ewing, somewhere around the 2000s, and in 2007, Rod Batchel and I forked Pyrex, which had kind of come to a certain store at the time, and founded the Python project, and since then we had a humongous success and huge rise in our user base, and over the time it's really become a major pillar of the Python ecosystem, especially the scientific Python ecosystem, but it's more general than that, so by now there are tons of Python code out there, and it's certainly, it's kind of a technology never, so as a program language, it really helps people to get their job done. So how do you use Python? This talk is kind of split in two parts, okay, so one is general introduction, I'm going to tell you a bit about the surroundings and stuff, and how it works in the surface, and then I'll show you a couple of examples afterwards. So in order to use Python, what you do is you write Python code, or Python code, as I'll explain later, Python then translates this into C code, so you need a C compiler to build a shared library for C Python, and then you can just import it and work with it there. So as you can see, there's a certain, an additional hurdle compared to writing Python code, importing it done, right, so you need to compile and build stuff in between. So let's compile some Python code, and it's just a stupid little example, so we have a class here, a method, some method that, you know, gets a, so you basically put a, it has a function in, and then you call it from this method here, and that's what we do down here, we instantiate the class and then we call this method, make the function be called a couple of times, and that's it, so really stupid code. Just a couple of lines of Python code, and when you run it through Python, it'll spit out a file, a C file of about 3,300 lines of code, okay? That sounds huge in comparison, and so in order to get there, it's actually quite simple, we have a script, so you can just say cytonize minus i, built in place, in place, i, cytonize minus i, and you work a file, Python scripts, and then it's just going to translate it, it's going to translate it to C, and then it'll build your extension for you, and then you can, as I said, import it and work with it. So why is that so large? In these 3,000 lines of code, there are lots of portability signs, so lots of stuff that says, if the C Python version you're compiling against is this, then optimize something in a different way than for different C Python versions. Very often that's the case for Python 3 and Python 2 differences, but also for new features that came in in your C Python versions, so your code will actually adapt to the C Python version you're currently building it against, and if it's a new version, it's going to use that, otherwise it will just fall back to some implementation, and it might be slower, but it's still going to work. The next thing we do is we copy all the original source code into C code comments, so you can actually, when you read the C file, you can actually find your entire source code in there, which makes it easier to figure out what's happening when, and which source code line contributed to what, you know, huge amount of C code, and in any case, it's definitely, it's definitely a lot of code that you really don't want to write yourself, and for which you want to use code generator. So we write C, so you don't have to, okay? Building the whole thing, I already showed this Cyton iscript, which is really nice, but it's kind of the thing you would use for local development, and when you want to distribute your code, you'd use distutils, and you'd let distutils build the extension module for you so that you want to install your package, have no problems in building it locally and, you know, getting it all set up. We also have a helper for that, so basically what you do is you would import the CytonS function, after which the CytonS script is modeled, and then you would just say, Cytonize some source files, and it will build the extension, so the metadata, the extension metadata for you, and it will compile that for you, okay? For the more involved cases, where you want to build against external libraries, for example, so you have, you know, linker set up and maybe include half and stuff like that, you can also use this, the standard distutils way of setting up the build metadata through the extension class, you can configure it, just look it up in the distutils config, configure it yourself and then pass that into CytonS, okay? Then it's just going to do the right thing for you, okay? So, I already mentioned that the code CytonGenerate is very portable, actually, so it generates C code that compiles with all major CNC platform compilers, on all major platforms, so Linux, Mac, Windows, and some others, well, less often tested, but in general work. And in all Cyton versions from 2.6 through 3.5, which is currently not released, but, you know, it's on the way. Partly also in PyPy, so PyPy is a port for the C API, and we have a lot of special cases in the CytonGenerate C code, which say, if compiling for PyPy, then do this, otherwise do something else, which help in, A, making it work at all in PyPy, and sometimes making it less inefficient. So, the language syntax of Cyton itself follows Python 2.7, with lots of feature back ports, whatever comes in from Python 3. We just adopt it as long as it doesn't conflict with any Python 2 features. And if you want to have actual Python 3 syntax, you can use the usual future imports, from future import, absolute imports, print function, this kind of stuff. Or you can just say leave a compiler directive at the top of your source file, which says language level 3, and then you get Python 3 semantics for your file. For language features, Python language features. So, one of the tests we are on in our continuous integration test is the C Python regression test suite of Python 2.7 and 3.5, and we're at about 98% success, so it's pretty close to full Python language coverage. What's definitely in there is everything you would normally know, like classes, control structures, loops, tri-finally, so exception handling, comprehensions, we have support for async and await, which will be new in Python 3.5, star unpacking, same thing, so this extended star unpacking, which is new in Python 3.5. So that will all be in the next Cyton release, which is currently in beta, so we have published a beta release yesterday. So the first beta is out for version 23 of Cyton, and we have support for profiling, tracing, coverage analysis, like lots of tools that help you understand, analyze your code, and make it better. Okay, a couple of deviations. We don't have frames inside of functions. Who here knows what frames are? Okay, that's definitely a clear minority. So if you never came in touch with frames, then this one usually hurt you. You'll notice frames in a couple of places, and that's tracebacks, because tracebacks are actually built on frames, but we support those, okay, so you'll get a nice traceback from your Python code if there's an exception being raised. That's all supported. And then obviously, yeah, there are bugs in every piece of software, and you can find them in the backtrack there. As for speed, so Cyton generates actually very efficiency code. There are many static and optimistic optimizations that it applies to your code. It generates optimized code for standard types and for many built-ins that you would use, and has static type inference within functions, okay? Okay, so it kind of can decide what you do and when, and it understands your code in a way that's... So it has better static knowledge about your code than the normal CPython interpreter would. One way we measure speed is with the Python benchmark suite. There's a bunch of software packages that were assembled to measure the speed of the performance of Python programs. It's actually a real-world code, including stuff like Django, some template engines, what else is in there? Some computational little benchmarks on very different... So lots of different stuff that you can run to see how fast your interpreter is. And we run that in Cyton, and the problem with that is that the applications that have been tested there, they are a bit outdated. Many of them were written in a pre-Python 2.6 area, and that means they don't use many of the more recent Python features. So it's a nice benchmark, but it's not very telling for any recent application that you would write. So in that benchmark suite, we're currently about 1.3 to three times, depending on the benchmark, faster than Python 3.5, and hand tuning those benchmarks a little to make Cyton generate better code for them, gives you something like four times to 50 times. And how do you do that? Well, static type declarations. So Cyton is actually a language that supports optional static typing in addition to being Python. And so in your code, you can say... You can declare variables, for example, having a certain type, and that is an extended syntax, which says cdef some type x for variable declaration, so it's extensions to Python language, which then are only understood by the Cyton compiler, not by Python. What does that do? Well, once Cyton sees these declarations, it can drop your code from object operations into plain C code. So when it knows that a variable has a C integer type, for example, it can say, okay, adding a value to the variable can be done in C, don't need to wrap any object, so that's faster. And the nice thing is that it's optional typing, so you can employ these type declarations exactly where performance matters, and that's the way we made these benchmarks way faster than what you'd get from simply compiling the Python code. Especially from computational loops, usually the speedup you get is somewhere between 100,000 times. Okay, the actual topic of today's talk is talking to native code. So I won't talk much about speedups, I'll talk more about how do you connect to external libraries, how do you talk to them, how do you see from Python? So since Cyton code translates to C code, talking to native code, external code, external code written in C or C++ is actually completely natural, it's totally straightforward, and so that makes Cyton a foreign function in the face, but the cruel way. You can think of it as Python with C data types, and the nice thing that the compiler gives you is that these types converge back and forth from Python objects automatically for you. Cyton understands pretty much all C data types, and most of C++ own data types, and it includes C functions, pointers, structs, arrays, and so on, and C++ classes, templates, operator overloading, and so on and so forth. Okay, and that's the end of the first part, and here's the demo. First of all, C features. Who does not know the iPython notebook or the Jupyter notebook? Okay, everyone does, that's cool. So in case there's someone here who's been working on it, so who's been contributing to Codebase, thank you everyone, it's a great tool, it's wonderful. And has Python support, so all you really have to do is you open a Python notebook, and then you say, load xcython, which enables Python support. And currently, as you can see, I'm using Python 3.5 and Python 23.1, so use release. And so here's an example. This is what a Python code looks like. So I start my cell here with double-percent-cython, which tells the Jupyter notebook that here's the Python cell, and please compile this for me and then run it. And I can compile it, and just works. And what I did here is I defined one little Python function. I defined a Python function with a typed argument, so the argument that's been accepted must be a C int. That's what I declared here. Then I have one function here, which is a C def function, which translates into a plain C function. And then the last thing I have here is a Cp def function, which is kind of a C function with a Python wrapper. So the Python wrapper is automatically generated for you. And now I can call my Python function, which is what I want. I can call the typed Python function, which does the same thing, with a difference that if I pass in something else, then it ints. Maybe C, for example, then I get a type error. And I get it on entry, so it's automatically checked for me when calling the function. Okay, then I can try to call the C function, and I'll see if there's a name error. So the C function doesn't actually appear at the module interface, because it's a C function, right? It's not a Python function. But what I can call is the Cp def function, which then internally calls the C function, and that's, as I said, a C function with a wrapper around it, so the wrapper becomes visible through Python code. Okay, a bit of function jit-compiling. We have a decorator called siphon-compile. So this is not a siphon cell, it's a Python cell, and you can import the siphon module and decorate a Python function with a decorator, and then run that, and when you call the function, it executes, and the result is down here, it executes the compiled function for you, and you can see that because Cog is not just a Python function, it's a runtime-compiled function, okay? So that's how you can do runtime compilation from within Python. Okay, calling C functions. As I said, it's entirely natural for siphon code to talk to C libraries, and the way this works is, well, we have a standard, a couple of standard declarations that we ship that includes a large part of libc, libc++, the C Python C API, so lots of C functions that you can simply import use, done. So the way it works in siphon is you say from libc, so the libc library, there's a module in there called math, so the math header, from libc, c import math, and then you have access to the libc math functions, and you can say math sign of pi by two, run that, and it'll give you one, okay? One nice thing you can see here is I didn't just say, siphon, I said, siphon minus a, and minus a means annotate, and what I get out of it is an annotated HTML version of my source code, and that allows me to see what siphon made of my source code, okay, because now here you can see it's exactly the source code as you saw above, but when I click on the line with a plus here in front, it'll show me the C code that generated for it, okay? So I can see, so what it does here is it does the computation pi by two, calls the C sign function on it, then calls the C API function on it, which converts the double C double results to a Python float object, and then it calls a helper function which prints the result, because that's what it says here, prints the result of sign, okay? The reason why it's a bit yellow compared to the other lines is that there's interaction with Python going on, so there's actual Python object operations here. As you can see, the result is calculated in C, but then afterwards I create a Python object from it and call the print function, and that's everything that adds up to the darkness of the yellow line here, so it's pretty dark, meaning quite an intensive Python operation going on, okay? Memory allocation, same thing. We have the standard lib headers, readily declared for you, so you can just say from libc standard libc import method and free, and then you can call method, have to check the results, obviously, if it's now raised a memory error, if it is, in other words, in this case, we just free it, okay? Works for me, okay? What you should do though, especially for little amounts of memory that you need, is use the C Python heat memory management, because that's pre-adocated memory, so it doesn't have to talk to the operating system, and it's faster for small memory areas. So what I'm doing here is I replaced Melloc by PyMelloc, same thing, check the results, and here I'm assigning some values to the memory buffer that are allocated, and then do a little computation, print the results, and free the memory, okay? Important here, try finally, because even if something goes wrong here, I want to be sure that I free my memory. Up, execute it, it gives me four, so it's adding up one and three, okay? Okay, one nice little feature that came in in 0.22, I think, so last release, Python version, is automatic wrapping of C functions. So here I'm C importing the math and standard lip modules from lip C, and take the sine and A to I function, that's actual C functions, and assign them to Python variables. Okay, which makes them objects. So I can execute that, and you can look up what it does. So here it actually calls a helper function that it generated for me, which takes the C sine function and wraps it in a Python callable for me, okay? And then it just adds it to the global module dictionary so that it becomes available to Python code. And then I can just call it from Python code, okay? So pi sine of half is apparently this, and I can take the A to I function, pass in the byte string, and it's one, two, three, okay, as a number. Okay, it's actually as simple as that. Okay, external libraries, here's a bit more often involved example, because what, so talking to lip C is nice, but it doesn't usually get you very far, okay? So there are lots of helper functions in lip C, but what you normally do is, so you want to talk to your own code, okay? So you have some C code lying around, or an external library that you want to wrap that you want to make available for Python, use it from your Python code, and so this is how this works. So I chose Lua, because I've also written a Lua wrapper for Python, which is called Luka. And so I happily copied some code from it for this presentation. So there's a Lua header file, and I copied some declarations from it, which is a state, so that holds the runtime state of the Lua runtime. I can create a new Lua runtime, and there's a function for closing the Lua runtime, so cleaning up, and I can load a code into the runtime executed, and then so Lua is a stack machine, and so I have some function to operate with a stack, and to call a function on it, and to stuff it. So that's all the declarations that are needed. Important bit here is it's not complete, so the Lua header file is definitely way larger than what I've copied here, but that's all I needed, okay, and that's all that Python needs to know. And once that's declared, I can actually just use it, okay? So here's a function, run Lua, Python function, and I pass some Lua code in there, and then first thing I do, I make sure it's a byte string, so I convert it to a gf8 of its unicode, create a new Lua runtime, raise an error if that fails, load the code into Lua, raise a syntax error if that fails because that's activated in the Lua parser, and then get a result from that, call it, execute the Lua code, if that fails, I just raise the runtime error, saying your code was broken, please try again. And then here's a very simple return value conversion, so I'm expecting a Lua number back, raise an exception if it happens to be not a number, otherwise I just convert it, so there's a helper function for converting the Lua number back to a C number, and then I return it. Okay, so that's all I need to execute Lua code in Python, or in Python in this case, to make it available for calls from Python. Again, try finally, so there's a bit of cleanup. After creating the runtime, the Lua runtime, I need to make sure that I probably clean up afterwards, regardless of what happens, so any exceptions shouldn't, you know, leave the memory leak or something, or leave the runtime in memory, so I'll just clean everything up afterwards, and then I'm done. Okay, so one thing I'm doing here, now I take some Lua code and pass it into the function, and at the end I just say, so I create a function, Lua function, and then I call it in the end and return the result. Did I execute that? I didn't, okay, I'm executing that, I'm executing that, and it gives me the 10th Fibonacci number apparently, so it's a recursive version of Fibonacci, and I can benchmark that, that's gonna give me about five milliseconds, and the thing I didn't mention is, why does this actually work, right? So I'm using Lua code, but I'm not linking against it, yes I am, so what I'm saying about here is, when this compiles in this details, and I'm creating my shared library for Python, please use this include here to find the header file, use these libraries to link against, okay, so I'm specifying that directly in my module, and now I linked against Lua 5.1, I know if I replace that by Lua.jit, which I also have installed, all I have to do is replace the include directory, so for fun I'm just gonna do that directly in my source code, I could also do that in my setup file, but since I have an iPad and notebook, I'm just gonna do it in place, compile again, run the code again, same result, and it should be a bit faster now, yeah, it's about twice as fast, okay, just replacing the Lua implementation. Okay, what else do we have? A little bit of nice syntax in C arrays, so C arrays are actually quite clumsy in C because they're basically just pointers, and sometimes C knows the length but doesn't really care about them, so Python knows the length of a C array, and it has better ways to deal with them, so what I'm doing here is I'm creating two C arrays of integers length 10, A and B, and then I'm assigning to a slice, so I'm saying I have five values on the right, and I'm assigning that to a partial slice of the C array, then one nice feature is I can assign by value, so C doesn't support that, when you assign an array to another array variable, it's just gonna assign the pointers, or it's actually gonna give you an error and say you can't assign arrays, so Python does a array copying for you, it's assigned by value, very nice feature, and then I'm just assigning to the other half of the array a couple of more numbers, I can just iterate over the array by slicing it, I could iterate over the whole array by just saying four I and B, here I'm just interested in the first three numbers, so I'm iterating over those, printing those, I just want in this case, and then I'm returning B, so what you see here, so I created a function for it, okay, so that's all I do, and you see the iteration actually uses a C loop over the array, then we have here the assignment, which just translates to a mem copy, pretty straightforward code, the partial assignment here translates to a temporary array and a mem copy, so this is all stuff that Python does behind your back and it just does it right, and then when I call my function, you see the iteration here, two, three, four, so iterating over the first three values, and then when I return in the array, it automatically translates to Python this and I get the mapped Python object as a return result, okay? Okay, I'm going to split that and I'm just going to show you a bit of C++, because you may not like C++ and there are lots of people who don't like C++, but it's actually really nice when used from Python, and that's the only way I would ever use it, okay? So same thing here, load like Python, same Python version I'm using, and now first I have to tell Python that this is supposed to be compiled in C++ mode, which enables all sorts of C++ features, meaning operator overloading, meaning class support, templates, and so on, and as I said, we ship declarations with Cython that includes libc++, including, for example, what I'm using here in the vector class, so I can just say from libccp, ccp vector, c import the vector class, and so here I'm defining the Python function with a stack-allocated C++ vector object, and then I can just insert a value, there's a pushback method, whoever came up with that name, on a C++ vector class, and then return it. What do you think that returns? Someone saying stack fault down there? Nope, because that's not what it does. It returns a list, okay? So that's a say for mapping from a C++ vector to Python this, so whenever you make Cython convert a C++ vector or a C++ list for you, any container knows, it'll just convert you to a list for you, or to a C++ map to Python dates, and so on. So you get the obvious representation in Python as a default, okay? Another little example, a vector, so the nice thing here is that Cython does the automatic memory management for you, so this is a stack-allocated C++ vector, meaning it goes out of scope and dies, okay? All handled. You can do the same thing with an extension type, so this is the way Cython defines extension types. Anyone implemented an extension type with a C API? Okay, a couple of people. What you do basically is you take an existing extension type, copy over all that it has, because there's a model C code, it's all struct definitions and stuff, and then you adapt it for your needs, because you're never going to infer them the way this works from the documentation, okay, so you just copy it over, adapt it. In Cython, you don't need to do that, okay? You just say, see Dev class, and it's done, okay? You have an extension type. Here, I'm adding a field, so an attribute to the extension type, to the object in this case, which is a C++ vector, and that makes, and that ties the memory management for the C++ type to the lifetime of the Python object. So when the object is created, it creates a C++ vector for you, and when the object dies, it will clean up for you, okay? Okay, so there's an add method, which is called pushback and representation, and so I can execute that, and then the wrapper is actually just implemented as wrapper of values of the C++ vector, and when I execute this, as you can see, the representation of the class becomes a list, because that's the representation of the C++ vector converted to Python object, okay? So it just uses wrapper of list, and that's what you get. Okay, I'll just leave it at that, and continue that over here. It's a conclusion. Cython is a tool for translated Python codes to efficiency, and it's very easy to interface with external native code, as you've seen, you just write the code, and it just works. You can use it to speed up existing Python modules by concentrating on the optimization, rather than a complete rewrite, which you would have to do, write it in C, avoid writing C, just do it in Cython. It allows you to write C extensions for C Python in actual Python syntax, in using Python semantics, using Python language features, so it allows you to write C code without actually writing any C code at all, and you can use it to wrap C++ or C or C++ libraries in Python syntax just by using objects, doesn't matter if it's Python object or C++ object, it looks all the same, it feels the same, okay? Thanks. Thank you, Stefan. We have time for one question, on my veto, short questions. Are there any questions? Down here, yeah, okay, anyway. Oh, I was wondering when calling CRC++ code from Cython, what is the developers of support for the new standards of C and C++? For what, for the new standards of C++? C++ 11, so on and so forth. It depends on your C++ compiler, so Cython doesn't really care. I mean, it can be C++ 11, can be C++ 19 or whatever there is, so it just depends on your C++ compiler. So the question was, does it generate C code or C++ code? It depends, so it generates C code for almost everything, except when you interact with C++ objects, in which case you get C++ code for it, okay? So it knows about operators, for example, if the operator plus will translate into HB, so you get C++ for code for everything that's C++ usage in your Python code. Okay, thank you. Thanks. Once again.