 And today I'm going to talk to you about Titan 3 and what's new in it, because we released it. We actually did release it, right? Yeah, so we really, I pushed it out last Monday, and it seems to be a pretty good release, so there are only a couple of bug reports so far. Usually when you put out a new release, especially a new major release, you pretty quickly get a lot of bug reports for it. But as you can see here in the timeline, we had lots of time to prepare this release. We started in 2018, so that's almost five years ago. That was the first Scython 3.0 commit, and to give you an idea, that was the time when Python 2.7 and actually 3.4 were still alive, and Python 3.7.1 was just released, right? So there it is ages ago. Our first alpha was only in 2020, and we already had more than 1,000 commits. That's hundreds of full requests up to then. And back in the final release, finally, then we had more than, well, almost 3,500 commits in it, changes from the 029 release, which we had before, and that's 639 PRs merged for the 3.0 release, so that's really a big release, and it's a big relief to have it out. So does everyone know, who does not know Scython? Who's never used it? Okay, still a couple of people. So to give you an idea, what is Scython at all? So it's a Python compiler, okay? So it translates Python to C and allows you to write C code without actually having to write all this nasty, ugly C code, right? And actually can't stress that enough, because we have a lot of users in large projects that say at the beginning we were kind of torn, we were thinking about you actually going for C, because C is language, C is a thing, we didn't know if we should rely on Scython as a language at the time, and now they say that was such a good idea to do that, because the people contribute to our projects, they don't have to learn C, right? Because most people who use these tools who want to contribute to the projects, they know Python, they don't know C, right? Often not at all, especially not C++, I mean most people these days, either you learn C++ or you probably have never heard of it, and so it allows normal people to contribute Python code to your project. Even though your project depends on C, needs to run at C speed, needs to talk to external C libraries that need to link to Python, and it really keeps it slow, click to see the code level, right? So you click on the file and you see Python code, rather than, you know, it's implemented in Scython, it uses Python syntax, although it eventually runs in C in a native code. To get proper tracebacks, you get this show me the code kind of feeling while having the same code, you know, run in C and be fast. It does support C and C++ data types as type annotations that allows you to, allows the compiler to turn your Python code into actual C code and the fast C code. And since all of this is going into C in the end, you can easily interface with C and C++ libraries by talking to the APIs, by just saying what I'm calling Python function here or C function doesn't really make a syntax difference, it's totally different thing in the end because, you know, one is fast, one goes to Python call overhead, but the new code looks the same, just, you know, here's one or the other interchangeably. And so it interfaces with Python data ecosystem, which allows it to process large data sets from NumPy and friends, you directly get the data unpacked into memory buffer and you can do the computations directly in native data types and C speed, even though it's a Python object that's like a NumPy object, a NumPy array, that provides this data, that you use for parsing the data around. So what's special about Python 3? Well, it's more Python, right? It's much more Python than ever before in Python. So Python used to be a bit of a separate language and now it's really Python with C data types. It uses type annotations, we support many more Python language features in the Python 3 version and you get really high level features like a fast data class, a decorator, a total ordering, standard library features, but we've implemented them in C, so you get them for extension types for C implemented Python classes. And there's a Ufunk decorator, which is also a new thing now. Does everyone, anyone, who knows what a Ufunk is? A NumPy Ufunk. Just a few people. So as a fast way for operating on NumPy arrays for doing element-wise operations, so you can implement a function that operates element-wise on the array and NumPy tries to apply it to the array as fast as possible. Now with the Ufunk decorator and Python, you can also do that at C speed. Yeah, it's faster than ever. So we have more code optimizations for your Python code, and we generate generally better C and C++ code, make use of new features, especially in C++ to make the code faster and more standard compliant. And we support more target environments, which is for nine versions of C Python in Python 0. So that is from... So if the standard needs to support Python 2, we help you with it. We support Python 2.7, and then all the way up from 3.5 up to 3.12, which I think isn't even, so it's going to be released next week or something, right? It's in the making. Okay, and we support... Well, mostly they're still a bit experimental, but PyPy support is pretty good these days. That's experimental support for Gravy and for RadPython, for the limited API, limited C API in C Python, and also for Nogil C Python, so for the Nogil forg of C Python. Well, since this is a major release, what did we break? Well, we wrote the world. Many packages did not specify that they wanted to stick to a zero-point-something version of C Python. So last Monday, the SCI broke, because I'm just saying that you need a newer C Python version than 0.29, whatever, does not prevent you from getting C Python 3 right at the moment, and it's not entirely compatible. It's a new major release. We fixed some things that we considered not well done in the past, and that may hit your code. So anyone who did not specifically say that they wanted a C Python version before 3, so 0.29, probably got their CI broken at that point. And notice that there was a C Python 3 release, so that was very good advertising for us. Lots of people noticed it that way. Okay, what did we break? Well, there's one major thing. Maybe some of you still remember what was the main problem in the Python 2 to Python 3 transition, what broke everyone's code there? A couple of things in TBI, that wasn't that bad. Unicode, right? The Unicode changed. So switching from something that was mixed type between bytes and Unicode, and you got whatever you wanted to restrict the separation between bytes and Unicode. That break actually broke most people's code at the time. It seems a simple and very obvious change, a right thing to do, but it broke lots of parameters. So we have our Unicode moment here. It's called no accept. So the change we had was that C functions now propagate Python exceptions by default. So previously, you often ran in the case that you had some function that needed to be fast, you implemented a C function for it, and it raised Python exceptions internally, because this allows you to mix Python and C freely. So it raised the Python exception, and then there was no way to tell your caller that there was an exception, and it just got swallowed and printed somewhere, but no one noticed. Very bad behavior. You should always propagate exceptions and tell your caller that there was a problem, right? So we changed that, and now C functions propagate exceptions by default. So we hear about exceptions being raised somewhere down in the C function code, and that means that the function signature changes, because previously there were no exceptions, now there are exceptions that's incompatible, and people notice that when they're working with external C libraries, because there are no mismatches between the signatures. Sorry for that. It's easy to fix. You add a no accept marker at the function pointers, and, well, depending on how many function pointers you use, can be some work. Okay. How important is Python? Well, who uses it? Just a couple of projects that use it. There are tons of projects actually that use it. There has been a Python language marker in Type-EI for a while. Many projects already use that, and so there are hundreds of projects that are marked as using one way or another. You probably know some of these, right? Even worse, like you can go to the non-focus stand outside, and you pick any of the stickers and it's most likely a project that has some kind of Python implementation in it. So we're basically the backbone of the Python data, so it's more one of those. NumPy is certainly an important thing in there, but given how important Python is as language in the area, we're somewhere close to NumPy. But numeric computation is not all you can do in Python. There was some guy at the Python U.S. back in 2017 already, so that was a while back. He was working at Instagram, and he gave a talk there where he went, okay, this is one thing we tried. He was actually presenting Python in his talk, and so Instagram, as some of you may know, is a big user of Django, and so they heavily rely on the performance of Django, right? And so one thing he did was he took the URL routing module in Django, which basically receives the URL, the request URL, and dispatches it to the code that responds to the URL request, right? He took that module, just compiled it in Python, did nothing else, right? He just ran Python over it, compiled it, and it ran three times faster. That's pretty good if you depend on the performance of Django, right? And he said, okay, we still don't know what Python is. We have no idea up to this point. We just ran it and our code was faster. That's okay for doing almost nothing, right? I checked before I get into this talk, and Django still does not have binary packages, so if any of you is working on the Django project, ask them if it's not worth providing binary packages that compile a couple of modules, it might really be worth it. Okay. Here's demo. Where's my demo? Oh, it's probably next door. Yeah. Okay. So some of you may have seen something like this before. I keep giving this talk for a couple of years already. I've adapted an office item three, so there is something new in it. I'm going to introduce Scython, going to tell you a bit how the language looks and behaves, and then I'll show you examples of how you can, one example, I'm going to show you how to speed it up and bring some kind of computation to see speed without even Python. This is Jupyter Notebook. Jupyter Notebooks have support for Scython. You can just say load X Scython. You can use load X Scython. And that means that from that point on, you can use a cell magic called percent percent Scython to write cells in Scython to have them compiled for you and have them run in native code. Okay. Here's my environment. I'm using Scython 3.0 final, Python 3.10, NumPy 125, GZC 11. And I should mention that. Well, once you start using Scython, you need a C compiler. Okay. So there's a difference from Python. Python, you just run Python in code and it does everything for you. When you use compilation, there are two steps involved. Once you run Scython over your module, it generates a C file. And that's a module that you normally import. So that's like any other binary module that you have. And in Python, it's generated by Scython, but that doesn't matter. It's just an arbitrary Python binary module. But that's a compilation step involved now. Okay. Little example. So I'm using the math module in Python. Use the sign function in there. And it just calculates sign 5. Fine. Works nicely. And I can do the same thing in Scython by just adding the Scython cell magic. Right? Okay. Takes a bit longer. As I said, there's a compilation step involved. And apparently it takes quite a bit longer for me because it's the first time I'm running this. Interesting. Okay. Doesn't print anything with just, you know, presentation effect. Let's try something else. Let's go for something completely different. Actually, what I want in Scython is not to use the sign function, the Python sign function from the math module because that's Python. I have to call through Python function when all I really want is to do a fast math population. So instead, I can use the libc sign function from the math header. And I do that by importing the math header, so the math functions from libc. So in Scython 3, I can just say from Scython C imports and then I say libc import math. So it gives me access to all the math functions. And then I take the sign function, assign it to variable, and I can call it from Python. What does that work? Well, what Scython does in the back is it sees that the sign function is actually pretty easily wrappable because it just gets a C double in returns a C double. So it's totally compatible to a Python function that eats a float and returns a float. And so it auto wraps it for you. I assign it to Python variable and that just says C function, Python variable needs wrapping, Scython goes, okay, I wrap it for you. Then I get a Python callable function and I can just call it from a Jupyter notebook. So basically what this does is it implements a Python function in the back which takes a C double, internally calls the math sign function and returns the result. So this is the same thing spelled out. Okay. Same thing here. You can call that. Apparently it's the same result, so I'm lucky. One little feature, that's Scython dash A. So that means annotate, right? Annotate my code for me. And that's a nice way for figuring out what's going on in the generated C code. Additionally to compiling my code and providing the functionality in it, it also returns an HTML display of my module which then tells me how much Python interaction that's going on. So the yellow lines tell me that there's some kind of Python API interaction going on, right? So we're talking to the Python C API here. And when I then click on this line, for example, you can see that it's calling Python C API functions, but it's also calling the plain C sign function in there. Right? So the sign call is literally a C function call and then afterwards in order to return the results back to Python and it needs to convert it to Python float object. Okay. Right. And so the thing where, so, I mean, this is pretty boring, right? There is a math module that's a Python sign function in there, so why not just call that? Because it becomes more interesting as soon as you can push more work into the Scython layer, right? So here I'm calculating sign of X squared. And the calculation that I can do is I can do the squaring and the sign calculation at the C layer rather than doing the squaring in Python, creating Python objects, and then calling Python function to call to do the sign calculation and returning to get another Python object. So I'm avoiding Python object creation here. I'm doing kind of unboxed calculations if that's something that means something to you. And the more work I can push into the Scython layer, so in the layer between Python and C, the faster the whole thing gets, right? Because this is translated into C. And you can see here that the squaring really is like X times X and C. So one cool thing that we added in Scython 3 is that these functions, the C implemented functions are really compatible with Python functions, so you can use inspect for example to see what the signature of this Python function is. Okay. I'm going to use them. Okay. Here's a longer example. So everyone likes Texas, right? Who does not like paying Texas? No one, right? Almost no one. Okay. So this is an example I'm already carrying around with me for a while. As you can see the numbers are from 2016. It's been a while, okay. So in Germany there are about 4 million people who have paid regular income. And the average income in Germany is like 44,000 euros or was at the time probably not very much different now. And so I wanted to calculate the average tax rate in Germany, right? It should be possible if you have the data. Sadly there's no official data. You don't get exact numbers. Probably because otherwise it would be too easy to mark people as probably someone who earns that much. So lacking official data I just created some attorney facts here. And I took a lot on the distribution because that's probably more or less what you would expect from an income distribution. Fit it so that the average is somewhere near the official average. And then I just plotted it through a matplotlib to make sure it looks kind of realistic. Maybe I should restart my kernel. Okay. And then, right, there we go. Right. So it's not completely off, right? It's more or less what you would expect from the income distribution. Then I looked up how to calculate the income tax in Germany and what Wikipedia gave me for it was literally an Excel formula, right? I was like, what's Excel? No. But the cool thing is it's easy to translate into Python, right? So you can literally just take the condition and the calculation and then you can write it down in Python code. Okay. So this is how you calculated the tax for a given income in 2016. So it's some kind of consecutive formula that cuts at some points and whatever. Okay. So in order to calculate the average income, you sum it up, sum up all the incomes and divide by the number of incomes. And then the average tax rate is just the sum of the taxes by the sum of the incomes. Okay. So lots of numbers. And then the average tax rate apparently is 24%. Okay. Now, in order to make the following comparable, I'm doing timings for this. And the first thing I'm going to do is I'm going to run the whole thing in Python. I'm going to take it for seconds. And then I have a function to make all of this comparable. And that's going to calculate the ratios for me, right? So speed up of, you know, n times. So that's 1.6 seconds. Okay. So this extra formula is also easy to express in NumPy using slicing. Okay. So you can slice, take a slice of the array, apply the formula to it and then do all four segments, sum up the results and, you know, get the final formula. Gives you more or less the same 24%. And when you run this, NumPy should be a bit faster. Sure way up to speed here. Like I didn't run this. Run again. Okay. So 126 milliseconds and 13 times faster than Python. Okay. Pretty decent. I mean, it's like a whole lot of formula, right? But it's also, it's kind of like, if you know NumPy, it's probably readable, right? Kind of. Okay. So that's 30 times 13 times faster than NumPy. NumPy has a different way to do this. That's a new funk. Or you mentioned that before. So you can just apply this function item-wise to the array. NumPy does this for you. And when you run this, also takes a bit of time and then eventually it's going to take, well, 900 milliseconds. So that's a lot slower than the plain NumPy version, right? Because we executed Python code here. So we executed Python code for each of the items. Okay. Enter sysn. Now, all I'm doing here is I'm taking the plain copy of the Python code above. So this is unchanged Python code. I can literally copy it over. I'm just going to run it in a sysn cell and see what that gives me. Same results, 24%. Okay. And then the ratio is, well, it's 1.3 seconds. So it is visibly faster, about 30% faster than the Python version, but it's way slower than the NumPy version. Even slower than the Ufunk version, right? Kind of sad, but yeah, that's how it is. Okay. We'll see how to make this faster in a second. You can also use the sysn function as Ufunk. Do the same from Python call in NumPy. And then we can use the sysn compiled function as Ufunk. And you can see that that's 560 milliseconds, which is visibly faster than the plain compiled version. So by the way, 2.3 times faster than the plain compiled version. At 60% faster than the Ufunk version, right? Then Python Ufunk version. So that's a visible difference. But it's still slower than NumPy. Okay. So how do we make the sysn version faster? Well, static typing. So I'm going to change the signature of the KKLA text function. I'm going to turn it into a function that can be called directly from C. So without the Python call overhead. So I'm adding a decorator C call, which allows sysn to do C calls inside of the module. And I'm changing the signature by saying, well, my income value is actually C double. My return value is also C double. So please do the calculations in C double as far as you can, right? Rather than a Python float object. And then down here for the calculation, I'm getting rid of the sum function. I'm spelling out, I'm going to use that later again. That's why I'm already spelling it out here. I'm spelling out the calls, the calculation, the adding, basically the reduction of the values. And more straightforward. Okay. Way here. And now when I run this, I'm going to compile it and run it then. And then you can see it goes down to some forms. So I'm going to run it for 40 milliseconds, apparently. So let's see. What's that? Oh, sorry, yeah. 40 milliseconds. And that is already the faster version. So it's three times faster than NumPy, right? Just by adding static types. And it didn't add much, right? It just, you know, changed the signature here a little. The rest is the same. Okay. Let's do some more. I can use the type size and function as a NumPy Ufunk. And some more. That gets it to 280 milliseconds. So that is faster. But not as fast as before. And I can, let's size and generate a Ufunk for me. Using the Ufunk decorator that we added to size three. And that way I can directly call or use it as a Ufunk in NumPy. Just the, you know, as before, double to double. And add a new Ufunk decorator. And then I can call the function on the array, take the sum of the array and divide it by the sum. And that, okay. And, yep. And that gets another bit faster. So 10 milliseconds. 1.4 So, yeah. 1.4. So 40% faster. So, sorry. Three times faster than the type version. Okay. And we can continue. I'm just going to jump all the way down here. You can do parallel looping. So you can, instead of the range loop, you can use a P range loop, which is a parallel open loop. And that gets it on to four milliseconds. And we're now 360 times faster than Python. Okay. There we go.