 Hello everyone and this is one room more that I was expecting for because I know that there's also amazing talks that I would like to go to at this moment So if you want we can all go to the other talks or we can stay here for some fun with C++, of course And I know it's just the last talk from all the tracks So bear with me there will be no weird things here And we will not like be trying to understand obfuscated C++ or something else So it's a really light talk with advanced topics So let's go for it. So well as you heard my name is Christian Currently I'm working as a software engineer in the acute company Which one of the companies that support the acute project maybe you heard about it later I will have one slide to explain the motivation behind this talk and why I'm talking about C++ on Python So first of all don't worry about the examples or the slides or whatever just go to that github repository There is a link to the slides and you can get all the source code is oh my god I feel like a star everyone taking pictures As you can check it out and at the end of the talk also is my github username so you can check it out It's called Unleached CBP Okay, so first of all the acute project in general is a C++ framework huge framework and With starting to do some Python stuff you need to have the mandatory comparison between two languages First of all of all we know that both languages are general purpose There also you have multi-paradigm you can do many things with them And then we start to notice some difference that may I'm pretty sure that most of you are aware of Dynamically type, static-ally type, compile, interpret and then memory management and about also code reliability So I know this is kind of if we have just raise your hand who is currently writing C or C++ code Okay, so I felt your pain too So I know that sometimes C++ could be complicated But it's just want to show you that it's not and you can even make things in Python better So first of all everyone knows that Python is beautiful I remember the first time that I typed I've that on a console when I was studying computer science and it was Everything works, but if you think about it, I mean C++ is not so Different, right? I mean you can write something like maybe you need to run an external Compiler or something, but you achieve the same thing, right? So, okay after all C++ is simple It's not because then you have things like meta programming and you start You know templates over templates and this is one of the reasons that people get scared of C++ But C++ is not so ugly and this is some extreme cases of people with a lot of time that a lot of effort and try to do These complicated things like a compile time if but Let's go about one of the goodies a couple of goodies that C++ and the latest Standards being 11 on Has I mean if you're familiar with COC++ now there is this amazing Auto type that kind of detects the variable that you have which is kind of a similar like a more dynamic Language, but not you can even use Some functions that to get the declarative time of a different variable to the to declare another one and stuff and Then again, we have the for loops that with the time it has been improved I mean this is kind of the more dumb old version that we have in C++ to kind of like go in a vector I know that you can do also with an iterator too, but it's kind of like going element by element But it's too complicated So then you can do more fun things like these and if you remember the author one then you can kind of go Which is not so complicated to understand Also, if you're a fan of Python lambdas, you cannot now so also do it in C++ with the latest standards and so on and so forth And you can have these for each man handling a nice lambda there to just sum it up some values But the most interesting things about C++ are coming in my opinion in C++ 20, which is I don't know I have this relationship with love and hate with C++ 20 There are some things that are really nice and some things that I have no idea why people is doing it But one of the things that I like is this one so you have this Amount sorry this type of ranges that you can use now and if you see the four They kind of like the type of the four there is kind of similar that can you you can have there So after all I mean, it's not so crazy different, right? The same with a more example, I mean, I know that the smart way maybe is the first line That's commented out, but you can write the function maybe like the second line that you see there Which is also quite similar what you can achieve in C++ 20. So after all, I mean, it's just adding some semicolon Santa yet, right? So the idea is that now it's not about like which language is best But how to make both languages to to improve each other and try to collaborate in a more Efficient way, so I want to propose now a new language that is cool. No, no, it's not but I'll just talk it Extending Python with C++ and for this we will go through a couple of example really simple ones and The first of all who here didn't know before the conference that Python was written in C We had some okay. We have some people. Okay So it is something that also was kind of shocking to me because I thought that it was some black magic assembly stuff But it was written in C So if you learn some see you could at the center and then you can understand other things like this joke for example, I Know that is really bad, but it's it's the only joke that I have on the slides But so I'm really sorry, but I need to do it So let's take a look at how we can create an amazing module based in pure C Python Okay, let's go for it. So the first thing that you need to know is that you just need to see a C file and We will go through these examples with the most simple thing that I would think there was a hello world function that just Returns the charge star or string whatever so I hope that everyone can read If see if you don't know see or C++ is kind of that you get the idea. It's kind of a function declaration that returns some Message, which is hello, you're a Python 2019. Okay, so for building a C++. I will just sum out Don't be aware. Don't be afraid Just to show you so that's all the code that we required to create a new Python module Written in C Python which one function that is just printing hello world. It's not so scary, right? I mean I need to confess the first time that I start to play around with this I was just copying all this weird structure that you have around that all these writers are the null null zero and Other things you just copy some stuff and it works So if you are brave enough after this talk just go to the C Python directory and play with it And maybe you can write other sophisticated models with it So this is a simple case right and this is just one example one simple function and we have this so What's the motivation behind this cute thing so Cute is just a C++ framework that is like huge and the main one of the main functionalities was to offer the way of developing user interface graphical user interfaces for people and Also cute implements many of the things that we have for free in Python in C++ Which is like a lot of abstractions for different classes to interact with more For databases or maybe doing command line or scripting alike in C++ And even other advanced things a different way of creating user interfaces actions notification threats all these things So they are providing a lot of goodies to the C++ world and at some point The project was but you know what which I mean this thing need to be maybe available in all the languages So people started to wonder about Python So Maybe you are aware there is already a set of binding which is really old It's called PyQt and this was developed by a different company Some people doing a lot of work there and this was fantastic and in the good project They wanted to kind of have an official Adoption of these things so they decided maybe we can create a new set of bindings and you know just put it there And see it's an open you know market So people can decide which to choose at the moment they are at the same level one is really old And there's a lot of examples and tutorials But the ad the Qt project at that point had a couple of options to do this with this huge C++ framework So they could write raw C Python They could draw a tweak. I mean, maybe you you heard about sick or they could use boost Python So if you take a look at the tweak implementation So with tweak We just just for you to believe me we have here our difficult function to understand and The only things that we need to do for this week is just to create an interface So this interface is in an i-file that also nothing really scary or something at the moment so we have a couple of Instructions to build it. I mean we run tweak and then we compile some stuff and then we can just use a normal Simple import the module and execute it And then we have the message nothing too fancy with boost Python is kind of a similar story We go there again. We have the simple cpp file with our Function there and then we just need to define boost Python module the name The namespace and then we define the function nothing too scary at the moment, right? So this was looking at really nice solution, right? I mean we were achieving the same thing, but the problem lied that many things that we do in C++ need to be specially treated Inside Python for example ownership of objects if you are wrapping an interface That is in C++ who owns the objects some cases will be Python some cases will be C++ So you need to deal with all these things Okay, so the option that at that time the developer had was okay This just one of the generator and we can maybe modify the rubber Generated code. So let's take a look of the code that sweet generates So just for you to have an idea. So this is kind of the code that generates and I will start to go down We're in 2% 3% Okay, we'll just page down four five seven two So I understand it. It's a little bit too much with this small function, right? so that okay, that's all and There is kind of a motivation to say, okay I know that they need to set up many things to get for granted and to automate many process And it's a really smart solution But if you want to modify this file, it will be embeddable. I mean you cannot do it So there was a lot of motivation behind these CPython and sorry and sweet and boost Python Boost I don't have the source code from the boost Python here But the shared library is roughly the same size so you can imagine that there are a lot of black magic there So what they say, okay? They started the development of this new tool based in the boost at the moment. They said, I know it's too heavy Let's write our own thing. Okay, everything was released properly then there was the project and continued the development and The good news was that last year officially was released this new set of bindings of a cute in Python And then their name of a new project was called good for Python. So maybe you heard about this or pie site to It was this all this story behind but okay, this is just a story But what was the important part to at least in my opinion? How do we do it? So this tool that you see here in the center called shevok and to is kind of like the the response for writing our own code generator tool to expose all these huge C++ framework to Python So there is a module inside that it kind of extracts all the API information from cute base is based in clang Of course if you want to do smart things in C++, you need to use clang Then we have a support type system Which is nothing else is an XML file then then you get wait something is being weird An XML file where you can define all these things that I told you about ownership or for example What do we do if a function has a void star argument in Python? We need to have a special or type or treatment or do some casting or something around So these two graphs this type system the information from the framework and generate some Rapper that the one that you saw from from sweet But of course more reduced and it makes more sense and it's more clean in my opinion And then with this we can just compile it and have a the same Python module So this tool is called shevok and the documentation is there and I don't know what is happening I'm doing anything and the slides are coming Okay, so the Japanese Ganges doesn't doesn't mean anything if there we have some Japanese speaker there here It's just three words that make no sense, but let's take a look of the token thing Okay So again, and we will go first for you to believe me that we have here the Implementation and we we are like cheating a little bit because of course is we are working with strings from the standard library Then we define complex header for this Which is just that and for this type system that I just told you and it's nothing else that this So we've had ttml. We said this will be a package called simple It will have a primitive type. There's a string from standard library and a hello function After all the them the compiling and stuff we will get something like this Let's go to bill And it's inside simple and here you have the module wrapper I will zoom out just for you to show that I'm not lying It's long, but it's not so long as the other one. So here you have it I don't wait. I don't expect to you to read the code. I just want to show you the magnitude of the code So at least this is being code general automatically generated calling this C++ function and exposing everything with C Python 2 To be able to use it from Python. So it's way shorter that the option that we had of course this Translate into having a more lightweight shared libraries and you can achieve the same things so there are other nice options out there and For this I kind of tend to recommend people to take a look of all the solutions that you have out there because it's unfair to say Yeah, this is the best solution. No, it's not. I mean, there are many things that you can complement to each other I don't know if you have the chance and or maybe know one of these ones The last one, for example, is the one that the other set of binding called by cute uses which is called zip Pi by the 11th. It also is a really nice project that appear in the I think in a couple of years ago, even a thing that to your pythons ago There was also a talk about pi by 11. So let's take a look of like How do you achieve the same things and With the other options? I think I am good with the time at the moment. So this is the case of pi band 11 So let's open again the file if you see the motivation here is clearly from boost python I mean, it's again a kind of a macro that kind of defines a module Then you have the definition of the function some documentation and so on so forth and Then you can do the same thing. I mean after you compile it and everything you have your simple Main that you can achieve the same so it's kind of like same idea as boost python But they are doing way more things and there is a lot of nice support there that I encourage you to to check it out CFFI well, this is again not really fair comparison because CFFI We had some talks already that we explained about this thing. It's not kind of like a Generating code, but it's just kind of loading something. So we can have this is the one here Well, this is the code that is being generated. Yes, so we go to the simple build and then you have to have like inline raw string that contains the function that we want to expose and Then we just compile this thing in some chair library that we can easily load from Python So if you check here, for example, the the main is the same thing that we were seeing in the other examples Just calling this a nice function in this case We are casting to a string and so on so forth, but it's kind of the same idea But again, I mean, it's not like you can write inline code to kind of read Lots of C++ code to kind of expose in this way The other option that I had there. Ah, of course See if CFFI is kind of focus on C when they support a C 89 and Remember which one all their standard but that for playing with C++ You have a similar idea which is CPP yy Kind of fine, right? CPP over and then it's kind of the same idea So you declare everything like in a string and then you they have this is a declaration that you can use this from the GBL Again same idea and nothing too scary And the last one is kind of similar It was a zip that I show you the one that is using in the PyCute bindings I will just show you This is just there So here you have the simple again same idea But the only difference that they require to define a zip file and it's nothing like Else at this. I mean, it's also is quite simple I mean, you just need to define the module and then you have some include and then the the signature of the function So there are many options out there to achieve the same thing, right? But what's the idea of? Doing this right and I Have been having some conversation with people sometimes that they say I yeah, but I mean I use Python for everything I mean you don't need C++ and they say okay Well, what do you know I do everything with numpy because it's amazing in this fast and it's Python, you know And then it's like yeah, you know, it's written in C No, no, it's not so okay There's even some people that is not aware that many of the popular libraries out there are using C++ or C so in this case you have a case of a numpy I just don't know the source code. The first line is all the Python files that you have That is not test and then you have the C files there Which is 96 which is still a lot of things that they are doing and if you want to maybe refer to a more modern module The case of PyTorch is a little bit more extreme Then if you look there We have no, this is this was this is so here we have 547 C++ files inside the PyTorch because of course it's based in Torch and Torch is C++ So everything is C++ inside. So it's kind of a good nice motivation of Using all these tools to start to write things to improve the Python We are all here some if we know C or C++ we kind of kind of have some responsibility in your hands in our hands to improve things and one of the example that just out of curiosity and please don't blame me about the things that I will show you is that Simple case I was helping someone that I was listing some files in Python and Getting all you know different absolute path and what do you use there? Glob, right how you use glob or if you are more like up-to-date person You use the path leaf and then also you have the glove access there so for this You know the deal I mean writing a glove is Things are kind of simple This is a hard-coded thing to just to play with the recursion thing But you know import glove and then you need to double Stars there and specify if it's recursive. No at the same way which path leaf is kind of the same idea, right and Then I thought I mean let's let's look at the implementation and see if we can kind of do something smarter in C++ So the first time that came to my mind that in C++ 17 Maybe you know there is a new adoption in the C++ standard, which is the module called file system This is based in the boosts file system. So I thought okay. I will just copy paste the hell word example there And I will put the file system and Call to list directories So again same idea. I will just replace the all the simple and hello And the only difference is this if there is any CPython core developer in the audience Please forgive my CPython, but this is just to show how simple we can achieve things. Of course It's just and say there's memory leaks there because when you append that Increase the ref counts and I am not taking care of anything But just taking one argument which is the recursive to see true or false and create an empty list And then just doing some dumb appends So the magic here from the C++ site is that I'm using the file system module I am using C++ 17 and there is luckily some directory iterator that you can kind of do recursively or not and That's it. I mean if you don't know CPython You just into this is just a function that will take here some arguments We create an empty list and then if it's recursive enough We are appending well, you know how it works some stuff and then we are returning this list nothing else So I thought okay. Let's see how this thing was working and then I wrote a really really simple Bash script because yeah, I was growing up with bash and I thought I would just have this scenario when I have 1000 directories with 1000 files inside of each empty files. I'm just to list them So I just create a shell script to measure this as you can see here the love the non-recursive option Which is just listing the directories inside this fake environment is point oh five I was expecting love to take so long But I think that it should be something about 20-something seconds I mean in the meantime I can tell you that I was aware afraid of maybe I was using user being time and maybe it's not the Proper way of doing it. So okay, there you have it recursive 33 seconds We're talking about all these files, then I tried a bad leave again. It was a little bit more And it was a slower than the club file then the other case is the recursive way It will be roughly I guess at the same I Should have have this prepared. Okay. There you have it 20 seconds and then the fast glove implementation that I have there It's less than a second. I mean, I know I'm not taking care of cash or you know The I don't know releasing the gil to use parallel computer or something fancy Just the simplest thing that you can think and then you have it I mean it's and then I thought maybe it's the same thing. I am doing it wrong So let's do the same performance and in Python Which you know the time module or something and after all the tries it was more or less the same results and It's not like I want to say yeah, this should be the new glove But this is how easy you can improve things there And maybe you are thinking in the next numpy the name the next pandas the naked by torture something like that Well, I will not wait for the results there, but it's roughly the same believe me So yeah, this is a summary is it's okay if you cannot read everything because you can check the slides afterwards About the version that you can use the licenses the compatibility with Python here I just want to highlight she bokeh in the thing that I am working with and also see because there is something that is called stable ABI, maybe you heard about it. Yes. Yes stable ABI good So this is just for developers to release wheels that are compatible with Python 3.5 onwards And I don't need to have different wheels with different Python versions So this is really tricky. It's just the way that you create objects are a more dynamic way But it's it's really hard and it's implementing is in those two options. So There you can have information from the pysad project. You can find me here. Also, you can check them The my social networks all the all the information and you can just Type make I add some make files there just for you to know the process of building all this rubber So you can start playing around today about it and just as a PSA that it's always good to support with your local groups So if you at any moment you come to Berlin, I mean we have I think one of the most amazing Python communities out there We have pie ladies Python users Berlin pie Berlin and pie data Berlin. So that's it. Thank you very much. All right Thank you very much. We have time for some questions We have actually five minutes. So a couple of questions for sure And if you don't like your boys in a microphone, maybe you can find me outside I will be here until Friday or we can hang up there a couple of beer We can discuss the truth behind C++ So if you have a question, yes Did you ever look into Performance overhead differences between those options? No, I haven't and I wanted to but then I thought me That's not a half an hour talk. So I was really think considering for the network meetups or conference So to kind of do this kind of thing because I want to measure code length Performance which is also critical I mean, I have noticed some stuff different between the SIP and the Shiboken for example that There was an issue that Shiboken was way slower because we were using Lists in C++ and she was using burglars And then we have for example in some tables or cases like this. They were kind of different But yeah, it's a good question and I will try to maybe next year. I will have one performance. Thank you You can shout Yes, lately there was a something that fixed that because I wanted to implement Breaking the QTBI having for example for Qt settings to have Explicit cast in types and I needed to do those name arguments So now it's implemented and it's out there. So you can do it and now there's an annotation Which is the at so you use the at between the name that you want to use and then you have nine arguments And you can parse it like a normal dictionary for the arcs and So yeah, you can you can ask me live if you want I can show you how to do it Okay, any more question? Do you hear me? Yeah. Yes does the Shiboken uses some Does it have some C make integration? Yes. Yeah, it's a make it's a make and you can make C++ project and use C make integration in your project. Okay. Yes, and does it For example, you you want to override land or some other Python features Can you override it in XML file for bindings? Can you name one? I mean the only override that came to my come to my mind is that for example, you can kind of like remove functions And replace it with a whole new Python equivalent of okay signature wise or functions We can remove arguments sometimes. Yeah items like Square brackets override operators operators. Yes. I mean we have some special treatment for a few operator I don't know how How much you can modify this thing? But at least I know that you can manually modify for example the lower lower operator in C++ and in Python to do different things So you kind of like override the Python's default stuff. Yeah, so for example, I have C++ method size and the other one is operator for accessing it and I want in Python to To just use square brackets for operating as long as you define this operator inside the type of the available It will be possible because underneath it will call this specific Rewriting that you have. Okay. Thank you. You're welcome Any other question? Yeah, the question I Will ask you a question. Okay So how is it is it to C++ class It's kind of the same thing that you saw there You need to just say the class will be an object so you declare you clear a new object and then inside you have Any function that you want to modify if you don't want to modify that you just close the tag and that automatically go to your header See the class take all the signature and expose everything if everything is Normal in the sense of like there is no boy star or weird, you know kind of like the pointers Walking around so yeah And can you then subclass that class from within Python as as you would yes Totally because you are exposing some a new type in Python So for example the simple things that you can have in Qt is a Q widget and then this is C++ You're exposed to Python and the the the recommended way of doing it is to use this as a meta class for your new class in Python that is I don't call my widget or something like that. So it's you can do it Very nice