 So, hello everybody. Let's welcome Francesco, who came the long way from Madrid to present this talk about different approaches of extending Python. Thank you. Hello, can you hear me? Hello, can you hear me? Okay. I'm Francesco Fernández. I'm from Madrid. I work as a software engineer in B-Code. He's a startup. We are working creating a dependency manager for C++. I also run the C C++ user group in Madrid and also Neo4j user group there. I'm very glad to be here. Thank you for coming. And today I'm going to talk about extending Python, the different approaches that we have to extend Python. Here's the roadmap. In first place I will talk about the motivation, why we have to write in other languages. Then I'll explain some basic concepts that we have to know to understand how this extension works. Then I'll explain a bit about native extension that C Python give us. Then I'll explain a bit about C types. Then about CFFI, that is an external library. And finally, I'll explain you my personal thoughts about the different approaches. I would like to warn you that there is a huge topic. There is a lot of things to explain. There is a lot of things to study. There is a lot of tiny details. Every system operating system treats cell libraries in one way or another. So, time is what it is. So, this talk is intended to be an introduction to this topic. So, I'll try to explain the basic concepts. Probably the examples that I'll give you there will be useless. But I want to explain to you the basic concept that if you have to apply these mechanisms in the future, you will have the tools. So, this is my intention here. And I also am a new user. So, I don't have the chance to test my examples in Windows. So, I don't know if they work or not. And I use mainly the C Python implementation to test all my this code. So, okay. What is the motivation? Why should we care about writing C or even other languages like Fortran? Why should we want to call these languages from Python? In C we have to deal with memory. We have to treat with some things like pointers that could be hard to treat with. Why should we care about writing in C? Well, I think that there are mainly three reasons. In first place, we have the speed. Probably we have some Python program that has a bottleneck in some part. Let's say that it has some part that has a very intensive calculation that takes a lot of time. Probably we want to write this chunk of code into C. So, we will improve our performance. Other reason could be using legacy code. Let's say that I have a very good writing library in C or C++ or Fortran. And I want to integrate with my new system writing in Python. This is another reason that we can use these mechanisms. And finally, integration. There are other libraries like, for example, I'm seeing a sticker of LiveGit. We can integrate LiveGit using extensions to use this library through Python. And another reason could be that this is in the other hand probably we want to integrate a Python interpreter in an existing program writing in C or C++. But this is out of the scope of this talk. But this could be other use of using Python C API. Okay. Now I'll explain to you some concepts that we have to know about artifacts in binary libraries. We have, in one way, static libraries. On the other hand, we have shared libraries. I'll explain to you the difference between these two kind of libraries. Okay. A static library is just a bunch of object files. As we can see here, we have a static library. And it's just two object files, A.0 and B.0, some met information. And every time that linker links this library with a binary, this new binary has a copy of this object. So it seems that in this scope, in this context, we don't want to compile a new Python interpreter with this extension. So it seems that a static library doesn't work well with these mechanisms. On the other hand, we have shared libraries. Shared libraries work in another way. Operating systems load a shared library once in memory. And every program that uses that library uses the same chunk of memory. Every program has seen some copy of variables, but they use the same copy in memory. And we can load, dynamically during runtime, this kind of library. So it seems that shared libraries is the good way to extend Python. Okay. Let's start with the three mechanisms. In first place, we have native extensions. C-Python gave us access to the CAPI. And CAPI is a Python API that defines a set of functions, macros and variables that provide us access to mocks of the aspects that we have to access to Python runtime. So we can extend how Python works using C code. Okay. Well, let's do a hello world. I thought that making a hello world using native extension couldn't be very, very useful. So I've been thinking hard, hard into a problem. And I want to translate this very, very hard code that do an addition of two numbers to a native extension. Okay. We have a module with a function that takes two arguments, A and B, and returns the addition. Very simple. You can imagine another hard function here, another static function or something that takes too much time. We can translate into C. And a native extension looks like this. Can you look, can you see it? Well, I'll go into detail in the slides. But the aspect is that we have our function and then we define our module in C code. Okay. So in first place, we include our Python API or Python header. So we can have access to all of the things of the functions, macros and variables that Python API gave us. So notice that if we want to use some system includes, we have to include after Python.h because Python API redefines some system stuff. So if we want to use Python system includes, we have to include after including Python API. Here we have our function. The signature is it returns a pointer to a pi object. A pi object as you know is the main object of C Python implementation. And then it takes two arguments. In first place, by convention it takes a pointer to self. Every function in native extensions takes a pointer to self as first argument. And then a pointer to a top of arguments. Okay. Then we declare three pointers, A, B and result. Then we get these two arguments. As I told you, ARX is a top. So I use this macro defined by Python API and I get the first argument and then the second one. And then I use another function given by Python API to add this to add A and B. And I get the result and I return the result. Very simple. Let's notice that this is an toy example. If I pass only one argument, it will probably break because I don't check if it has two arguments or whatever. Then we define the public method that my module will have. And this is a list with the name of my method, a pointer to my function, some flags, and my docker stream. And finally I define the first method that will be called, first function sorry, that will be called the first time that I import this module. And it just init my module. It gives a name, my public functions, and a docker stream for this module. Also Python gave us some tools to deal with compilation of my native extension. We have extension class that if we gave a name and a list of file, it will take care about compiling my code depending on the platform. We don't have to take care about, okay, I have to compile with position independent code because it has a library, it will take care about it. So I compile it and we can try my fancy module, I import it and I call my function, okay, take the addition of my two numbers. But I like to go deeper and think about how this works at sea level. Okay, so I did a bit of research and I finally ended in this man page. The man page of DL open. This is a system call defined in Linux and VSD. I don't know if Windows is available. And this function, what it does is, given a path and some flags, it loads a share library into memory and the number of references for this share library. So I grabbed the Python code and I ended up in this file. We can see here that we have a call to DL open with a path name to my module that I'm loading in case that it's a native extension. And if everything is okay, it will store a new reference to my handle. And this is another system called DL sim that looks for a symbol in my share library. With this call, it's looking for the first function that is called the first time that a module is imported. Do you remember this function? Okay, it called this function that registered my module into my dictionary of important modules. As far as you know, we have to deal with memory in C. We have to, if we allocate some memory, we have to free after using it. If we don't need to do it, we'll have memory leaks. But we are lucky. In Python, we have an automatic reverse collector. It uses the reference counting algorithm. So we can have access to this algorithm using these two macros, PyInGref and PyDecref. So if we are dealing with Python objects, we can use it and it will take care about memory. It also gives us a cycle detector because if you know how reference counting works, basically it takes an account of how many external references have an object. And when this counter comes to zero, it free the object. Basically, it's a very, very basic introduction. But we can have cycles. It means that we can have reference to the object itself. So Python garbage collector has also a cycle detector. But what happens when we have to deal with errors? In C, we don't have exceptions as far as you know. So by convention, we have to return null pointers convention and register my error into a variable that Python interpreter has. In code, it looks like that. We register our exception with some type. In that case, it's a base exception with some message and we return null. Well, also there are differences between all of these codes from Python 2. If we want to work with Python 3, there are a lot of differences between the API. But also there are differences between how can define the modules. In Python 2, as you have seen, we have to call a method with some arguments. Depending on position, it has some semantics. In Python 3, we have to define this extract with a lot of more information. We can define more things. But basically, we give an end to a doc stream and a reference to our public functions. And we create our module with this structure. This is all about native extension. Okay, now we'll study C-types. C-type is an advanced foreign function interface for Python. It allows us to create to call function from certain libraries and also allows us to create access and manipulate C-types, C-data types. Okay, here we have a list from the official documentation and here we have a relation between all the types. But basically what Python does in underlying is Python has a mechanism for something like flags. You translate from these flags to C-data types. Let's say, for example, an integer is flagged as i. So with some mechanism, I translate my pi object to an integer. Basically, what is C-type does? Then we can define also a structure. We have to enter it from a structure that is defined in C-types and we have to declare the field of the structure with our C-types data types. We can use previously defined structure as fields. I'll give you an example. I have a Fibonacci recursive algorithm implemented in C as a shared library and I want to map as Python code. So I can call this algorithm using Python. I will also implement the same algorithm using Python and I will measure the differences between these two implementations. Here we have our algorithm, nothing new to see here. Here is the thing. First of all, I import C-types. Then I load my shared library into memory and the line is opening using dlopen and I have a handle to my library that is storing my variable libfib and then I define a function C-typefib that takes an argument. Basically, what this function does, it is a wrapper that calls my Fibonacci function implemented in C. I have a reference to my shared library, load it into memory, libfib and I call my function implemented in C fib and as an argument it takes an integer type of C. I do the cast to an integer and below we have the Python implementation. So I measured the times using Python. It's not a very, very good benchmark but we can show the differences using this implementation. We have 63.8 microseconds per loop while when we're using Python implementation we get 3.62 milliseconds. Okay, well as expected. But as I told you at the beginning one of the reasons for using this kind of mechanism could be using legacy code. So I thought, okay, could we use Fortran as an example and we can take some code randomly getting into the internet and try to wrap using Python. So I went to Github, I went to trending repositories, Fortran there are a lot of Fortran code nowadays it's just a joke and get this Fortran lib that is just a bunch of mathematical stuff and you want to wrap using C-types. Here we have our Fortran function that returns a real that is mapped to a float in C if I'm wrong. It takes a list of reals and it returns the mean of this list. Nothing. So I compiled it as a library using G Fortran and I listed the ABI, the symbols that are public and are exported. I cannot be able to get nicer symbols if someone here knows how to get nicer symbols using G Fortran, I'll urge you. So we're interested in this symbol. So okay, let's see how this works. We load our share library that is called live statistics. This is a share library that is compiled with G Fortran. Then I get the reference to the function that I want to wrap, that in this case is this thing, this long thing. And with C-types I can configure the argument types that a function will take. In that case, as I told you my Fortran function will take a list of reals in C will be a list of a write of floats. So I'm telling here to C-types, okay, this function will take a pointer of an array. I give the dimension of the array in that case and also I configure the result type of this function. And this function will return a float. Then I create an array of two elements of floats and I pass my array to my function and it returns the mean of these two numbers. Pretty straightforward. So okay, let's look into C-types code. I went to the C-types code and I look for DL open again. So here we have for DL open. So the mechanism is mostly the same that is followed using native extensions. We get a reference to our share library and if everything is okay we return a reference to our share library. The concepts are the same, okay? Now we will study CFFI. CFFI is also an advanced Fortran function interface for Python. In fact, both C-types and CFFI use leaf-ffi as library but C-python has its own copy into the distribution because it seems that it's very difficult to install. It's a bit messy but if we are using CFFI we have to install manually. It also allows us to call function from share libraries. Also we can create access manipulate C-types but the difference between C-types and CFFI is that we can work also in API level as we can see it's mostly the same as C-types but the author of CFFI want to get less friction when we are working with Python and C because if we are working with native extension we have to learn the whole, not the whole but a lot of stuff about Python API and it's a lot of things to learn. If we are working with C-types we have to learn the API for example we have to learn how to translate our types from Python how they translate to C-code and we have to learn how can we create an extractor. If I show you we have to inherit from a class it's a bit, we have to learn things, a lot of things. Also we want to use Cython, we have to learn a new language so they are trying to minimize the things that you have to learn to use their API. Their API is a very minimum amount of functions it's also a recommended way to extend PyPy and I'll show you some examples here we have an example from the documentation of CFFI here we will get a reference to printf that is in Lipsy so we are telling CFFI okay I'll use this symbol that is I'll use the function printf with this signature I don't have to tell the whole signature of the function I could get the compiler to infer some things in that case that printf takes a pointer to char with the format of the things that I want to print then I open a share library by convention when we don't pass any argument to they open it opens Lipsy we create an array of char with the world in that case between CFFI and C types here we are writing mostly directly C code we only have to learn that new create new variables in C as C code and then in variable C we have the reference to Lipsy so we can call printf directly with my art okay but it's mostly the same as we have seen with C types and here comes the interesting thing with CFFI that we can work at API level what I mean working at API level okay we can use C code directly written into python in that case it works okay in first place we define the things that we will use in that case I am defining the signature of my Fibonacci function okay it will return an integer it takes an integer with name n and then I call the function verify with my C code here we have my recursive algorithm writing in C and CFFI so I don't have to take care about writing this into a C or file compile it and load it as a share library into my code so I don't have to take care as I have to do with C types to compile my code separately and then open it CFFI does it automatically for me and then I call my function okay in some scenarios it will be easier than doing it with C types into different phases what happens if we pass for example an string we have defined that Fib will take an integer so it should break we get a type error when we try to call Fibonacci with a string we get a type safe in that case do you remember the example of extract that C types we have to learn the API we have to enter it from a structure that is defined in C types it looks like a class in Python but we have to define some fields it's not like our regular classes in Python it's a bit of a mix between two walls in CFFI we can define in that case an extract and also a type and we can access directly from our Python code we can write it into C and then we can access as Python code we don't have to have this mix between the two walls okay so here I'm defining an extract with two fields x and y that are floats so then I can create a pointer to this new type notice that CFFI when we are dealing with a race structure and pointer it allocates the memory for the type that we are creating the pointer and filled with zeros so in that case point dot x and point dot y will have zero values okay and they will assign as Python code the values and we can use over the R code without taking care so if we go deeper into into our into CFFI code we will find again our DL open frame we are using the same mechanism again we are loading a shell library into memory and if everything is okay in that case I create a new object this is how can we create Python objects into using CAPI and I store it and I return my reference to my shell library okay so as far as we can see the mechanisms are mostly the same so okay my conclusions we have three different ways two of them are very very similar if we want to extend how Python works we will use Python API with native extensions if we want to call external code it depends on you you have to think that C types is distributed with standard libraries so you don't have to have external library external dependencies CFFI is an external module but it's more portable because you can use CFFI for example with PyPy so it depends on the context we have seen that three of the approaches applies the same principles they are based on DL open and DL sims operating system calls and we have differences into portability we have seen that there are also differences implementing native extensions in Python 2 and 3 so we are a bit tired if we use C types to some version if we write a native extension and if we use for example CFFI it will be more portable and in my opinion native extensions could be harder because we have to learn how Python works internally it could be great because it's nice to learn how things work internally with C types we have to know very bit about how things work and if we use CFFI also we can have extensions with I don't know 15 lines of code and that's it do you have some questions? there's a microphone over there next to the camera if you have questions I don't have a question as much as a comment if you want to create those native extensions with the C Python API when you use the boost Python project this becomes a lot easier I mean it's like you just write your standard C or C++ code and then you write two lines or like that so you don't have to learn about the internals well but do you have to depend on boost that is a huge library so it depends on your context probably if you have to integrate a lot of C++ code you will have to include boost too but compiling an integrating boost could be a bit difficult I mean it's not too difficult but of course you would also depend on boost but in the end you also get a shared library and then you just open it in Python and stuff like data type conversion is automatically handled for you without any other code visible in Python so this might be a viable option I could be an option but I don't have enough time to explore the whole ecosystem this is the main stream waste but this could be a way too ok thanks a lot Any more questions? Thank you very much Any other thoughts?