 All right, let's welcome our next speaker, Vinayak Mehta. Vinayak is a software engineer and an active open source contributor. Hi, Vinayak. Hi, can you hear me? Yeah, I can hear you. I still remember your talk from the last year about command line applications and interfaces was a really good one. Thanks. Yeah, so today Vinayak is going to talk about writing Python C extensions and cross-platform views. Very nice. So over to you, Vinayak. Good luck. Thanks. So, hey, everyone. I'm Vinayak. And today we'll talk about how we can write Python C extensions and package them into wheels so that they're installable by users on any major operating system straight from PyPI. This is a basic outline of the talk. We'll write a basic extension module using the Python C API. Then see how we can do that using Py�11, which is a package that offers seamless operability between C++ and Python. After that, we'll briefly look into shared libraries and dynamic linking and see how we can build wheels for our extension module and finally automate wheel building using CI build wheel. The goal of this talk is to give you a good starting point for how you can go about building wheels for your extension modules. The day begins at the RECA center, which is a self-directed educational retreat for programmers in New York City. Last year in August, I started my RC batch with a goal of removing a platform-specific dependency from Camelot, which is a Python library that can help you extract tables from PDFs. The dependencies used to convert a PDF to a PNG so that Camelot can do line recognition and identify tables on a PDF page. The problem with this dependency, GoScript, which is a large PDF interpreter written in C, is that it isn't available to install from PyPI. Users need to install it using the system package managers like apt, Pacman, ABU, et cetera. And on Windows, they need to go to the downloads page to get the EXE installer. And then hope that Camelot, which is installed from PyPI, works seamlessly with GoScript. In the past, there have been instances where users would install the incorrect GoScript version for their system architecture, leading to sec faults. Or Camelot would not be able to find the GoScript executable on the path, leading to an import error. This led me to search for a pure Python PDF to PNG converter, which could be installed directly from PyPI. I wasn't able to find one in pure Python, but I found a C++ library called popular that has a CLI utility called PDF to PPM, which could let me do this. I was able to write a wrapper on popular called PDF to PNG, which takes in the path to a single page PDF, converts to a PNG, and then writes it to the given output path. I was also able to build wheels for every major operating system, and then include PDF to PNG as a requirement for Camelot, so that it can be installed directly from PyPI. In the stock, I'll describe all the steps I've followed to package the PDF to PNG's extension module, which depends on the popular shared library. On the side note, if you're interested in other things I've worked on during my RC batch, or what it feels like to do RC virtually from another time zone, you can check out my blog, where I wrote a blog post for every day of my batch. So C Python, the reference implementation of Python. As the name suggests, there's written in C. It exposes a rich API using which we can extend Python's functionality. The main use cases are writing extension modules so that we can use C or C++ modules in the Python interpreter, and embedding Python as a component in a larger C or C++ application. We're going to look at the first use case in this talk. We're going to look at how we can write an example module, which just lets us add two numbers. So we start by creating an example.c, and including the python.h end of five, which will give us all the functions and macro definitions needed to use a Python C API. Since all Python objects are treated the same way by Python in the programming language, they're also represented by a single C type, by C Python, the reference implementation. In C Python, everything is a Py object. This is what a C function to implement or add Python callable looks like. It takes an pointer to the cell Py object and a pointer to the argspy object, which will contain all the arguments that we pass to this function. We then define two long integers, a and b. And since we get all the args that we pass into this function in the argspy object, we can use a Py argpass double function to parse those parameters into individual variables. You can think of it like scanf from C. Here we are parsing two long integers from args and storing their values in a and b. And L is a format unit for the parameter that we want to parse. And there are two of them, LL, because we're parsing two long integers. The string after the colon and after LL is used as a function name in error. And LL is not able to parse our args and returns what? We need to return null to raise an exception and pass it up the call stack. Then we define two Py objects, p, a, and pb. And then use a Py long from long function to convert C longs, a, and b to two long Py objects, p, a, and pb. And finally, create another bi object called r, which stores the result of the bi number add function on p and pb. In the end, we return r, which is the result of the addition. But before we do that, we need to use PyDeck ref to decrease the reference count for p and pb, the two Py objects that we created. This will ensure that p and pb are deallocated by Python's garbage collector. If you're writing an extension module using the Python C API, you need to be aware of Python's reference counting model for memory management so that you do not leak memory. You can check out the resources in the last slide to learn more. After creating our add function, we need to create an array of py method dev structures, which will define a list of functions that are module exports. In that list, we specify the name of the callable that should be available in Python, the C function, a flag that says that C function should be called with self and args, and finally, a function doc string. And after that, we need to create a module definition struct, which holds all information to create a module object. Here, we can add the module name, its doc string, and the list of functions exported by the module. Finally, we need to create a module init function called py init example, which needs to follow the convention py init underscore module name. And call the py module create function and set it with a reference to a module definition struct. And that's most of our C extension code. To package our extension, we'll use setup tools. We start by importing extension from setup tools, passing the name of our module, the bar to our C code, and then call the setup function with this EXT modules list. When we install it using pip, we'll pick it up, use setup tools to build the extension, and install it in our site packages. And if we import it in the Python repl, we can see that pip use setup tools to build a shared library for our module. And voila, we should be able to use the add callable now. We can also store its return value in another Python object, since we returned it from the function. Now let's look at how we can create the same extension module using py by 11. Py by 11 is a lightweight header-only library that exposes C++ types in Python and by subversa. We can use it to easily create extension modules on top of existing C++ code. So this time, we create an example.cpp and include the py by 11 header file instead of the pythons.h form before. We create an add function, which takes in two long integers a and b, and returns the sum, which is also a long integer. And finally, we use the py by 11 underscore module macro to initialize the module. We pass in the module name as the first parameter. And the second parameter, m, defines a py by 11 variable which we can use as a main interface to bind our C++ functions. We then define the module doc string with m.doc and add all the C++ functions that we want to use as Python callables with m.def. The first argument for m.def should be the name of the Python callable. Second argument should be the reference to our C++ function. Third argument should be the function doc string. And the rest of the argument should be the parameter that the Python function accepts, in this case, a and b. And that's most of our C extension code using py by 11. Now we get to a set of dot py, where we again create any XT modules list after importing extension from setup tools. We pass in the name of a module, the part of a C++ source file, part of the py by 11 header files using py by 11 dot get include, and specify the language as C++. In the set of dot py, again, we call the set of dot function with the CXT modules list. And finally, we need to create a pyproject.toml, where we define all our build requirements, so that PIP can install and use py by 11 at build time. We also need to set up tools as a build backend. Finally, when we install our extension module using PIP, which will then use setup tools to build it and install it in the site packages. And if we import it, we can see that it built a shared library out of our extension module and put it in the site packages just like before. And it works like before too. With py by 11, it's easy to write an extension module or wrap an existing C++ library to create a new extension in terms of not having to change existing function parameters and return types. The API to write an extension is also a bit less verbose compared to playing with py objects. With a Python C API, we have two first-person coming parameters, convert them to py objects, and then finally return one too. We also have to raise appropriate exceptions. But py by 11 can automatically take care of all of that for us. In the case where I wanted to wrap the PDF to PPN popular utility, I could just take its C++ code with its set of files identifiers in the main function and turn it into PDF to PNJ.CPP, where I change the name of the main function, pass it into parameters, the parts to the single page PDF and an output PNG, and initialize the module with the py by 11 underscore module macro. And after building it for different operating systems, I could access the C++ convert function in my Python code like this. Let's look at how we can build extension modules that depend on shared libraries. But before we do that, let's briefly look into shared libraries and dynamic linking. To understand shared libraries and dynamic linking, we'll go through an example of how a program might run on Linux. Let's say we have a C program, which prints a string called hello world. When we compile it with GCC to get an executable run it, this is what happens. The program sees that it needs the code for printf from the C standard library called Lipsy. And since it does not have it, it asks the dynamic linker LD for the path where Lipsy is stored. The dynamic linker looks for it, finds it in one of the default parts, and gives it to the program, which can then finally finish executing. Here, Lipsy.so is the shared library, and the whole process of finding it is dynamic linking. Shared libraries are loaded into memory the first time a program that requires them is executed. And if during that time another program that requires them starts executing, it can just reuse a copy that was already loaded into memory. On Linux, shared libraries follow the Lipsy.so naming convention, for example Lipsy, Lipsy, and Lipsy popular. This is a shared library search order on Linux. Ellie first looks into one of the default directories, then looks for all the directories listed in its config file, then looks for any parts on the LD library path environment variable, and so on. You can check out the full list on its main page. On Windows, shared libraries are called DLLs, or dynamic link libraries, and have the .dll extension. The Windows dynamic linker works like this. It does not search for a DLL. If a DLL with the same name is already loaded into memory, or if the DLL is on a list of known DLLs for the current Windows version. Otherwise, it will search for a DLL in this order. It'll look at the directory from which the application was loaded, then the system directory, then the Windows directory, then the current directory, and finally the directory is listed on the path environment variable. Let's try our own simple shared library on Linux. Let's say we have three different files, mod1c, mod2c, and mod3c, which contain three functions that print some strings. We can compile all of them into a shared library called lipfunk.so by passing the shared option to gc. Now let's say we have a program which expects to call these functions one by one at runtime, and then print hello world. We can get an executable for it, and link it to lipfunk.so like this. And when we execute it, we get an error. It says that the dynamic linker can't find any lipfunk.so. Yeah, that's because our current directory where lipfunk.so is located is not one of the default directories the dynamic linker looks into. We can temporarily fix this by adding the current directory to the LD and library path environment variable like we saw in the Linux search order. And yeah, it works. All the strings that we wanted to print have been created to a stadium. Now let's see how we can use that information to package our extension with shared library dependencies into wheels. In part one, we look at building shared libraries that our extension module depends on. You might not need to build one because most C or C++ shared libraries that you want your extension to link with might be directly installable through a system package manager. But there could be some instances. For example, you might want to wrap a large private C or C++ code base, in which case you might need to build a shared library first. The build instructions could also vary wildly for each C or C++ project that you're looking to build. We look at the tools we can use to build our shared library on each of these operating systems. And we'll go with the examples for building popular and linking it with PDF to PNG. Let's start with the Linux. Each Linux Restore has its own package manager and its own latest version of shared libraries. That's a problem because you can easily get version mismatches when you compile an extension with specific versions of some shared libraries on one Linux Restore and then install it on a different Linux Restore which might have different versions of the shared libraries. To solve this problem, the PyPA came up with a subset of shared libraries that are available on all major Linux Restores. If you compile and link your extension with a subset, it is guaranteed to work on many Linux Restores. That's calling it MediLinux. According to the MediLinux 2014 spec, this is the shared library subset you can link with and then stop worrying about these libraries not being present on your user's Linux system. PyPA pushes MediLinux Docker images for different versions and architectures to Quade or DIO. So you can pull the image for the architecture that you want and run a Docker container where you can clear on with building your shared library. In the case of popular, we need to install these packages using yeah, yeah, and then build a lip-populate or SO shared library which is what we need to link with PyPia for PNG. We'll put all these steps in a shell script so that we can reuse them later when we are looking at automating wheel builds. For building a shared library on macOS, we either need a Mac computer which runs macOS or we can use FastMac, which gives us SSH access to a macOS server on GitHub Actions for six hours. It is pretty useful for debugging potential build issues. On macOS, we can install packages using brew and then build a shared library like we did before. And then again, we put all those steps in a shell script which we'll reuse later. For building a CRC++ project on Windows, we need a Windows machine. And we also need to install Visual Studio 2019 Community Edition with the Python native development tools. Windows doesn't come with a system package manager. So one option to install the dependencies that we need is VC package, which is a CRC++ library manager. Here's how it can use VC package to install the shared libraries that we need. And since VC package won't install shared libraries on the default search path where CMake could find them, need to specify the VC package installation root directly to CMake. VC package is also installed by default on the GitHub Actions Windows runner. And this environment variable is also available there, which will be helpful when we are looking at automating wheel builds. Again, we put all those steps in a batch script for reuse later. After installing shared libraries or building them on our own like we just did, we need to make sure our extension is linked with them when it's compiled. We can modify our set up by to do that. If the extension is built on Linux or macOS, we'll create a list of directories to search for popular at link time. We'll also need to create a list of library names that we want to link with. In this case, it's popular because we want to link with lippoppler.so. If the extension is built on Windows, we'll also need to add the VC package installation to directly to the search path. And create a libraries list with the names of all the libraries that we need to link with. There are a lot more libraries in this list on Windows compared to the last slide because we're using VC package, which doesn't install the dependencies in a standard search path. Unlike Linux and macOS, we're going to handle that. And if your C or C++ project needs some header files, you can create a list of all directories where they might be present. These are all the ones that popular needs. And at the end, we also specify the pi by 11 header file directory using the pi by 11 dot get include function. We can then create a list of EXT modules where we pass in the name of our extension module, the part to the C++ source code we want to build, the include directories, the library directories, the libraries themselves, and specify the languages C++. Finally, we call the setup function with that list of modules. And then when you start using PIP, we should be able to use a convert function from our extension module in our Python code. You can also create a deal for our extension module using PIP wheel, which will create a deal in the current working directory. If you look at one of these wheels, you'll see that our extension module has been built and placed into the wheel, but the shared libraries that it depends on aren't there, which means that if we ship it to PIP, users who install it would also need to install the shared libraries using the system package manager. This brings us to our next section on how we can bundle shared libraries inside our wheel so that users don't need to install them separately. On Linux, we can use audit wheel, another tool that PIP has put out. You can check if the many Linux wheels that we built are compliant with the shared libraries subset that we saw earlier, and also bundle shared libraries which are not on the subset into the many Linux wheel. Basically, we can use it on the wheel we saw earlier. We run audit wheel repair with output directory and the wheel we need to repair as inputs. In our case, we temporarily add the popular build directory to LD library path because it's not one of the default directories LD can look into to find lipp popular.so. And if you look at the build again, we can see that all the shared libraries that our extension leads are nicely bundled into the wheel. And their names are also mangled with a hash for uniqueness. On macOS, you can use delegate which works in the same way as audit wheel. We can list all the dependencies for a wheel using delegate list steps and then run delegate wheel with output directory and the wheel that we need to repair as inputs. Again, we add the popular build directory to the dyld library path which is the same as LD library path but for macOS. And again, we can see that all the shared libraries that our extension leads are nicely bundled into the wheel itself. On Windows, because of the dll search order, we could basically place all our dll in the same directory from where our extension loads which is the first path in that search order and we should be good to go. Which means that we can copy over all the dll from the VC package installation directory and specify the package data keyword argument for the setup to start setup function. So we make a minor tweak to our setup.py by adding a copy dll function which finds all the dll in the VC package installation directory and copies them over to the directory where our built extension will be present. If the platform is Windows, we call the copy dll function and add the start dll pattern to the package data like this so that dll files are included in the wheel that we build. And if you look at the wheel, all the dll's are present there and our extension should work. In case where your build process puts dll into another directory, which is not the same directory as your built extension module, you can use a windll.loadlibrary function in your modules in a .py to load the dll before anything else happens. And since Python 3.8, you can also add a dll directory to the dll search path using os.adddll directory. In bundling shared libraries with generic names like we just did, we could run into and contribute to dll help. If you remember the Windows dll search order, the system won't search for a dll if a dll with the same name is already loaded into memory. Which means it's possible that the dll version set is shipped with a generic name, might not play well with dll versions that other wheels ship with a generic name and vice versa. Depending on your particular scenario needs, you can go the dll mangling way where you handle the names of all the dll's like audit builders and then update their import tables to reflect the newly-mangled names. To do this, you can write a script which unpacks your Windows wheel, recursively looks for dll dependencies for the extension module in the VC package installation directory, mangles their names using their share 256 hash, and copies them into the same directory as your extension module. It should then modify the import tables for your extension module and each dll using a library called macho macho mangler. And finally, it should zip everything again into a view. This is what the Windows wheel looks like after those steps. You can see that the dll names are mangled with their share 256 hash and that import tables are also updated with those mangled names. You can also check out del wheel which works like audit wheel and allocate button windows and does these dll managing steps that I just described. Finally, let's look into how we can automate wheelbills using CI build wheel and GitHub actions. CI build wheel is a useful tool which can help us build wheels for all platforms with minimal CI configuration. This is how it works. We create a GitHub workflow which will get run on each push of attack. In the environment variables section, we specify the CI BW build variable with CP3 question mark which should build wheels for all Python three minor versions on Linux, Mac OS and Windows. Except Python 3.5 which we specified in the CI BW our discourse skip variable. We can specify all the commands that should run before building our wheel on Linux using the CI BW before the Linux variable where we pass in the shell script that we created earlier. And then we also specify how our many Linux wheels should be repaired with audit wheel using the CI BW repair wheel command Linux variable which contains the same command that we saw earlier. You can do the same for our Mac OS build script and specify how we want to use Delo K to repair our Mac OS wheels and we do the same for our Windows build script. Then specify how we want to use Del wheel to repair our Windows wheels. We then define a matrix of operating systems we want to build wheels on. Check out our GitHub repository, install Python. On Windows, we also need to set up the developer command prompt for MS Visual C++ which we can do with this GitHub action. We then install CI build wheel and run it like this where we specify wheelhouse as an output directory. We do the same for Linux and Mac OS like this. And in the file step, we upload the build wheels as a build artifact which we can then manually upload to PyPI. Or just add another job which can do that for us automatically. The first step would be to download the build artifact from the last build job and then upload it to PyPI using the PyPI published GitHub action from PyPI. We just need to configure our PyPI token as a secret and we should be good to go. And when the build and upload job finishes, we should be able to see our wheels on PyPI. And that's mostly it. When I was playing around with PyPI 11 in the video that Julia once wrote some years ago, I tweaked it a little bit so that it resembles a PyCon Australia mascot curly boy and added various types of food emojis that it can eat. And I built wheels for Linux and Mac OS so that it can be installed just using PIP. If you're interested in its source and the build process, you can check it out here. Here's all the code that you can check out and learn more. And here are some awesome talks you can watch to learn more about Python's extensions and their distribution. Thank you for listening. You can reach out to me on these links. Thanks a lot, Vinayak. It was really a nice talk. You have two questions. I'll read them out for you. Let me put them on screen as well. So PyVind 11 in first versions 11 of C++. What about the newer versions of C++ like 14 and 17? I think I didn't show the PyVind 11 build.exe class that's in one of their examples that they put out in their GitHub organization which lets you add other C++ versions too. So I can post a link in the room later. Sure, that would be great. Hope that answers the question. The next question is about how to disable shared library, DLLSO, discovery on Windows Linux? Oh, I'm not sure about the question. Why do you want to disable the discovery? Okay, I think this can be taken offline then. So, yeah, that was a really nice one. I see people typing in the metrics channel. So if you have any more questions, please feel free to reach out to Vinayak on the breakout October channel and he would be happy to answer them all. Thanks once again, Vinayak. I really enjoyed. It was a pleasure listening to you once again this year. Thank you. Thank you. Bye. Bye.