 Thank you. Hello. Thank you very much for being here. You are really brave. I was expecting an empty room. So I am happy that at least you are curious about this topic. The main reason why I'm giving this talk is because once I searched for a talk on this topic and I couldn't find anything, only blog posts and discussions around all the time. So I hope that you are still interested into this thing. And a bit of a disclaimer. This is a really controversial topic. So you might have your own opinion. I really encourage you to talk to me later on. I want to know what do you think about it because it is a problem that sadly is still not solved. And the other thing, there is a couple of references in the slides that are not true. So I encourage you to find which one are not true. Okay? That I give you some homework, though. So this is how it started. The other day, I wanted to, you know, for some reasons, wanted to get a list of all the files that I had by a backup SSD. And I said, okay, just it was not a large, really, backup. As you can see, there are only two millions of files. So I went there and I said, what do you use to do that for some large file system, right? You can use Glob, right? I guess that everyone here knows Glob. So you do something like this. You get your files. But the problem that I found is that it was a little bit slow, right? But then I said, what is the better approach? What is everyone else encouraging people to use? Pathlif, right? Who uses Pathlif here compared to Glob? Okay. I'm really happy that you're not using Glob. Nothing against it, but it's faster. So I thought, okay, you give it a try. I know it returns an iterator and I'm casting it to list. So of course, there are some overhead, but still it's way faster. So then I thought, is it fast enough? I remember this is only the motivation for the story. And then I thought, hmm, maybe I can use any other language because here can anyone program in more than Python? Maybe other languages? So I thought, which other language do I know that I can use that is popular to beat this? Here we have a clue of the language that I was using. Can you guess which language was it? You can shout if you want. Maybe if I zoom in with the picture a little bit. Of course it was C++ because you have the sky, all the tone of blue, so we all love C++. So I was using C++. So the good thing is that, as you know, you can extend the Python using the C API. Maybe who here has written, even if it's a hello world extension in Python, in T for example. Okay. So people at least are aware of it. You can extend it, right? So we will create a new project there for the extension. I hope that you can read well the thing you can. I want you to answer this to me. What is the tool we use to create a new empty project for C API extensions? Wait. Why nobody knows the tool? Come on. I mean, we have this tool since forever. No? Okay. Of course we use cargo, right? So I have like a proper version. So if you don't believe me, you have there that at least it's the Python package manager. So okay. So we do cargo new and some project, right? Something. And then you go to something, and then you have all the things that you want to write, right? Everyone in following, right? So this is the base thing that you can get, of course, and then you can hear a base structure of thing, and then we can build things with cargo like you usually do with cargo build, and then you have your wheel generated, right? For some random project. Cool. Everyone is following. Perfect. So in C++, we have the file system directory, and this is something that was integrated in a couple of versions ago, and I was curious, like, could I do something that is better with this? So this is the implementation of the function, a globe written in C++. It's really simplistic. Maybe it's really not really optimal or anything like that. As you can see, there's just an option to say, like, it's recursive, it's not recursive, whatever, whatever, and then we have some lists and readers on that. So these are the times that we got, and with FastGlove, I got this. So, of course, what is the outcome of this that I managed to have? Who have thought, like, using a compile language is faster than Python? It's like ground breaking thing, right? I mean, 100% times faster than Glove. So, of course, I was ready for this, right? And going and ready being the number one post, whatever, changing the life of many Python developers. But the package has a couple of questions that we still need to answer. I guess that still you can not, if you see some issues with the wheel there that we have there. So which are these questions, right? What happened if people is not using Python 3.11? Can they use that package? Do they need to provide one per Python version? What about changes in the C API? Can they only reduce the scope to whatever is included in 3.11, for example? There were a couple of things included in the 3.11. The limited API from 3.11 onwards, with everything exploded, something changed internally, opaque type, slot IDs, internal microchanges, many things that we need to figure out. So, luckily for us maintainers, we have PEP 3.8.4. This is an historical PEP. If you go to the documentation, it really says this is just for historical purposes, but this is what evolved into that little page of the documentation that you have seen many times, for example. In a nutshell, you can define the limited API at the beginning of your code before the Python.h. You can specify the version of Python, you can just leave 3 as well. If you are using Windows, you link with one DLL and not the other one, then the tag of the wheel changed, and then you will have something like it's ABA 3, and you might have some performance issues depending on what you are using. So I wanted to include this PEP as well because the other day in the C Python panel, there were some discussions about incompatible changes. So I wanted to show you in case you didn't know, how things kind of work, right? I thought that this was an old discussion, and when I took this screenshot, I thought, okay, the 2020 was the last time someone discussed it or amended it. I noticed that a couple of months ago, there were also some discussions as well. So this thing, it's still evolving, so they're still figuring out, or maybe as you saw in the panel, things are changing on how to change things. So in a nutshell, this is including everything that is public API. If you were in the panel, you know that sometimes we have public API that should not be public API. And the policy says something like if you have incompatible changes, of course, it's only accepted if you have a large benefit-breakage ratio. You have a duplication process, but if not, then you need to follow some procedure of having two consecutive minor releases, continuous releases. There are some exceptions that happen time to time, and you don't need to care about soft deprecations, which is when you stop using something and put any warning there. Okay. And I wanted to put these slides, maybe, because there was a question about how do the people deprecate stuff? So in a nutshell, I don't want to bore you with this because it's only text. Discuss the change, add warnings, wait for Python releases, check for feedback, final removal, and then any question do ask the series calls. Okay. In case someone thought, oh, the pronunciation of this guy is really bad, I'm not saying ABI or API in a weird way, they're two different things. So in case you don't know, I'm sure you all know. ABI, application binary interface, you know, just for use, I mean, linking all the whole thing for the internals of the Python and API, it's what you're using from the API that Python provides to have your extensions. And this is more or less where they, in a nutshell, what they do. So ABI-related, AB3 on your tag and the other things are related to functions. Okay. I promise you that was a boring part, but it was really necessary because, you know, talking about policies and stuff is not really fun. So what's the catch? We're using the Python-limited API. The subset is limited, right? So you will encounter that the whole things that you have available for you, you cannot use them all. So that's really a problem because maybe you have a large project using, you know, some CBI and then it's a problem. Functions are as low as the macros. Most of the macros are not part of the limited API. So if you were using one of those, that's also a problem. There is no guarantee that when you have set up the limited API, your whole code conforms with this. We'll see something later on in an example. There's no guard about calling invalid functions, so functions are outside the limited API as well. And you might need some extra functionality that is not provided. So can I use maybe, I don't know, generate a wheel for each Python version? Some of you, I guess, at who knows CBI will? Okay. A few hands. It's a really amazing project. It's really simple, and I think it's a really magnificent that they can generate everything with it. But the problem with this that I have is that then you encounter projects like this. Maybe you don't visit the pages in PyPI, but this is the Numpy Release, for example. Nothing bad about it, but as you can see, many things around. So a little bit confusing. So why is that a problem? And now I want to tell you a little, little, sorry. So when we're releasing this project that we're working on, and if you are not aware of Qt, just imagine framework, do graphical interfaces, other applications in a nutshell, really old as well. But one of the problem is it has many, many, many, many different types of interfaces. So in the mobile version, in the audio classes, methods inside the classes, this is a huge thing, right? So of course, if you ask me how challenging it was to support this thing to Python, it took many, many years. So for technical preview, we had a wheel that was 400 megabyte. Okay. As you know, PyPI, the limit is 100 megabyte. And per project, it's 10 gigabytes. So imagine. We had support for the three main desktop platforms, and also some different architecture in some cases. So that means we're having like a minimum of four per Python version. So this doesn't sound so bad. Maybe it's still for you, but it's not really an optimal scenario. Because first of all, we needed to ask for an exception, right? So these are a couple of slides with a few technical details that I just want to share there. I will try to go fast, so don't worry about it. So first of all, the things that we did is that most of the macros that we were using were implementing as a functions. We needed the buffer protocol, which was added in 3.11. So we implemented our own, some flags for debugging and noting when modules are loading, and other functions there that you can see there are some special case. Because you want to have the source code of your project with a function that, of course, is valid when you are using limited API, when you're not using limited API, when you're using Python 3, and when you're using Python 2. Remember, this was many years ago. So just for your information, in case you're not familiarized with functions and macros, for example, the simplest one to get an item for a list, and this is the macro for it. So you see, we leave aside all the protection or checking the parameters, arguments, whatever. So I guess that maybe you would say, oh, yeah, but that's really convenient. So let's only use macros, but that's really, really bad, because then your code will explode with any change, right? So then we did other low-level things. I don't know if someone has the light of experience of working with PyType but yeah, we needed to change some slots inside, adapt some stuff there, so really boring things. And then we were just creating our own hip types, which is a lot of, like, you know, work, whatever, whatever, so this is the part of this technical, but anything. So we couldn't use the whole limited API for this, and the problematic was with this, because, as you know, many Linux distributions do not use setup UI for packaging the things, right? So we needed to use CMake and other things. So we ended up with a source code that is filled with PyLimited API if dev everywhere. So of course, the team was doing an amazing effort, and it's really thankful, at least for, this is one of our external developers that did a lot of the heavy lifting things, and if you're interested in this topic that might be a little bit boring, but necessary, you can find in there, in this URL. We also then experienced a little bit who he's as, maintain a package that has PyPyWheels. I'm sorry, it's difficult. Yeah, but that's another side. Many things that conflicts and things that don't work well when you try to have everything in the same place. So is this still an issue? One of the things that I do in my spare time is I'm one of the moderators of PyPy, so I handle if at some point you request for an increase, maybe you will see my face there, and then I started to notice a pattern. So people were showing me, oh, yeah, we have this following release, each release is two gigabytes, and sometimes we release something. Some other people, oh, yeah, whatever, whatever, so 900 megabytes, 400 megabytes, and so on and so on and so forth, and it's really a never-ending story. A lot of project out there, now with a whole boom of data science and machine learning, we have these huge wheels, so they are providing one wheel per Python version, so of course storage is not infinite, but it's still an issue, so yeah, many projects, I just put there all the things that I could find quickly. So that's our project. The whole magic that you can do for this to compile these kind of things is that for the whole infrastructure that we had before, the PyProject file, first of all, this is a simple one, and then you have a simple setup py, and on the top, I think on my file I had the limited API definition as well, but this is not enough to create an ABI tree wheel. So for that, it's a little thing to have. It's these little things here. So is someone still using only setup py here and not pyproject.toml? No? I'm glad to hear. It's a little bit wonky, so you still need to use this file, at least we defined it. Maybe you can specify here, limited API, that doesn't change the tag. So I'm sorry, but that's the problem. And then when you do that, of course, we go back to our beautiful tool and use. And then you can see there that then we have the ABI tree wheel, right? So, yeah. This is what the limited API guarantees you, of course, that you will have a package that will be compatible with many Python versions. So there in the example is 3.7. So that means that I can install for 3.7 onwards all the packages that I have there. So that's one of the caveat things. So is anyone here using already packages, ABI 3 packages for the things that they project there? I see one hand. Only one hand? Oh, my God. Okay. I hope that you can take this with you. So the problem with this thing is that you will say, okay, cool. So after I do that, it's everything validated there. What do you think? Can you say yes or no? So it is not validated. So you can still manually modify the levels of your wheel, put ABI 3 there, whatever, whatever. And it's still an invalid thing, but there are no internal mechanism that you can have to start this kind of thing. So, luckily for us, maintainers, there are a couple of packages around. There's a first one is this audit wheel that you can maybe you heard about it, just mainly to see if you're complaining with all these PEPA 600 and stuff. But there's also the ABI 3 audit that is this really cool tool that I found like doing some research here if someone is developing this thing here and no, please let's talk later. And you can see that it really reports some stuff there. You can see there that, for example, this is a binding generator of Pi Qt, which is another project, it's not Pi side, and they are not really ABI 3 compliant because they have a symbol that is not part of the limited API. So that's really a problem when I'm saying that it's not being done. So then you say, okay, that's too much work. Can I rely on a binding generator here? Who here uses a binding generator? Okay, a few hands. So if you're using Pi bind 11, for example, you heard about it, you don't have support in Pi by 11 for generating wheels with ABI 3. I think that CFFI has one, but anyway, internally we're using Shibokin that we do have. Well, it's simple, but we pass a flag and it works, but it's complicated in that, but you can still generate it. You need to make it work as well, but we promise you will be valid. First, you can, of course, use Pi by 3 and that also generate wheels that are compliant with ABI 3. So okay, everything is good, right? Everyone loves the stable API and limited API. I don't see the following person in the crowd, so I think that I will not be ashamed to show the comments. So then I read this one. And then I thought, it's really important that what we are using currently for providing are easy to lie for us. It's really painful for the people that is developing the internals of Python. So when I saw this post, of course, I could share a little bit of the sentiment here with Marc that, of course, changing things internally will break ABI compatibility forever. So I invite you to look for that. There is a lot of insight inside this post, a lot of people complaining and saying maybe you're saying, we can solve this in Python 4, but what happened with Python 4 and the many other things is really problematic, right? Because so far, most of the changes that you have seen before in the talks, they really require that you will start modifying some stuff and this can be really, really, really tricky. So, of course, it's kind of like a love and hate relationship for this. Does anyone have read this, by the way? Yeah, you know for sure. Okay, so yeah, because it's usually like at the moment, I was saying that it should be like this, right? I mean, the internal C-Python core developers like developing everything, ooh, running around and then improving and getting more membership, but it's usually like this because we are really, it's a burden on their side, but we need to somehow pick a side, right? So are we doomed with all these things that I am telling you now that, of course, you will not be motivated to use it? And I don't want to trick your mind, but maybe not, we are. And HPY, I just saw a talk before, might be the answer. If you don't know it or you didn't watch the talk, maybe you can watch the recording afterwards, but if you go at least to the website, you can see there that one of the goals there is to beat this. To have some kind of universal binaries that, of course, we will forget about it and live happily ever after. So, final remarks. Well, I used less time than I was expecting. Final remarks, as a maintainer at least so far, is that I would say that still, this is an open discussion. I saw the other comment that I saw and discussed Python.org, and they still were in 2022 saying something like, yeah, we should sit together, let's go to a spring, let's discuss the save API and everything and everything. And, yeah, I think that, still, if you are maintaining a project, I will say that please adapt the limited API. I think it is better for in that regard. You will find sometimes symbols after you generate your wills and your users start to use them, and you will find symbols or you will find this reference, whatever, because the limited API breaks between versions, so it is really interesting for you to see that and report and help back. And, yeah, so now the question that I am open to you, and if you want to ask me now a question or something or maybe we can discuss later, should we back this promise overall? So, yeah, thank you very much for your time, and that's it. Thank you, Christian. We will see a talk that also talks about the problems that we still have there. We have time for questions. There are two microphones here in the room and you can also ask your questions on Discord, so please come up to the microphone or if you don't want or you are a remote participant you can also ask on Discord. H-the-bum. No, but I still can hear. Okay, we can try. Does this one work? Yes. Great. Thank you for a great talk. I think this is the topic that deserves more attention than it gets. So, awesome. And the question, when you migrated to UniversalABI and you had to migrate to heap types, did you also consider migrating to module state or did you keep the global variables for your types around? We kept the global variables for some time, but there might be some changes in the future that we got, but yeah, we kept it. Thank you. You're welcome. And if you are too shy, don't worry, we can catch up later. And if you have any questions, please approach me and let's discuss about this uncomfortable and boring but necessary topic. Yes, this is the right place for a conference. Thank you very much. Please give another round of applause for Christian.