 before lunch break. The next session is about CFFI and we don't need presentations for the speaker. Armin has been a long time well-known member of the Python community working on Python and CFFI and other stuff. So, welcome him and thank you. Okay, so today I'm going to present mostly CFFI and I'm going to talk a little bit about 5pi as well because, well, we need to have one 5pi talk at every Europe Python until we have any this year, so well. Okay, so first CFFI. What is CFFI? Well, first CFFI is a project that we created about 2012 and it is actually a very successful project according to download statistics of PyPuy. Well, you can see numbers like it's 3.4 million downloads every month nowadays and it's actually, it has beaten Django, cool. And I mean the main reason why it is so much successful is that there are a few very successful projects like cryptography that have switched to it. So it means that every time you install cryptography you also actually install CFFI as dependency. Well, 5pi is probably a successful project, it's harder to say for sure. And I will talk more later. So, let's start with CFFI. CFFI is how do you call C code from Python, right? Because obviously you have C code, like everybody has C code. Like most libraries out there are actually C stuff. And if you want to call one of them, then you need something. So CFFI is just one more thing, one more solution to call C code from Python. And the name CFFI comes, well, it's boring, just means C for function interface. It shares ideas from a lot of projects actually. The original motivation comes from Luagit. Luagit's own FFI module is similar. But then we took a lot of ideas from other projects like siphons, C types, and so on. So here is a demo. Let's say you want to call this essential function from any POSIX system, get PW9. What do you do? Well, you first do man, get PW9. You see a man page. The man page contains this. Like it tells you, okay, you need to include this and that. And then you get this function, get PELUNEM that takes a char star argument written in your strict password star. And a bit later in the main page, the strict password should have roughly these fields, like PWNAM, PWPASSWORD, PWUID. UID is a tip type. UIDT. It is all fine if you're programming in C and this is all a mess if you're programming in Python. So what do you do? With CFFI, you write this code in Python script. You import CFFI, you make a CFFI builder. You do CFFI build.cdef, triple quote, triple quote. And here is a big string. And this big string, you copy and paste parts of the man page. Like I'm going to say, type def int UIDT, except I'm not exactly sure it's an int, right? It could be long, short, whatever. Because it's C. So I'm going to say int dot dot dot, which means, in CFFI, it means some kind of int, but I'm not sure exactly which kind of int. And then you do same with the strict password. You say it's a structure that has a field PWUIDT. I know that's a UIDT. But then the strict passwords contain tons of more stuff and I don't know what they are. And they really depend on the platform that you're running on and so on and so forth. So you just say colon, colon, dot, dot, colon. And other fields here, right? Here the dot, dot, dot are really meant as dot, dot, dot in the source code. It's not meant as this demo glosses over details, right? And then you copy paste the line forget by PW name. That's easy. Okay. And then the man page also had something about include. So we paste them here, some other declaration. And in this FFI builder dot set source, we also say, give a name, PWUIDCFFI, that's the name of something that we want to create. Okay. So you put these two slides into one file, one Python file, you run it, you run it, and up you get PWUIDCFFI dot SO. And now this PWUIDCFFI dot SO is a standard C Python executable extension module. So once you got it, in your main program, you just import it, you import lib from this module, and then lib is something that has an attribute, well, a function, built-in function called getPWNAM, and you call it, and when you call it, you're going to get a struct password so you can read the field, PWUID, print it, and this works. So in this simple way, we have made an interface to call this C function from Python. And that's it. Okay. So what I'm going to talk about now is, yes, it's not completely as simple as that in all cases, so I'm going to have some more examples about more complications. Like, the first one is that actually in this built-in module, you get two objects. There is lib, there is also an object called FFI, and this FFI contains general helpers that you may need to call at some point. So the general helpers, yes, sorry, this is also, this is also other things that you can do in the C def. You can declare your function. You can also have types that are completely opaque, like dot dot dot. This is, for example, what you get if you have a C library that has an interface, like make a window and returns your window star, but you don't need to know or care what is the type window, and then hide window, destroy window, all these C functions. Okay. About the FFI object now, in the FFI object, you get a few helpers. For example, if you really want to make a C structure, like here, I want to make a structure that is of type char underscore. So with char brackets, that means, like, if you know C, you know exactly what char bracket is, right? So this is generally the approach of CFFI. You need to know a little bit of C, but then if you do know a little bit of C, then it's easy, because it's the same. So who you can, with FFI.new, you're creating an object of type char bracket, and you're initializing it from a string. So you get in P some C data of type char bracket, and it owns 12 bytes. And if you count, actually, it's the number of characters plus one because there is a terminating null character as traditionally in C. And you can, well, you can index it. You can read or write two individual items. You get also another kind of C data, for example, by the code that we did before, lib.getpwnam. Well, first, we did it before, in the example, by giving directly a string, but you can also give it a char bracket, which means an array of characters, like P in this example. And, well, in any case, you get, as a result, Q, which is actually another C data of type struct and it leaves this address in memory, and then you can index it, sorry, you can get its attributes. So those attributes are just the field names of this. So from this, from such a Q, you can also cast it to void star or to anything else. This is a C rule of cast. You can cast it to another pointer type. You can cast it to a integer type, like if you want to cast, if you have the pointer and you want to really get the number that represents, that represents this pointer address, then you cast it to an integer type and, well, I mean, I could have written ffile.cast long or int, but instead I'm using the type int ptr t, which is an official C type, that means an integer that is large enough to contain a pointer, but it's just the same, and I'm getting a number that is an integer-valued pointer. So this is the kind of thing that are in the ffile object. You also have ffile.string. So this is, for example, where in my structure I have PWU ID, okay, that's 500, but I also have PWU name, and reading it returns a char star. And then from this char star you can convert it back to a Python string, if you want. So this is what ffile.string is for, and so on. One example, if you're doing something a little bit more complex, and you really want, you have a Python object like this x, this x in this example, I want to have this Python object cast it to a void star that the C code will just carry around, and then at some point later the C code will give us back the void star, and from that void star we want to go back to the Python object. I mean, this is standard, for example, in all callback systems, like if you register a callback for a C library, typically you give it the function to callback, and you also give it some kind of void star argument that the C library will just store and it will give it back to your own callback. So in order to do that you would use ffile.newHandle, cast any Python object to a void star, then you save away, fish it again, you get a void star that happens to contain the same value as a void star, and then from this value you can go back to the original x object using ffile.fromHandle. So this is just one example of more advanced things. Well, CFFI as a whole supports more or less the full C language, which is actually not so huge. I mean, it supports the full C language, I mean, of course, not the full declarations of C, like what you can, what types you can declare, how you can call functions, various calling conventions, and so on and so forth, like on Windows CDef, on Windows CDecl versus STD def, no STD call, sorry, this is supported by CFFI. Okay, so it's more than this short introduction suggests, of course, like if you really want, if you have some larger library, the C library, typical example is not such a library, well, you don't want to expose directly this library, but instead you want to expose some kind of Python, some kind of Pythonic wrapping of the library, so what you do is you write your Python wrapper that itself uses CFFI, but it uses it internally, like you write your classes on nice functions in Python, and inside internally you would use this C data object, but you would not actually expose them to the rest of the users of this wrapper that you're writing. So this is typical. So basically, instead of, well, instead of writing, for example, C Python C extension module, where you would write in C a bit everything, like you write your C Python native types and so on and so forth, and then you get only the C extension C that people import and use directly, well, here with CFFI the idea is more that what people import and use directly will be the Python wrapper that itself uses CFFI. Yes, well, there are actually a few other use cases that I did not really speak about, now you can use CFFI in the mode that is called ABI as opposed to API in which it's a mode where you don't have any C compiler involved at all, and then, well, you get more like C types, as in you have to declare exactly your structures and your functions, and you're not allowed to make a mistake, and you're not allowed to use the dot, the colon, colon, no, the dot, dot, dot syntax I showed. Well, there is also support for embedding instead of extending, which is the case where you have your big program that is written not in Python at all, but it just wants to import and use Python for embedding. So for this case, there is a mode of CFFI in which you can write, so you write Python code, you declare, you declare with C def, the thing you declare with C def becomes the interface that is callable from the C code, and then the rest of the program calls this interface and calls into your Python code directly. Okay, well, C is a dot basically. Yes, so let's talk about PyPy for about three minutes. PyPy is a Python interpreter. It's different from the standard which is C Python. The main goal of PyPy is speed. When you run PyPy, you get an interpreter. It looks very much like C Python. The essential difference are four instead of three greater than signs in the prompt, but was it just the same? You replace Python with PyPy. It's cool, et cetera. Please use it. The main difference is that, for example, it implements a very different kind of garbage collection. It's a moving generational incremental garbage collector. If you don't know what these technical terms mean, it's fine. Well, what I mean mostly is that because it is moving garbage collectors and we have trouble implementing the C Python C API interface. So it's hard for PyPy to import a C Python C extension. It's possible because we did tons of facts basically, and it's possible and it's slow, et cetera. So it kind of works. I would say it works better and better as in we can mostly do it for NumPy, for example, nowadays, mostly. So an announcement, et cetera. But yes, well, PyPy is great right now if you use Python and don't rely on a lot of the extension module, for example, everything. A lot of examples of web services are like this, like you import Django or stuff, whatever, huge libraries. That's typically written all in Python, so it works very nicely on PyPy. Well, the API is large and it's a mess to implement in PyPy. Well, I would argue actually that this C API of C Python was actually part of the success of PyPy, sorry, the historical success of Python, why Python worked or started to be really useful like 10 years ago or 15 years ago. It is also because it has this C API and people actually use it to actually build interesting things on top of it. But, well, and then you have all these binding generators that have been built on top of it, so you don't need, well, you can write C extensions manually, but you can also use these other tools that would generate the extension for you. And CFFI is just one more such tool. Well, the CFFI is a bit different, I would say, because the goal is really not to expose any part of the C Python C API. As in, yes, you can write C code with CFFI, but the C code that you write should not use any PyObject star or any PyInt from long or any of these functions from C Python. So it means also that it means that it is possible to port this, this whole CFFI module to other interpreters than C Python, and that's what we did. So that's one of the motivation for CFFI in the first place, is that it is possible to write a PyPy version of CFFI, and indeed we did. And the examples, the demo I showed in the start, in the start of this talk, well, it works just exactly the same on top of C Python or on top of PyPy. Well, it is actually faster on top of PyPy, because PyPy's JIT compiler knows a little bit about CFFI and is able to compile, to read, produce machine code that will directly call the C function, for example. So it's extremely fast, basically, on top of PyPy. But it does not mean that it's extremely slow on top of C Python. On top of C Python's performance is acceptable as well. So, yes, it works on C Python on PyPy. It would be easy to port to other Python implementations. It has not been done so far, as far as I can tell, like JSON or R on Python. So, yes, the main benefit is that it is independent on PyPy. It no longer depends on the C Python C API. Use CFFI, it's easy and cool, and it is supported by non-C Python implementations. It's a conclusion of my talk. Thank you, Armin. Are there any questions? It was first. I have been, well, working with CFFI like a few years ago. I also did, well, mainly on PyPy, which is a C-based IoT framework. And I encountered, like, it was really hard to create, when you have complex projects, to create all the headers so that, well, pre-compile all the headers to feed them to CFFI to create the library. So I actually, it wasn't documented at the time, I don't know if it's now, but you can actually run the user compiler to pre-compile your header so that it includes everything. This has improved, yes. Now it's a cleanly separated two-steps process. Like, you really write a separate Python script that declares what you want, then you run it once, you get your extension module, and then you use it from your main program. So it's better than it used to be, yes. Thanks for the talk. That looks really cool. I have a question about PyPy, actually, that I'll ask you because there is no separate PyPy talk, it seems. What is the status of Python 3 work there? Like, it would be nice to get to 3.5. Is it anywhere near? I suppose if I were to give an estimate of time, like, I cannot obviously, but imagine that I could give an estimate of time that I would say that next year should be nicely progressed towards PyPy 3.5, yes. And what kind of help do you need? Money, people? Yes, well, we need people on time. Get on money. All right, forget money. Hi, thank you. Another question about PyPy, there is some kind of tool to embed PyPy like PyInstaller or PyTweaks, something like this to embed and distribute binaries of this? I don't know. Any more questions? I'm the wrong person to ask, I suppose, but yes. I'm sorry. Armin, thanks for the talk, thanks for PyPy, thanks for CFFI, it's amazing, I use it quite often. And I was wondering, when you have the declarations with the ellipses, like, dot, dot, dot, don't care, you figure it out, okay? Can you, like in very simple terms, explain how it goes out and finds out? Because it always works, okay? So it's very good. Like this, for example, UIDT is some kind of integer, but we don't know which at all. So the magic is to write one piece of C code that will work just by compiling it with normal C compiler. So, well, every single one of this dot, dot, dot is a different kind of magic like, for example, the type def int UIDT. It probably contains, how does it work again? Yes, yes. I mean, it must be something like, like you write one big C expression that says size of UIDT equal equal one, question mark, then I'm going to use this else, size of equal two, then I'm going to use that, et cetera, et cetera. And then you do an extra round of magic to know if it's signed or unsigned. Like, yeah, I mean, for signed versus unsigned it's something like you take minus one, you cast it to UIDT and you ask it, is it positive now? So we're at the end of the normal session time from 30 minutes, but food won't be there until quarter two. So people want to ask more questions and sit around, but just want to let you know that if you have to do half past, you have to do it now. Thanks for your talk. I have a question about Defines. We have a project with a lot of Defines that are constructed dynamically during compilation from a lot of nested macros. And it's possible to use them by name in, I mean, in Python code, because actually I don't know they're writing. Defines like constants? Yeah, for example, we have a driver that uses some I.O. operations and these commands are constructed from Linux macros. Actually, I don't know what they're doing. Some shift, some source, some orc. Can I use them by name? Yes, I mean, use dot dot dot. Basically, you say here in CDF hash define, name, space, dot dot dot. That means it's some integer, I don't know which one. Figure it out. Any more questions? If not, thank you, Armin, and see you next year.