 So thanks for being here third day of the conference. So good that you're all still awake. I'd like to start with a quick survey So I think you all use Python. So you just go at the moment Okay, I'm syson. Okay, fewer people see or see plus plus Okay, so because I must admit so the official title is go lang to Python the unofficial title is use as many programming languages as You can for one particular project and yet basically what you'll see in the next couple of minutes so First of all, why am I doing this? I have this little hobby project so I want to basically we build one of my favorite applications, which is a regular expression tester and I do use interface with Python It's pretty nice. It has all the good Qt build up bindings that I need But to have let's say more options available for the back end. I'd like to have slightly better performance Better concurrency or parallelism support. So I wondered which language model you're there and After some trials, so I looked into C++ first. I ended up with go because go is quite nice Let's say middle ground if you go from Python to a completely compiled language like C++ or REST You still have your garbage collector. So you have automated memory management. You do have Well a good nice standard library You have a quick compiler, which means that you don't really have to wait much if you just want to try other things So it's good documentation. It's a good IDE support. So it's a pretty nice language. It's not the perfect language not by far but I Think it's mostly good So if you compared to REST and C++ and in recent days We had this talks about how to integrate these languages with Python They will in general be faster than go They will allow it you to do more things than go But they are also, well, let's say more initial investment, more efforts than go to a certain extent I'd like to say always mode more fraction Whereas with go like with Python I can sit down think about the problems that I want to address and say okay That's maps pretty straightforward to this particular language concepts. I don't have to think about memory management or well The borrower checker or any other implementation specific things So before we go into the scene one morning And has anybody seen these particular pictures before? Thing if you find them the name is Rube Goldberg device See just some illustration like that That's the self-operator webcam and the idea is just when you beat eat something you want to wipe your mouse afterwards In order to do this You can just Tuck on the spoons and activate this bird somehow then this clock then this little rocket up and to the upper right and afterwards Wipes your mouth so it works But it's amazingly complicated and you might somehow when it doesn't yet get slightly easier to a certain extent this is this talk because Admittedly at the last slides you will see things can get slightly complicated So what I'm going to do is basically give you a quick overview on how things work We look into four different scenarios wrapping simple function wrapping objects Then more complex input output parameters and finally callback functions you'll see a lot of code basically all the code that is necessary and Given the time you will see it rather quickly So if you have any questions afterwards Look at the slides. They will be uploaded If you have still questions email me It will most likely take a week or two before I upload the code to give up before Because I have to do some cleaning up to do before but I'm always open for comments That's that starting with the basics Remember my little desktop application. I wanted to do something so Python works with go multiple options You could go the RPC route so you could use something like Google RPC with protocol buffers and Basically run two processes to communicate over well a socket local domain socket or a local a local host interface I Didn't really want to do it because I wanted to share the object model that I have to go And I also wanted to have a Python extension. So I didn't want to start two different processes I just wanted one process to run with all my little go long sweats and go routines in So I decided to go for bindings. There is already a binding that allows you to even generate Python bindings from go. Let's go pi. It is currently under active development but as our last check last week, they still have an issue that they have Not implemented the pointer sharing roles that were introduced when I think go 1.6 came out So it's getting there. It's not completely complete. And so I decided well I have an excuse to do it on my own so I did and The tool that comes is seago seago is part of the standard go distribution that allows you to Mostly do two things first of all. Well, it integrates see code with go as the name kind of implies So with seago, you can call existing seacoat Which I think is the main purpose, but you can also Export your go code as a C library Which is slightly well less used It is not too complex to do this But you have to be aware that there are a couple of limitations So as soon as your pro go program becomes a seago program Compilation speed case that goes down because now you have not only to compile go. You have also to have to compile C The build process can get more complex. So you start seeing header files both imported and exported ones and Also since go offers us so many nice things go routines garbage management, etc You have to somehow integrate these two worlds. So go has different calling conventions and see Go uses threads, but Schedules that's go routines on these threads and all these things have to be somehow moderated seago does that for you But it comes with a cost and this cost is in performance. So one of the questions is always especially calling Go code from C which also means Pison to see to go. How good is the performance? That's something we look into at the end of this talk Okay, so before we go into the architecture any questions so far Okay, so if I'm going too fast, it's always fine to interrupt since Most of these things build on well the slides before High level architecture So I would like basically to call Go code from Python Unfortunately, it is not quite as simple So we have go as a basis. So let's assume that we have written some nice go code. It's not specifically for Python I don't want to Write a seago code because I like my go tooling fast compilation speeds, etc So first of all, I need a first layer that exposes my go code in seago which Basically gives us a C API and also provides C data types once we've done this we wanted to use this in Python and And now there would be two ways either use the Python C API and write it myself or use something existing that I used Sison So we export an API where the seago and we use this API via Sison and we are back in Python and things are nice It's a simple scenario if things get slightly more complex So if you want to chair objects and have to make sure that the garbage collector doesn't steal them But you still want to use it on Python and if you want to pass a long more complex data structures and let's say integers and Strings you need some support. So we also will see two Areas of support code one in go and one in C C plus plus Now you remember while I brought in this group a loop goblet machine It's a bit process. So making this thing works It looks like this. So as my first step, I do a go build command that takes my go files the also seago files, so these are also go files in terms of how the build system treats them and potentially my C C plus plus support files and builds a C library in this particular case on archive file plus a header file So I have a C library The second step some takes a C library uses is with Sison and I hope have fully have my extension module. Okay. Now Let's make this slightly more practical What we want to do is first step is just to wrap a very simple go function So we just want to add two integers Then we want to make this function available Wrap it in Sison and finally use it from Python. So that's all For those of the you you haven't seen go before it should be pretty straightforward happens here So we have an add function It's uppercase a so the function would be exported from the go module from the go package where it is defined We have two input parameters both in teachers and we return another teacher and they are just getting summed up So it's well, ultimately one liner Now the seago code looks like this Wherever possible. I wanted to things to make things as explicit as possible. So I Defined all variables by hand. You could make this shorter But the overall pattern is mostly the same. So first of all you tell seago That you wanted to export this function to the resulting C library Then you Define your new parameters. If you have seago available. You have access to some C data types. That's this T C dot that you see In the first line so C dot in and then I have always when I wrap this function to do Basically three things take my parameters Converts them into the format that go understands. It's basically the same format and it's just for sake Then do the actual work cause a function That's where these bar result in M is called in the package main lip symbol then Do another type conversion back and return it And then I have this particularly a build line and if you look at this build line You'll see what it's not still on GitHub because it's all Slightly crummy right now But I just tell them that I'd like to have this particular C library built in the particular C Built mode for go and it appears on my father's name actually two files the header file and the point our file the archive file The go-lan generated header file is quite long So if you cannot read the part on the left, you don't miss much It's mostly type declarations and go the relevant part is mostly to the right That's our little function right there. So you see it's named as it is in the original go C go file and It's a pretty standard C function. Okay, we want to use this in in sizing. So we need a set up high file and It's not so much interesting It's well, you just call it which library you want to use. So that's the upper line C go lips equal it up You tell them the extension name. So how will this extension be called when you want to import it into Biden and Well, finally you set up some Infrastructure the language is MC plus plus year for reasons that will become clear when we go into the more complex input and output parameters But with this and with this particular build command you well get your extension and You could already use it In size in itself You don't have too much do too much. So you have to tell them about the header file and Basically define or declare From your function and then you can write a very short wrapper function That just tells them that you want to have a function that is available in Python CP def not only see You define and the variables with parameters and you call it and then you can import this particular module and do an addition and Biden Which is most likely the most complex way to do an addition possible, but there it is Okay now So far we can define functions But we don't really have states or everything that we want to go to work with needs to go in as a parameter and out as a result I already have said that I wanted to have the object model available So what I want to do is just basically initialize this object Which for my good would be the application state maybe the file system where I'm look for the files that I want to search and I wanted to and call methods on that The reason for that is that I don't want to copy all the data that I could potentially need from the go world into the Python world so I need some Way to make a go to type with its methods available in Python Then I have to ask myself how can I accomplish that? So how would it most likely look like? So in go I have a user defined type with methods. It's in Python slash size and as an extension type. I Have messes on both sides. So that's fine And in go I don't really have constructors as such I have functions that we turn my values. Okay, that's I have factory functions and constructors in Python so far. So good Then slinks get slightly more. Let's say creative. I Have to think about how do I actually get rid of my Objects when I don't use them anymore And go I have a garbage collector that takes care of that for me. Of course, you can use it wrong, but it's mostly Well, it is automatic unless you make something you do a mistake in Python it's usually the same. So I do have my reference counting management memory management However, I have the Celia in between and I have to think and so how do I tell Go that's a Python system needs an object or doesn't need an object anymore So that's something that we love to think about There is some Language feature difference a go usually uses error result values. So you have an additional very Output parameter or result parameter Basically like in Python if you were to return your result and then it's a tuple a boolean or another error value That can be met And there are some features that are quite unique to go but that I have ignored for now specifically interfaces and channels so for now it's What the Python system sees is just plain odd objects with methods on them, okay Now let's look at as the first object that we want to work with This is not just well for demonstration person. We have a person a person has a name an age and hopefully many friends and For now we just want to be able to create persons and And we want to get all the information that they could potentially have So we want to get their age the first name the last name and the string which is mostly just conquered first name last name name So it's basically the double under string function and Python Okay, we want to work with that Differences what do we see so first of all and now we first have a function that does not return Just an integer but a reference to an object. So here. It's a pointer and this needs to be expressed somehow and Well, the other thing is we have a string as a return value with a slightly more complex, but we'll have to see so What can we do? if this were For example C++ you could just expose a pointer to your C++ object in Syson or Therefore in Python It would be your job to ensure that the lifetime management works, but you could just look at the memory In go you cannot not really so they are pointer passing rules There are very limited circumstances where you can return a go pointer to a C program and for practical purposes Rather do not do it at all. So we need some way around that On the other hand, we need to make sure that as long as our Python program uses an object this object is Kept in go and if at some point in time we want to get rid of this object the go runtime is free to garbage collected And in order to do that we don't have this direct access But what I came up with after some googling around and reading some examples It's basically to have some kind of proxy for this pointer Which means that I have one central object store usually juice and per object type Where I map the pointer to an integer basically just a handle and back So every once once a while that I want to surface an object and make it available to Python I give this object ask my System for an integer handle and pass this integer handle and the other way around every time I wanted to call this object I take the integer handle Get the original pointer call the message and continue. It's not complex. The API for the thing looks like this So it's basically just two mappings one each direction Counter that tells me what the next handle should be hopefully I don't run out of 64 bit numbers quite soon and then a mutex to protect us in case there are multiple threads accessing it So this house Two problems First of all, I don't have to chair my pointers. Good. I have another Identifier for my object and also as long as this go map holds the pointers the garbage collector will not connect it So what can I do with it? I have just one function that when I call it takes a person pointer and returns an identifier So either a new one or if the object is already knows the existing one I have another one that goes the other way around so for my Handle give me the object back and I can also remove it Which just we move them from both maps and in its own time the garbage collector will get rid of it so In seago the same patterns that you saw earlier So first of all, I have some input parameters and I had to break the lines that doesn't get so long But I pass in the C Versions of the parameters I have before so two strings. You're just plain or chart pointers. So no terminated strings and Just one number I Convert them into their Go version I call my go function. I get an object back This kills. It's an object point. It's an pointer to a person object in good terms is struck and Then I get an ID for that converts the ID and gives the idea back Okay, I have a new object Distractor is it's not really the Right word because it's not deterministic, but it's basically a you are free to dispose of that in some point in the future And it just drops it from the map now Just one way to call a method So I wanted to call the person first name method You might remember this method does not take any parameters So the only parameter it's takes it's in Python term selves the object That would be the object ID here. So I take the object ID. I Convert the object ID to back to the struct pointer. I actually call the method the far result line Once again much boilerplate and not too much actual work and once this is done I can Get the C string or can get the first name as a result. I can convert it to a C-sharp pointer the C string a function comes with C go So that's one of the provided one and I can pass it back I have to be careful because now since I'm in C land I have to be responsible with my memory management and this C-sharp Pointer is allocated. So it needs to be freed most likely by the Well, future owner who gets this doing back See Joe see go Jen in the build process generate some headers again And I can use these header in Cyson. So Cyson knows my C library and Then I can write an extension class. So I c def class to use it all Now let's start in the middle and the C init function the pipe and Cicent C init function is basically used Before in it and you can also use it to allocate memory and to do all low-level stuff that you want might want to use For all practical terms. It's basically the unit. So build my object here my object is Only the handle it does not really have much in terms of member data Because everything that I want to use lives in the go world Every thing that every member that I want to access needs to have a get a set on access a function Because it keeps things nice and simple. You could to design it differently, but for my purposes it works quite well I have to think about one other thing now if you call this function with first name last name at age What happens is that I request a new person from seago So I call the function that provides me with a new function What could happen and we'll see this later on is that I have already a handle so the Integer or rather the long that identifies the object and I want Just the C python wrapper for it So I also if I just handle this object ID in I need to do something slightly different There's also one additional issue in terms of ownership So let's say we Have we'll see this later. We have a function that returns all the friends for a person So in our terms it returns a couple of object IDs If I see this object ID Do I create a new size and wrapper class or do I we use an existing one? You could in principle we create a new one. However, the problem is if this one Drops it would call its the alloc function and it would tell Golang that it we don't need the Object anymore So in to make things simple The idea here is just to see this object ID is already known If you want to use a size and wrapper for this particularly ID, we use an existing one So we only have one and That's why we have these known dict up there and while we use we graph while I use we references just to Be able to check whether I already known object Without forcing the object to stay in memory. I don't think that's the most elegant solution So most likely this will at some point in time Wonder down one level and be handled in C++ But for now it's well the solution I came up with And it's quite simple really in well, it's long, but it doesn't really happen that so I get my Python parameters for the constructor convert them into something that is ego understands Chart pointers again pass them into the sego call sego will then turn them into go and down in The go world will have a new object or can get rid of it the actual working function so to get the first name is Surprisingly short. So I call these Sego function with my object ID I Get a chart pointer back. I convert this chart pointer to a Python string So to unicode using this nice example function from I'm sizing and now I have a Well sizing object Okay, and we can use it and we can take a breath So this has been most likely been Quite a lot of code at once. So just to iterate we have something in goal and If we wanted to use it in go we could most likely just Use a pointer. We can't watch this pointer all the way to Python So we need to map the pointer to an ID. We use this ID and Python and this ID says Everything that well if you wanted to do something with this object. I need the object ID. Okay, it's basically self we need to map this all through sego sego is Basically see in terms of API So I have to map everything that I have either in go for Python or in Python for go Into something in C It is nice for integers for strings. We already saw that I need a Char or a chart pointer that needs to be allocated It needs to be converted to unicode back and forth and All this to be able to create an object which then lives only in go memory and to use this object so like I said, it's Slightly more complicated than you'd like to want it The good news is that all this code can be handwritten. We'll see at the end that is Well possible So anybody has any specific questions before we go on because things will get worse before they get better Okay question. Sure. So I'm not sure if it makes sense But if you want to implement a package with an implementation with go at sego and the Python package itself How was the structure would look like? That's on the or maybe you will show it later I don't want to switch all the way back. So in terms of structure you Basically get this thing. So the package looks like a normal size in package. So it's an dynamic library that you can load This dynamic library happens to use the size in file So the PyX file plus the header and the archive file. So the static library that sego builds for us So you have a dynamic library point SO file, which is a Python which Includes the archive file from sego and this archive file from sego Includes the whole go runtime. So usually even for our first little function. It would be for megabytes about So from your perspective, it shouldn't make any difference whether this thing was written in C or in Gola Once this is all happens the build process can get complicated Which is one of the reasons why I've now only tried it from on the Mac because sego itself Like we said like you said earlier requires a simple build step and this build step also requires some C or C plus plus code If things get realistic so building it It's not so nice the package once we have the binary package. It's like any other size and package Thank you Okay So far we have basically passed very simple data strings integers now we wanted to make our person a little bit more Powerful now we wanted to add friends get their friends And also want to get some information about their friends So all their first names and a mapping on how many French with which age do I have? Actually, it doesn't make much sense to use these functions I just wanted to present how slightly more complex data structures So for the first one if I add a friend I can't add a friend twice So I need some indicator for an error and in Golang I return a second value an error value and I also return the number of friends that I currently have So it's first challenge. How do we express errors second one? I return persons which means here object pointer So these are not primitive data types anymore. I need to build Python objects that we present each and every friend So now I need to return a list or a slice of strings and finally another favorite a map or a dict of Integers to integers and Design decision time again How can we represent that? So the first three we already saw Integers or floats or whatever are basically the same in every language There are some things that you have to keep in mind we see but it mostly works strings Painful but doable so you have to convert from strings to pointers back to strings Good thing is that Golang is also you to have eight encoded so you don't have to take care of that too much Object types we talked about it a lot. So we got the ID and the idea is wrapped in an extension type that works Now I have to have a gap between My nice world and go where I have all these data types and in Python where I also have these data types and And In C actually I don't have that so I don't have really have a nice standard C list I have arrays, but they are slightly limited so I would at least to have to include the lungs I don't really have a map or a hash their libraries that have that and Error values of C has many different ways to express error values There and that gave me an excuse This excuse was also to include C plus plus There is slightly less insane than it sounds The reason why I use C plus plus is that Sison has C plus plus Support so if I can get C plus plus vectors or C plus plus maps to Sison I can more or less directly use them Also, I don't have to write my own container data types in C So the idea is I need I can write or can have this data structures in C plus plus And the only thing that I need to do is to make them Well, agreeable to see go See go does not directly call C plus plus so I need a C API for my C plus plus code. I Don't know whether you've always done this before but it's it almost always looks like this So I tell my compiler that if it is a C plus plus compiler, it does should not Mangle the names so it should treat them as plain C names and Since this is just a C compatible header file Nobody needs to care that under this all these void pointers are actually vectors and maps and other things So to keep things simple when I've done this I have a C API For C go and I have nice data types underneath well, and I have some Extremely simple C plus plus So all in all I think for this what I do here. I have m50 lines and all these things do is Give me some basic API to push values onto vectors So that would be our list of slices or include values in maps So I can then return a pointer to this map from the C go to Syson and Syson can then use them Okay, so if you look at this, it's Not too complicated. The things before loop is the most interesting part of it there. I use my Opark list functions. So I just Get a pointer to my list See who doesn't need to know that what this pointer refers to and then I call a Function with this pointer and the information I wanted to add and somehow memory a vector is built This vector isn't available for Syson. All right So now I can work with more complex data structures, and I didn't even have to write them myself Arrows I think there are many many talks about the best ways of error handling for those in the go community You may have already read about try. So there are opinions. There are opinions in C plus plus Python is relatively straightforward. We use exceptions I could have done different things here So I have could build a custom struct to return my return value and an error for every function that has an error what I did here is Quite simple. I just used a double sharp pointer as an output parameter If an error occurred I set it somewhere other than null with an error string and then I use this error string in C and Syson and Just throw an exception with this particular error so Here's basically where this happens. Here it signs a pointer if I get an error value So I check for error if it is there built an error string And then I can use this in Python and if it's not null So if there's an error then I throw an exception. It's not the most elegant way to do it because I lose many of the Potential in information that an error value in go might have it's an interface. So it could have have much more information But it at least lets me know what roughly went wrong Yeah, and it looks like that so I Create my persons again. I add some friends. I get some friends counts by age so Somewhere under the lid a map is constructed passed up to Syson and Syson can work with this map and Well, I try to Add a friend that is already known I get an exception You who okay? Now you have to be slight. You have to be strong because this is a They part of the talk where I was sitting yesterday when I went over it again and just okay How was this supposed to work again? so starting with the design We want to call back. I Wanted to be able to have a function for my person Where I pass in one function that tells me based on the age of the person if I should return This person so maybe I just wanted to have everybody been under 16 or everybody who's Ages in a certain age range. I want to function looks like that So I pass the function into a go function I just check for every friend whether it matches this function if yes, I include it into return if not then not And this led me to the following thought process We want Python code so I want to write this function in Python not in go not in see We need to somehow get this into the cgo layer cgo does not know Python code however So what it does know our function pointers Now Syson can define a C function and if I can define a C function I get a function pointer Okay, so I have a way to get my function pointer Syson to the rescue can also execute Python code within these functions So now I have a function pointer to something that executes Python good and Now I have to call this function pointer in cgo Just one more step Cgo cannot directly call function pointers So we need a helper function in C that calls this The function pointer that in turns goes to the Syson function that has a Python code. So it's slightly complicated It works somehow It's thank you so Just to give you some time for questions because truth been told it's quite a lot We need some C code to represent our function pointer and the helper function and we have Just these and cgo things Cgo has in the cgo function. I have a local function definition the fun thing in the middle Which actually is a closure around the function pointer and calls it So I'm now into C land and I can call the function pointer with our underlined in Python code Insighten I Can just take my function the object fun. I can assign it to a global variable I'm not happy about it either. There is an alternative, but it's even more complicated. So it's for later and Then I can call these global function down in the filter by hc and I can return whether it matched or not and Surprise surprises actually works. So we have this rather senseless London, which checks if the person is Ordered into tenant if the name is divided by two and we get just one of our four friends back and well it works You're there congratulations So was it worth it Yes, and to a degree So it's 130 lines of Gola Mostly before cross avoid quite a lot. It's quite a lot more of cgo. There's a little bit c and a little bit c plus plus and 200 lines of side So if I had to do this for every object by hand, it's doable because the code is very well, it's always the same convert your functions convert your results and Copy paste the function name into different files. It's not too bad My hope ultimately is to generate that so the nice thing about go is it has a parser module So it's quite easy to have a nice view about your go code. It's a simple language. So you don't have to Do well, you don't have to account for c plus plus macros and stuff like that So my idea would basically is to write a code generator. So that's not so much a problem Another one is a slight challenge. I just wanted to benchmark is and that's just a benchmark I built a couple of persons. So in this example, I built 4,000 persons Each person gets assigned up about 50 friends and Then I want to find the persons that have the most friends So I take all this 5000 persons iterate over them and just count how many how often the names show up I do this by using The term the person's name as a string. So I just take this let's say Jake B 43 SM a key in a counter and I count over them and I wrote this in three different versions version one where I call this the string function so One function for every person that gives me the whole string Version two where I get all three attributes on their own and Finally one version where I do everything and go Now the surprising thing is Python is faster Wait quite a bit. So if I do everything and go goes faster, okay But if I do if I use the Accessor calls then go wins some Python wins and why is that? because every call from Python to go have a significant overhead. So if you compare just these two you see that the string function That gives me the complete person string It's about eight ten times slower and go It's worse for the first name function that does less So I have quite a bit of overhead to when I call this function call Goes through all these layers. So I convert my Object ID to a pointer. I call it something on the pointer. The go runtime has to make sure that all the stacks are lined and This is something that is well, I'm not quite happy with me that at the moment So good news is things work bad news is performance is still not where I would like to have it So maybe message Q&C or something like that might help but well Ultimately go and Python go together Thanks Maybe we thank you maybe we have time for one question, but should be a fast question. If not, you can reach Stefan later on No question. So give a big to give a little bit. Thank you to Stefan. Thank you, Stefan