 Stefan, stay with all of you. All right, cool. So we've only got 50 minutes, as he said. So I'm going to whiz through this quite a lick. So there's nothing wrong with your audio. It's just the way I'm talking. So I'm Steve Payne. I work for D.E. Shore, which is a global investment firm based in the US. And PGRMI is an in-house tool that we have, which allows you to basically be able to invoke Java methods and so on and so forth from Python and also vice versa. So PGRMI is Python Java remote method invocation. And essentially the way it works in the common case is that it will create a shim object on the Python side that represents a Java object. And then you can basically just treat that shim object as if you would treat any other Python object. And what's going on behind the scenes is essentially transparent to you. But you're actually kind of calling into the Java process on the other side. Now, this isn't necessarily an original idea. There are other open source implementations out there. We're currently working on open sourcing PGRMI, but that's still working progress. So if you're chomping at the bit to have a go with something like it, there's things like JEP, which is an in-process version of similar sort of thing. Py4j is probably closest to PGRMI in terms of like the general features it has. And JPI is another in-process implementation. And JEP allows you to call from Java down into Python. Py4j is Python to Java. And JPI is, I think, Java down to Python as well. PGRMI will go in both directions and it pretty much has most of the features that these other guys have. So without further ado, we're gonna kind of start kicking off. So the way that we're gonna see these examples is that we're gonna go for a sort of client server model where we have a parent child and there's gonna be a Python parent and a Java child, which we're essentially just gonna drive. And in order to actually really do anything, you're gonna need handles on objects and things. And in some of our common use cases, we have a special method that allows you to get direct handles on objects called get object instance, which is implemented on the Java side. But in this case, we're just gonna be kind of just grabbing raw classes and monkeying with them. So let's roll. So first of all, we're gonna import an unpy because of course you have to have a numpy and pretty much any presentation. And we also import PGRMI and get an instance, which we're just gonna call C because it's quick to type. And now we have our PGRMI instance with various arguments up here, just kind of keep it quiet and make it multi-threaded. We're gonna grab a couple of classes. We have now some classes instantiated on the Python side. So let's see what we got. And you'll see here that when we sort of get it to print the details of this class, it just gives us a class which is namespaced within the DShore PGRMI namespace and then within its own Java namespace. And the same for the hash map, you kind of see that that's a hash map. So these are just Python classes as you would just use them generally. And so now let's actually use them. So first thing we're gonna do is we're gonna instantiate three lists. The first list is just gonna be like standard empty one. The second list will construct it using a collection of some numbers which some people might recognize. And the third list is just gonna be a regular Python list which we're gonna instantiate using one of our Java lists. And you can see here when we print them out they kind of just look like a bunch of lists. Similarly for a map, yeah, we're gonna instantiate these guys. And as you can see, we get the first one which is an empty map. Second one, which is a map with like 22456. And the third one, which is a dict instantiated from that Java map on the, you know, as a Python object which again, it just has the same contents. Now you'll see in the example above we instantiated the Python list using a Java list and it didn't really need any particular magic. Now because the dict constructor in Python needs to set a kind of an iterable of tuples there's a little bit of syntactic sugar that PGRMI puts on the map.entry objects to allow them to be interpreted as pairs. And PGRMI for most part it's just reaching into Python, sorry reaching in from Python into Java and using reflection just to instantiate all these classes on the fly. But there are some classes that it knows a little bit about and can kind of do some handy magic with but you can kind of see here that we can use Python syntax. So we're just gonna pull a value out, you know one of our Java maps which knew the value, the key corresponds to the value two the key one corresponds to and the index four in that list corresponds to the value five because that's just the way we roll. Now things get a little bit interesting with Python and Java because they have similar type systems but they're not exactly the same. So Python kind of only really has one type for floating point and integers outside of NumPy that is income the built-in types. And whereas Java has like a whole bunch of specific types Java has method overloading Python doesn't and Java has also objects and primitive types of things like integers and floats and so on and so forth whereas Python just has pretty much objects. However, Python does have NumPy which has kind of a slightly richer type systems inside it so we can use that when we are starting to bump into potential ambiguities which can happen in overloading situations. So here we're gonna grab a bunch of classes from the Python in Python from the Java side and we're gonna basically instantiate a whole load of things here. So basically just a bunch of ranges but they're gonna be done in such a way as some will have type information, some won't and we're just gonna be doing a bunch of inference here. So we're gonna grab these guys and they're now instantiated and you can see here at the top we instantiated a bunch of things that were just pure Python objects and then we use them to instantiate a bunch of lists which are actually Java lists. Now, the interesting thing here that you'll know is that the ArrayList just kind of took a constructor so the ArrayList constructor just took a collection which PGRMI was able to use the Python collections that has been given and sort of turn them into ones that Java could use. And so if we actually kind of look to see what we got we print out a bunch of stuff here. You can see that for the first guy it interpreted the zero as a byte because the ArrayList under the hood is really just has a collection of objects. PGRMI doesn't have really any type information they can work with. So it basically makes its best guess but in the second example of the list here we gave it a list of N64s which are NumPy types and so it actually knew that these guys could be long. So even though the first value was a zero it knew that was a long. And similarly we kind of gave it a list of sort of small bytes and we sort of pulled these things out and the big one which turned into an int. And so PGRMI is making some type inferences under the hood. And similarly, we can kind of just get direct values where we say, okay, give me the byte object of the small integer which is seven and that works fine. We can also say do the same thing for an integer. Yeah, it's a small number but we can treat as an int but in Java, if you attempt to compile code that will cause an overflow you'll get a compile time error. And similarly, if we now try and instantiate a byte object by giving it an integer, PGRMI will complain at you because the value is too large to be represented as a byte. And so it will kind of have the same guarding semantics that you get with Java. So one interesting thing here that you sort of can see going on is that if you kind of look back up these values are just kind of appearing even though these are Java objects they're actually appearing as very sort of vanilla types in Python and this is boxing that's going on. So we basically have taken these kinds of types and so we're letting them masquerade as regular Python types. And we see the same with strings. If I basically say here, okay, we're just gonna get a string and I say, give me the string value of this thing. It prints as a string which just looks like a Python string but it's actual type is a special box type. But that being said, you can just call split on it like you would any Python string because it actually inherits from the Python string type as well. And if you start taking these objects and passing these box subjects back to the Java side the PGRMI code will automatically figure out that these are really Java objects and correctly unbox them. So what we're about to do here is we're gonna create two integers which are gonna be separate objects on the Java side and if we call hash on these guys we're just gonna call the regular hash method and that'll look the same but we can use identity hash code from Java to actually kind of see what the pointer potentially is or at least a hash of the pointer. And if we do that, we see that the integer, the hashes are unsurprisingly the integer itself but then under the hood, the identity hash code shows us the fact that we actually have two different pointers lurking here. The other thing as I mentioned before is that PGRMI would attempt to turn various Python containers into the sort of similar ones on the Java side and we've seen this working with ArrayList but you can also just use it with Arrays. We also saw it with dicks above as well. So here in the Arrays class there's a method called asList which takes as you can see here in the type signature it takes an array of Java objects. And so we have our NumPy a range of int64s and we can actually just pass this to arrays as list and this will give us back a list object from the Java side which we can just see we printed out it's just a list. And so there's a lot of sort of type coercing that's going on under the hood between Java and Python by PGRMI in order to allow you to have very sort of natural communication between the two languages. And simply you can just go the other way. So we can just tell a tuple, hey, here's this list that we have from the Java side and you can use it to instantiate a tuple object as well. Okay, now I also mentioned function overloading. So the ArrayList class has a method called add and what we're gonna do here we're gonna just take a quick look and we told that signature actually comes in two forms. So add can either take just an object or it can take an integer and an object where the integer is the offset out which you want to add. So it's basically allows you to insertion within the middle of the list and things. And so here we go, we'll print out our list of int64s which is our Java object and we're gonna append something to it and that seems to work because we got back a true and we're gonna stick something at the start and then let's print it out and whoa, now this is kind of fun because we have what looks like a string and then a bunch of numbers and then another string and so we can do that. Well, yeah, we can because ArrayList isn't really sort of like strongly typed. The generics that Java uses are just a compile time thing. They're kind of a sugar that you get at compile time and type ArrayList means that at runtime none of that is really enforced. And so an ArrayList is just a collection of objects and so you can stuff into it, whatever you want. And so if you actually look at the types of all of these things we will see that at the end and the beginning we have stuffed in strings into this list. And so you can basically do the same thing with Java that you can do with Python and you can stuff any old thing into a list and it can be happy, but be very careful if you then pass this thing into a function which takes a list of longs because you will get runtime class casting errors. But speaking of errors, exceptions in PGRMI work in the same way that you would kind of expect them to work in Python. So here we have that list and we know it's about 10 minutes long and we're gonna try and get the zeroth element and then get the 1000th element. And unsurprisingly, we said something went wrong because we got an exception thrown and here we see that we have the Java exception but it's stack trace and everything. So exceptions are basically similar. You see, we were just catching an exception here and it was the index out of bounds exception. But now we're still gonna get a little bit more fancy. So we have our map that we created above and we have the computive absence method which is a Java method, standard method in the maps in the JDK these days. And as its second argument, it takes a Lambda operation which allows the map code to compute the value if the key is not present in it. And so we're actually gonna do this by passing in a Python Lambda and we call it, the map was previously empty. We called it, we got back the 11 and we printed out and lo and behold, yet we called the computive absent method with 10. It computed 11 as the appropriate value and stuffed it in. So what's actually going on here? Now this is where it gets a little bit interesting because this is why I had to put in the numWorkers argument right at the top because we had to make the system multi-threaded because now Java is having to call back into Python and Python has to have threads sitting there ready to receive these callbacks because the method you're calling from the Python side, Python's just gonna be waiting on that recall to come back. And so there have to be threads working to help out. And similarly on the Java side, there will also be threads. And now we can actually go a little bit bonkers here. And so we're gonna do computive absent with a Lambda on one map, which is gonna call computive absence on another map, which is gonna call another Lambda, which is gonna be calling back and forth. And so we're kind of going from Java into Python and Python back into Java and then Java back into Python, which is then gonna call back, back, back into Java. So this is getting kind of crazy, but let's just see if it actually works. Oh, the moment of truth, yes. So you can see here that previously we had 10 and 11 in that map and now we have 100 mapping to 51. So that kind of works. So it's always good when the crazy things work for you in a live demonstration. Finally, similarly, Python has duck typing. And so we have interface, class interfaces on the Java side and we can exploit the duck typing from Python to actually pass in something which looks like a Java interface provided it implements the right types. So here we're going to create something which looks like a Java runnable and we're gonna call it Python runnable. And it's gonna have one method and it says Iran when you run it. So hurrah, that works. And so now we're gonna create a Java thread and pass it the Python runnable and let it roll. And lo and behold, that thread starts and it calls a method which says Iran, which is calling back into Python from the Java side once again. But here we're just passing in essentially a duck type interface and that's pretty much all you need to do apart from just giving it the right proxy base class to inherit from which implements a bunch of the methods that the thing needs to run. And that's pretty much it. I mean, you can also do a couple of other things like if we have the map, here we have map one sitting on the Java side, we can actually kind of essentially do a copy by value and pull back a local copy which is just the Python type but is the Python representation. And this is essentially just using C pickle under the hood. So we take the Java object, we render it's a C pickle on the local Java side and then give it to Python which knows how to turn that back into a regular thing. And then finally, at the end of all of this, we have just a very sort of cheesy random script which I just throw up through together. And this is often the way that we'll be using PGRMI within the company where you create a script that's mostly being driven by Python but is allowing Java to do all the heavy lifting for it. And so here we're gonna get our script, it's not doing anything particularly exciting but it just prints out numbers and it has some apples and that sort of thing. And so that was a very seat of the pants, quick introduction to PGRMI. Hopefully, as I say, we'll have it open sourced at some point and but in the meantime, if you're waiting for that you can use things like Py4j but watch this space and thanks for your time. Awesome, Stefan. We have a question from the audience. I'll just quickly put it on the banner. Do we have an API doc for PGRMI? So it's not a public thing yet and so this is all kind of within the company but the API in itself is actually like fairly well documented locally. So once it's open sourced it should all be kind of out there but at the end of the day a lot of what you sort of saw here covers kind of most of the basics of the thing and pretty much all we need to know and as I say, it creates all of the Python types on the fly. So apart from the old bit of sugar that it adds to a few classes here and there and some of the boxing it does for the local types. Essentially the API you get from the Java class the API that Java classes have is the one that you get locally on the Python side. We have another question. What would be an actual use case that I would definitely need this in action? So we actually use this for like a whole bunch of things. One use case which is very handy is on the fly debugging because you can like literally attach to a running Java process and then just start making calls within it. It's all running in a separate thread so you have to kind of worry about thread safety and that sort of thing but it can be very, very useful for going into a live process and kind of monkeying around with it. We also use it in cases where, because within the company we have a lot of infrastructure built in Python sorry, in Java that's doing essentially like a lot of heavy lifting work which just isn't feasible to do in languages like Python and Java is not a scripting language. If you try and write something that looks like a script in Java it just ends up as a complete mess and this is kind of why PGRMI was invented in the first place within the company was because we kind of had scripty like stuff in Java and we just wanted to have something much better on the front end to drive it with. And so essentially a lot of these use cases are when you have big infrastructure within Java that you potentially want to drive from Python. And similarly, you might for instance have like a big heavy weight system but that somebody wants to interface with from the Python side. And then it's a very sort of simple glue language because you literally just like connect to the Java process and just make calls on it. So we use it quite heavily for various things. Oh, you've gone quiet, I can't hear you. Oh, sorry, sorry. Is it possible to call into Python from Java example to implement workflow steps in Python? Yes, so what we saw here was the basic case of a Python parent spawning a Java child but it is possible to go in the other direction and have a Java process spawn a Python child process and then for Java to call down into it. And we actually have code that will work in both directions and that's quite a common use case too. Another question is, how do you tell Java where to find the Python APIs? Well, you really like the Java side has to know what it's calling. So the Java side will just effectively have embedded Python in it if you're calling down into Python from Java. And so the Java side isn't doing what the Python side is doing essentially. It's not kind of creating sheen classes on the fly but you can call directly down into Python from Java. And then when you call a method, the arguments you pass in will be like Java classes and types and that sort of thing. And the results that you get back will be transformed in the same way. Why not, Jython? So yeah, Jython's great except it doesn't play very well with things like NumPy and that sort of thing and like any C extensions make it really tricky. So it would be great, it really would if you could just kind of run everything in the same VM but there's a lot of kind of hokey stuff that goes on there. Also, Jython has a different threading model to Python where, because Python has the gil whereas Jython is actually like, in a way, properly multithreaded. And so there's probably a lot of code that plays it fast and loose with threads and lack of locking and relies on the gil which would break pretty heavily. So I honestly would love to be able to use Jython but I think it's main, the main barrier to entry there is the fact that it just doesn't play nicely with C extensions because it's just not compatible. Great, that was great Stefan. And especially thanking you because the show is also the Platinum sponsor of Python India 2020. And another similar announcement that I would want to make for the attendees that D.E. Shaw is hosting a knowledge sharing event at the Expo booth. So head there, there are some exciting gifts to grab. All you have to do is share about what Python libraries or framework that you're using and that intrigue. And for Stefan, I just wanna tell you that there are a lot of people in the comments that appreciate if you make this thing open source and they would love to contribute. So very exciting thing that you have here. We're working on it but it's one of those things where it's quite, it uses a lot of code within the company. And so it's like extracting it from that without kind of breaking all of its functionality is the fun bit. So, but we're working on it. Great, thanks a lot Stefan for making it so late given your time zone. Sure. Any closing words? Sure, so I'll probably be around for another 15 minutes or so on the 2020 stage deli thing. But then I'm gonna go to bed because it is 2 a.m. Thanks a lot Stefan and have a good night. All right, thanks very much. Take care, bye bye.