 Hey. And I timed the talk, and it was 42 minutes, so I think I'm in perfect shape. My name is Brian Duggan, and I'm going to talk about Pearl 6 on Jupyter. I want to do a little poll first. How many people have heard of Jupyter, Jupyter Notebooks, or iPython Notebooks or similar things? Okay, about 50-50. Good. And how many people use Pearl 6? Or know something about it? Okay, good. Okay, so my name is Brian Duggan. I'm here from the U.S. from Philadelphia. I work for a company called Promptworks. We are a consulting company. We do projects in a variety of languages, Pearl, Python, Ruby, JavaScript, Elixir, Go. We're a polyglot organization. This is an outline of my talk. So I'm going to start off with a little introduction about Jupyter, the Jupyter system, Notebooks and the console client, some of the architecture for Jupyter, what's it all about. And then I'm going to talk about some of the specifics of the Pearl 6 kernel that I've been working on, including auto-completion. And then I'll get into, after talking about the console, I'll talk about Notebooks. I'll talk about Magics, which are sort of, they're sort of like macros in a way. They kind of transform the input or the output of different Notebook cells. And then I'll go into some more clever things like Combs, which are ways of doing asynchronous communication between the Notebook and the server. And then finally, I'll go through a sort of a fancier Notebook example and show you how to put some of these different features together. But first, if you have a laptop, you can play with Jupyter Notebooks while I'm giving this talk. If you are looking at your laptop anyway and want something to do, you can go to this URL, which is also linked in the schedule. This is the Git repository for the kernel. GitHub slash BDugin slash P6 dash Jupyter dash kernel. And you'll see a button there that says Launch Binder. That links to a site called mybinder.org. And when you click that, it'll automatically spawn your very own Docker container that's running Pearl 6 and interacting with a web Jupyter Notebook. Okay, so a little bit about Jupyter. So Jupyter started off as the, it was called IPython at first. And the goal of the project was sort of to take people who were familiar with using a REPL, a redevelop print loop interface, and just make it a little bit better. So some of the improvements that were made were, for instance, making it, they called them rich interactive shells. So instead of only sending text and getting back text in your, in your REPL, you could also get back graphics. So you could generate graphs, visualizations of data. You could separate the server from the client so that you could do more complex calculations. IPython, when it became Jupyter, part of the reason it became, it was called Jupyter was because they were focusing on three main languages, Julia, R, and Python. And in fact, just a couple of hours ago, I went down to the data science track and there was a talk about Julia and they were using a Jupyter notebook. So it's pretty popular. There are about 500,000 of them on GitHub as a statistic I read somewhere. They support about 40 languages and now, April 6th too, obviously. And they're very popular in the scientific community. So a popular use case for a Jupyter notebook is to go through a sequence of steps in a computation and to sort of show the results of each one. And after you do this sort of session of interacting, showing the input and the output, you can hand it to somebody else who can then repeat the process easily. So it's a way of sort of sharing results and sharing algorithms between people. Some academic journals have even had some integration with notebooks. Nature did something with them on their website at one point. Okay, so let's talk a little bit about the architecture. Okay, so your typical read-a-val print loop looks like this. You read something and then like one plus one, you evaluate it, you get two and then you print it. So you print the string two. Usually this happens all in one single server. You have a single program, single process. Come on in. And it's sort of a simple interface to any program. So the Jupyter architecture basically takes the R and the P and the E and splits them up. So the read and the print go over here on the client side and the avow goes over here on the server side, actually even further over here, not just. So there's a server in the middle and then over here on the left is something that they call the kernel. So the client can be either a web page which is using WebSockets and HTTP or it can be a console client where it's using curses or terminal stuff. There's I think a QT client also that has another sort of interface. And then the kernel is sort of independent. It can be written in whatever language and it communicates with the server. You can also have multiple clients on the front end and multiple kernels on the back end and the server in the middle does the brokering between the two. So every time you open a new notebook it gets its own dedicated kernel. So you can have several sessions running at the same time with the server in the middle making sure that everybody gets the right messages. So here's another diagram. This is from the documentation, the Jupyter documentation. So in the web use case you have this notebook server in the middle that's sending stuff to the browser with HTTP and WebSockets. It uses 0MQ to communicate with the kernel. I'm not going to talk about that protocol but it's kind of messy. And then the server also keeps track of the session. So it writes to this notebook file and the notebook file is basically just a bunch of JSON that says like here's the input, here's the output, here's maybe something about the type of the output and a few other things. But it's a simple JSON file. It ends with the IPYNB for Python notebook suffix. And if you take one of these JSON files that ends in that suffix and you put it on GitHub, it'll magically be rendered as a nice visual sort of representation of the session. So just the same way that GitHub renders Markdown and these other things, it also renders IPYNB notebook files. So it's a handy way to sort of share this is what I typed and this is what I got. So what happened when you typed that kind of a thing? Another note, some other assumptions, some other aspects of the architecture. There's not really security. It's not really about security. You're running your own code. So even though there's a client server thing, if you want to crash everything, you can. If you want to use up all your resources, there's no sense of your typical client server thing. It's like you assume the client is dangerous. But in this sense, you're controlling the server. So the security has to happen outside of the system. Yeah, that's it. In addition to GitHub, there are other places that render the notebook files like nbviewer.jupiter.org. Okay, so we'll take a look at the console client first. So once you install everything, if you type jupiter-console and say dash-curnal equals pearl6, this will launch a console client. I like to alias that to ipurl6. And you get this nice interactive shell. And you'll be greeted with this prompt. It says in brackets one. And if you type something in there, then you'll get out brackets one. So if I type in the string hello world, then the result of that expression becomes out brackets one, which is the string hello world. And you can do the same thing within two, and you get out two, and so on. There's some special tricks you can do. So for instance, the underscore, which came from, believe it or not, Python uses underscore just like we do in this situation. Underscore refers to the result of the last output. So in two, it's a little dark here, but it says underscore dot flip. And what that means is take the last output and call the flip method on that string, which reverses the string. And then you can also refer to the variables. So then those become immutable variables inside pearl6. So there's an out brackets one. And if you just type that says out brackets one dot uc, that's calling the uppercase method on the string hello world, which gives you hello world in all caps. It's a little bit dark there, but I don't know if you can see there's an in one, out one, in two, out two, in three, but there's no out three. Can anybody see why there's no out three? It's a little dark, but the reason is because in three, it actually says say hello world or it says say out sub one. So it's sending stuff to standard out. So basically, in addition to having cells that are the last expression, you can send things to standard out and those things will appear in between the cells, but they won't be put into the in and the out variables. So auto completion just sort of works the way you would expect it to for the basics. So if you declare a couple of lexical variables here and then you start typing dollar something dash and you hit tab, you get a list of the different lexical variables. In Perl 6, the way to do that programmatically is to call the dot keys method on lexical colon colon and that's the sort of the local symbol table. So that's how it works behind the scenes. You can also auto complete methods. So if you have a variable or an object or pretty much anything and you type dot, then you're about to make a method call. And so then if you press tab, it'll look for the methods. Now this gives a little bit, it gets a little interesting in Perl 6 because there's a very rich type hierarchy. So this is a picture of the type hierarchy. Maybe I can zoom in here. So this is the type hierarchy. So int is in the middle here and you can sort of see that int inherits from real and inherits from numeric and int also inherits from something called cool which is something that could be an integer or could be a string which inherits from any which inherits from the generic class. And so there are actually a lot of methods on integers and so if you just press tab you'll see a huge number of methods most of which you're probably not interested in. So by default if you just do a dot and you press tab you only get the local methods from the int class. But then if you start typing something then it searches the whole type hierarchy to try to find out what you're looking for. And that distinction is made. So if we have, we'll do a little example here. We have 12, we do tab, we get a lot of stuff and the way it finds out what the methods are is it calls dot carat methods on number 12. So these are all the methods just for the int class but then when it wants to find all of them it sends the all flag like that. And then this goes on and on and on and there's a lot of stuff in there that's probably not what you're looking for. You probably don't want to uppercase your integer but it's there in case you are doing something else that's not in the immediate class. There are some other cute shortcuts for auto-completion. You may or may not have heard that there's some cool Unicode stuff you can do with Pearl 6. One of my favorite things is the set operators. So if you have two sets and you want to find the intersection or the union you can use the Unicode union or intersection operator. If you don't like typing Unicode there are always ASCII variants of the operators. So for instance usually it's open parentheses, vertical bar, closed parentheses. It's kind of like an OR for sets. That's how you do union. So if you start typing open parentheses and you press Tab you can see all the Unicode set operators. So it's very handy for typing those in. Similarly, subscripts in Unicode behave like exponents. So if you don't want to do star star you can type star star and hit Tab and then it'll be easier to enter Unicode exponents. The atomic operations. So these are useful. One of the cool features about Pearl 6 is the threading capability. So it's very easy to start multiple threads that are then using the same variables. So if you're not careful you can have race conditions. In order to help you with that Pearl 6 provides a number of atomic operations. Atomic dash, something. And if you don't want to type all of that out you can use the Unicode atom symbol which is a shortcut for some of the atomic operations. I'll show you an example of that later. Okay, so let's look at the basics of a notebook. So if you're not using one of the online versions that provides you with your own docker container if you've installed it locally you type Jupyter Notebook and it'll pop open a web page and it'll start a server and then you can go to File, New Notebook, Pearl 6 to start a notebook. You'll have cells like this as you can see already this presentation is actually a notebook. So you get your ins and outs on the left after you type something you see the result of the execution underneath it. Control-Enter will execute a cell. So here we can do 2 to the 27 minus 1 and then you can edit the cell and rerun it if you want to so we can go back and we can say 2 and then use auto-completion to say 120 7 minus 1 so it looks a little nicer and then for instance, does anybody recognize this number or know what the answer is if I ask whether or not this is a prime number? Any guesses? Yes, everybody knows. So correct, it is a prime number. This is a Mersenne prime one of the ones that he got correct there were a few that he got incorrect but this one is prime. There was just in January the largest known prime number was calculated it's 2 to the 7 million something I tried it in the notebook and it did not calculate. So these are the important things Control-Enter, Alt-Enter will give you a new one underneath if you see a little star that means it's busy and then like for reproducibility we're all the inputs and sort of take it from the top and rerun everything. So output cells and standard out can have mime types so let's say we have a little expression here that's just an SVG so the Q is just a purl 6 quoting construct this says here's a here dock and this is sort of a very tiny SVG if we run this we're going to get a red circle so it says oh it's an SVG what else so magics are ways of telling the kernel what to do like I said they're sort of like macros but they can do a little more they're a little more powerful so just like you can have SVGs if you want to explicitly say the output here is HTML you can do that too so if you start a cell with percent percent or if you want it to be a comment you can have it start with number percent the first line is interpreted as a magic and there's sort of a list of them you can see in the documentation so percent percent HTML says I'm generating HTML so this string here instead of being escaped we'll just get rendered in the browser so you get your bold and your italics you can also generate latech so this is as you can imagine a feature that was very useful for scientists sharing results with a lot of equations the notebooks themselves support latech cells but you can also produce latech output so in this example I'm producing a latech that's a fraction with one in the numerator and two in the denominator and I'm wrapping it in a latech equation block and that gives me a half this one any latech people can tell what this is going to be before I run it quadratic equation you can get as complicated as you need to with your latech so some other cute magics we have bash and run so if you start the cell with bash that means actually I'm not writing purl 6 I'm writing bash so here's a little bash script that echoes a bunch of strings into another file so remember there's no security you control the file system you can do whatever you want even though it's a client server situation so we're going to make a file here that has three lines and then we're going to cat the file so we'll run this and here's the file we just created and then run we'll take a file that's somewhere on the disk and run it and what's kind of interesting about this is that it doesn't just run it in a sub-process in its own lexical scope or in another process it runs it within the context of the current cell so that means any variables that are declared inside that file are now accessible to you inside the notebook so I'm going to say run calc.p6 which is the file we just created and then the printout dollar distance which was declared inside an external file so that's something that's it's a little hard to do just in pearl 6 alone but you could do it in the jupiter notebook javascript is also possible so there's a javascript magic if you say percent percent javascript it means okay this is not pearl 6 and it's not bash this is just javascript and you can just do whatever you want on the client so in this example I'm saying element.append and jQuery is available so it's saying add a div to this element and put some text in it that says hello world okay so let's run this with javascript you can also use external libraries it gets a little bit tricky depending on the way the import is done with javascript but one way to do it is with require.js so in this example I'm loading a library called chart.js I put in comments the cdm location but I'm serving it locally since I didn't trust the wireless situation so I'm going to run this and this loads chart.js into the browser and so what can you do with a javascript graphing library well let's make a data structure in purl6 and send it to the javascript graphing library so here's a little data structure that I'm making in purl6 this is just a nested hash we've got a type sending an argument type bar saying we're making a bar graph I've got some labels going from happy face to smirky face so this is just a range of all the unicode characters in between those two and then I've got a few pieces of data underneath that are going to be on the graph that I'm generating and if we run this we get this data structure so remember it's just evaluating the purl in this case and then I can generate some html and take the data structure that I just created and insert it into the html so I use the html magic and I use json fast and then I just write a little bit of I write a canvas and then I have this little interpolation here where I kind of turn it into json one place and then turn it back in the html and then we run it and we get a nice graph so it's sort of an interactive graph so already you have some nice sort of minimal interactive data visualization just by generating purl6 data structures and using javascript graphing libraries and you can do this kind of thing with whatever 3JS for 3D visualizations any client side utilities will work fine ok so let's talk about something a little bit trickier and that is something that they call comms in the jupiter world and I'll just use the definition from their documentation so the comm api is a symmetric asynchronous fire and forget style messaging api it allows the programmer to send jsonable blobs between the front end and the back end ok so the jupiter kernel supports the comm api and I'm going to show a few examples of how this works and why it might be useful ok so the way it works is you use a variable called $jupiter and you register a comm and what that means is you say take this name and associate it with this function and the function then is going to get two channels so in purl 6 a channel is a built in data type a channel is a thread safe queue for stuff that's going in one direction or another so I'm registering a comm here called counter and the arrow there is saying here's the here are the parameters for the function it's getting a channel which is called dollar out and then within the function I'm creating an event loop that is essentially saying every one second send a new number to the dollar out channel ok with me so far so the react whenever supply.interval1 means supply.interval1 is make a new supply where every one second you send me the next number and then the whenever says associate this call back with it and then react says make a new event loop and then we call done when I reaches 5 we run this code we are creating a calm and then on the front end we use the jupiter calm api and say give me a new calm that is registered as counter on the back end and then make a new div element and associate with this calm a call back on the front end that says every time you receive a message put it into the div element ok is that straight forward enough sort of ok that's what we are going to get 1 2 3 4 5 very exciting yes ok and I have another example with comms to show you the 2-way communication and this example so bear with me for a second we have an integer and we are going to start it at 0 now every second we are going to send out the sum of that integer and the next number that comes back we are also going to start a separate event loop that is going to receive messages whenever we get a message we are going to add that we are going to assign that to our integer so we are going to send numbers we are going to start counting like 0, 1, 2 but then if somebody says here is 100 then we are going to say ok 100 in 3 100 in 4, 100 in 5 like that so we have 2 event loops they are both using variable dollar num therefore to avoid race conditions we declare it as an atomic int and when we assign to it we use atomic assign so this code would work fine if we didn't do that but there would be the possibility of race conditions ok so let's run this which registers this comm and then on the client side we are going to get one of these comms and we are going to send it the number 10, then the number 20 then the number 30, then the number 40 so we are going to keep on increasing the number this being added to so what we are expecting to get back is 0 and then we are going to send 10 then it is going to say oh I am not going to send you 1 or 11 and then we are going to send 20 and it is going to send ok I will send you 20 plus 2 which is 22 so this is an example of the kind of thing you can do with two way communication and you can sort of start with this as an example for yourself if you want to build on it and make something that does something more important ok so now I am going to finish with a slightly fancier notebook to sort of illustrate the kind of thing you can do with a visualization and with messages coming from the server to the client ok so we are going to use the D3 visualization library to visualize the Fibonacci sequence we are going to make a sequence of every second we are going to send a new number in the sequence and we are going to send some other stuff too ok so we all know the Fibonacci sequence 1 1 2 3 5 the ratio of successive terms 1 to 1 2 to 1 does everybody know this that it approaches the golden mean 1 plus the square root of 5 over 2 ok this was credited to Kepler I learned and so what we want to do is we are going to visualize this by constructing a golden rectangle which is going to have this ratio so this is how we are going to do it we start with a square in the middle that's 1 by 1 then we put a square on top of that one that's also 1 by 1 then we put a square next to it that's 2 by 2 then we go down we make one that's 3 by 3 then we go over we make one that's 5 by 5 and so the Fibonacci numbers are the lengths of the sides of the rectangles 1 2 3 5 and we are rotating we are going this way this way this way this way so we basically have a sequence of these directions which are vectors so we are going the direction 1 comma 1 and then 1 1 let's say negative 1 in the x direction 1 in the y direction so we have these different directions and then by multiplying the Fibonacci numbers by the directions we are going to get a vector that is the diagonal and the goal here is that we want to compute these coordinates for the different corners of the rectangles so the diagonals then so are the first list times the second list where times means multiply a scale or by a vector so for instance 5 times 1 comma 1 is 5 comma 5 so these are our different diagonals and then the corners are going to be the sums of the diagonal so far start with 0 then we add 1 then we add the first 2 to get here then we add the first 3 to get here and then we add the 4 to get to the 4 does that make sense yeah so this is going to be our algorithm now so now we are going to do 4 lines of purl 6 1 for each of these lines to generate these 4 infinite sequences and then we are going to send them out to the client one at a time and see if we can make a visualization ok so the first one is easy this is one of the well-known ways of sort of showing off some of the functionality of purl 6 where you can use the sequence operator dot dot dot to generate an infinite sequence so you say 1 comma 1 and then star plus star means so whenever you see a star, star is called whatever star and when star is used in an expression it turns the expression into a subroutine and then becomes an argument to the subroutine and if you use it twice then the second one becomes the second argument to the subroutine so star plus star says make me a routine that takes two arguments and returns the sum of the two arguments so by saying 1 comma 1 comma star plus star this tells purl 6 how to generate the next element of the sequence so we say 1 star plus star dot dot dot infinity and the infinity unicode symbol works so now we're going to evaluate at Fibonacci up to 20 to show the first 20 numbers so let's run this one alright so far so good okay so for the next one the direction so the directions are pretty easy just these four directions repeated an infinite number of times so we just write the list 1 1 minus 1 1 we have to put the vertical bar at the beginning which sort of flattens it and then xx is the list repetition operator so this makes an infinite list of these four numbers and so let's print out the first five of those to see how that looks okay so the diagonals now this is going to get a little tricky here so you remember how I said we're multiplying a number by a vector so the way to multiply a number by a vector is to use the multiplication operator surrounded with a pair of angle brackets a pair of angle brackets makes an ordinary operator into a hyper operator and it says multiplying these numbers together multiply corresponding numbers together so 1 hyper times 1 comma 1 is 5 comma 5 so we want to combine these two arrays and the way we're combining them so we combine them using the zip operator zip says combine corresponding elements of the array but zip takes an argument the operator so the argument to the zip operator is the multiplication hyper operator I think I said that right and so we're going to say directions zipped with the multiplication hyper operator over the Fibonacci numbers and this will give us 1 1 you know sort of scaled so okay so far so good and then for the corners does anybody know what operator would be useful for the corners okay so for the corners what we want to do is we want to find the sum so far okay the sum of everything so far can be done using the triangular reduce operator okay so you all know the reduce operator right reduce means take an operator and just put it in between all the elements of the list right reduce says like take one take the first thing off use the reduce operator and then you get something else then take whatever that is use the reduce operator and do it with the next thing so triangular reduce says okay it's just like reduce except all those little steps you did along the way like hold on to those save those and that's why it's a triangle cause you sort of generate like one and then say you're generating more and more each time the way I like to think of a reduce operator is if you do a triangular reduce operator is that if you do a reduce if you do a triangular reduce over addition then from a math perspective you're turning a sequence into a series so the two words sequence and series are sometimes used to refer to just a sequence of values or a sequence of partial sums okay so we are using triangular reduce but it's not just addition because they're vectors right so it's hyper addition so to make the corners we have reduce which is the square brackets over sorry triangular reduce which is the square brackets over hyper addition and then we put zero zero on at the end okay and if we look at the first five corners that looks like that and oh by the way we're also going to need the rotations zero ninety one eighty you'll see why in the end so we're just going to make this list here so we have five infinite sequences and now let's compute the ratios cause we want to approximate the ratios so you can you can take every pair of things using rotor if you want two and then go back one that's also a rotor so this is like take two steps forward and one step backwards and give me a bunch of pairs so that kind of looks like this and then we're just going to use regular old map to divide the two things one after the other turning into a fat rat to make it arbitrary precision running a little low on time but this notebook will be available and we also want to look at our accuracy so doing a little algebra we can see that two r minus one two r minus one squared minus five should be approximately zero so we're going to use this number however close we are to zero that's how close that's how good our accuracy is so far in the Fibonacci sequence in the ratio okay so we'll take the ratios and convert them into this form to get an idea of the accuracy and then we need to make a bunch of messages to send to the client okay so here we have another little operator thing happen I I didn't think of this but then I was on the Pearl 6 users mailing list and Larry mentioned this just by chance the other day and I thought I have a way to use that and that is a reduction using zip and comma so we have a bunch of arrays and essentially we want to take the first element of all of them second element of all of them the third element of all of them and so essentially we're zipping them together with comma one at a time to sort of make a bunch of tuples we have zip comma in between all of our arrays we can use reduce to simplify that and so then we're mapping that to a little hash notice some of these things stir we're stringifying because it's arbitrary precision nude stands for numerator denominator so we're going to get those things and we also stringify the accuracy okay and so now we set up our calm on the back end and on the client side we bring in D3 and then we have some JavaScript which I will not go through which I've defined for adding a square and zooming out a little bit finally we create a calm on the front end and every time we get a new message we we add the square to our drawing we zoom out a little bit and we show the ratio that we got from the server so let's see what happens here so here we get our ratios we're drawing using the rotations we're drawing a new rectangle and the ratio of the width to the height is getting closer and closer to the golden ratio every time for every square that we add and you can see the accuracy so there's no rounding here right this is arbitrary precision so you can sort of see the accuracy on the bottom you can see the exact ratio as it approaches the golden mean so in conclusion pearl 6 has a lot of cool features that work well with jupiter the previous example so it sort of slowed it down a little but in general you could use it for something that takes a while computationally that just sends back data as it's able to make the computations the unit code support in the kernel is helpful for things like infinite sequences and rational arithmetic can be cool for doing analyses there's still a lot of open questions the one on the bottom especially is interesting what do we want in the native pearl 6 how rich of a ripple should it be or how minimal should it be that's it thank you all thanks to the people that helped me with this presentation and if you want to run this talk as a git repository I'll put this link into the FOSDEM program but you can actually just spin it up and just go through it yourself and change it and break it and do whatever you want thank you maybe I think I'm out of time for a question okay okay question or anything yes how did you say how difficult is it how close is it to a native python compiler so there's still python running in the middle doing the server so it's basically the pearl 6 is doing the 0MQ communication to the server and so it's implementing the same protocol