 Day of the conference or sad face So our first speaker for this session is Michael Bright. It's going to be talking about Jupiter for everything else Thank you So Jupiter for everything else seemed like an arrogant title to me But at the end of the day, it's just this premise that you don't actually have to be an astrophysicist to use Jupiter for many other uses and So one of them is actually command line if I'm quite a few uses just using command line So I hope even the astrophysicists will find something interesting to take away from this talk This isn't what I do at work either So I'm a Brit who I did my Brexit in the other sense 24 years ago. I live in Grenoble in France And I work for you the packet enterprise What I'm showing is nothing to do with my work though when I can I do slide Jupiter in there for some things And I also run a Python user group in Grenoble So if any of you pass by Grenoble meant to present you'll be more than welcome People will be will be glad to see something other than my beta talks of conference talks Okay, so I'll talk briefly about I Python due to Jupiter themselves. Sometimes it's not that clear Or the difference between them given the the big split Look at the Duke Jupiter project and the ecosystem because I figure if you're gonna call it Jupiter for everything else You have to look a bit at the existing stuff And there's a lot of stuff in the Jupiter universe And then I'll get on to some of the things that I've been playing with So Jupiter for everything else Okay, so it all started out with this afternoon hack in 2001 by Fernando Perez who basically wanted something to help scientists in this Exploration process where you do some work individually then you share it with with colleagues When you get to a certain point you want to maybe test that on a production system And then publicize and then eventually when your work becomes established then educate people and He wanted to start something that would help with that Exploration loop and the publication associated with that And so in an afternoon he wrote this thing 259 lines of of Python the I Python interpreter. It was just a console application Actually, this screenshot isn't that this is the latest version. I Python 5 which came out just a week or two ago But the basic idea there is the same of a text-based console a redevent redeval print loop with a series of input and output cells history and also External plotting capabilities That was the beginning And then in 2014 at sci-pi They announced the big split at that point everything on this slide Existed and was called I Python Okay, and I Python had grown into this thing where the actual interpreter was the I Python kernel or other language kernels which were fronting Printed by these console or cute console or what was then called the I Python notebook So that was all Python and it was multi language and it was kind of difficult to go along and say hey You know you want to do Haskell Julia are Use the I Python notebook Didn't quite make sense. So the big split was partly a naming exercise Becomes Jupiter and Jupiter because the principal languages proposed were Julia Python and are But as we'll see there are many languages available And so now When we talk about I Python, we're in fact talking about just that Top-left ellipse there the I Python kernel Okay, so we have I Python kernel or a whole host of other language kernels That can work with the front ends on the right and They are all part of the Jupiter project Jupiter project represents everything that is language agnostic Okay, so the notebook I guess people know a bit the notebook, but again, it's the same concept of Relevant print loop that extended to the the browser where we now have rich media in a web page So we can have a mix of markdown cells input code cells and the output and the output itself can be rich media being itself HTML Everything that's associated with that so JavaScript CSS SVG it can be hyper tanks links It can be an iFrame embedding a web page. It can be an audio player to this year or even YouTube video Personally, I mean why did I do this talk? It's because I really like the notebook concept. I like that you Develop a narrative in your notebook. It's not something that you would generally use for developing code that you can do The risk is that it's very incremental and so, you know, you miss the the design stage, which is important but nevertheless for exploratory work, it's a really nice environment and we'll see I've been using it actually for running demos or For developing a lab for developing a lab. It's very nice So you have a preset series of steps, even if the lab itself is not running the notebook But it could be at least you have a way of Validating your documentation of the lab Okay, so the narrative is very important important element There are many public notebooks. I have no idea how many there are Apparently there are more than 200,000 on jitub and jitub Since May last year They will render Jupyter notebooks Jupyter notebooks are actually in a JSON format So it's not something that's obvious that jitub should be able to render But that is the case and there are several sites several galleries where you can look at existing notebooks I said I like this last one Not sure if I can go there. Okay Well, the network is working well I like this site because you can look at the most recent notebooks We can see yesterday there are about six new notebooks added to this repository And we can also see there the most viewed So just to say there are many resources for viewing notebooks online These of course are just static notebooks not notebooks that you can run Nevertheless, there are Cloud-hosted environments where you can run notebooks. I wanted to just mention Some of the big cloud providers now so Microsoft with Azure machine learning studio Google the data lab and IBM data scientists They're all providing online environments where Jupyter is integrated into their data science labs I think that speaks a lot for How the community is building upon Jupyter Otherwise, there are also another couple of cloud-hosted options try Jupyter And binder. I think I have a slide on okay I've gone first on the Azure ML Studio. So this is Microsoft machine learning studio environment They provided initially Python as a language in that and the more recently added think in March the added are as well It's still early days for that, but they seem to have serious plans the use of Jupyter in their environment Okay, try Jupyter is actually a rack space hosted Environment they're using See a docker demo image which has a set of language kernels already installed Not just languages. This one actually includes the Apache spark environment I can see there are example notebooks. Welcome to spark with Python and Scala They're also bash Haskell Julia Are and Ruby kernels there That's just an ephemeral instance You go to try Jupyter.org you will get this dashboard and these notebooks of these languages available to play with I will talk later about binder and how I've been using that which also provides ephemeral instances Okay, so just to talk about the Jupyter project itself some of the components of that So the notebook runs in the browser It has its notebook server, which is written in Python itself and has many HTML JavaScript elements associated and behind it. We have a language kernel whether it be Python Ruby or whatever And the project includes so language kernels widgets so things like ability to have drop-down boxes menus and stuff within the notebook And then language kernels So we look at the kernels So there are about 50 kernels available. This list probably probably isn't up to date I should say though your mileage might vary I've tried a few while I'm running Windows here So it's my own fault, but some of those things can be quite tricky to install some are Incredibly easy to install I think I have installed them working the meta kernel bash I have some Callisto prologue Callisto scheme conch, which is a Sort of Unix Python shell that was announced to Python a few weeks ago What else Python of course? But there's there really are a lot of languages squared there Okay extensions There are a lot of notebook extensions. I'm not going to go into detail about those I'll just mention some that I find a particular interest so rise I'm running rise now. So this presentation is actually running in a Jupyter notebook and there is this rise extension which allows you to run a review.js presentation from within the notebook There's also nb presenter. I only discovered this last week. That's what I love about Jupyter You know your presentation is already and then you discover another amazing extension So nb presenter is produced by the guys who produce anaconda Continuum IO And it's another way of doing slideshows within Jupyter I've installed that but I haven't dared use it use it for the conference. I was too frightened of breaking my existing slideshow NB grader, I mean there are hundreds of extensions. These are just ones. I find a particular interest NB grader is something that can be used in in classrooms There's a thing called Jupyter hub which allows you to have a multi user server serving up Notebooks different users in the classroom setting and NB grader is an extension which allows you to set an assignment cells will be Defined as automatically graded or manually graded How many points are associated in the particular part of an assignment? That's quite an interesting development Okay, if we look at the ecosystem and some of the future projects So some of the more interesting ones are spark magic So we've seen on the tried Jupiter. There's already a spark integrated into that image, but that is It's spark integrated into the image in general when you want to do some real data analysis with spark you're going to want to have something A real spark back-end externally. So spark magic is a project to allow that a standard way to integrate into external Spark environments so I can imagine with as your I mail for example, they will no doubt use that to integrate with their own spark Declarative widgets. I mentioned widgets top-down menus and so on today. We can use ipython widgets But that's python specific. It's in the ipython kernel Declarative widgets will allow to define menus drop-down boxes and so on within the HTML Which can then be more easily integrated in the different language kernels Dashboards we'll show you in just a moment Content management. There's a lot of work on looking at How we can import a notebook as a module? Okay today a Notebook is standalone. That's what you get but there's work looking at how to Bundle a notebook with other dependent files In port Notebooks as a module or as a cookbook and also do things like tables of contents There's also the kernel gateway project, which will basically allow Non-notebook clients are basically any web front-end for example To integrate elements of Jupiter So I have an example of something similar in a moment where O'Reilly Integrated cells of a notebook into a blog post Okay, so I will just show one of those incubator projects This one is already Available you can pip install it. I've not actually done that yet. I think it's in it's 0.6 release something like that and the idea is Being able to move away from that linear notebook Concept and to be able to arrange yourselves into a grid as you would want to do something Which you want to use as a dashboard This isn't an incubating project as such, but it's up and coming. So I think it was last week It was announced as a pre alpha Jupiter lab Jupiter lab will be the new interface. So The interface I showed to try Jupiter I was showing the dashboard in the future This Jupiter lab will be the standard interface where we will have the same Possibility of having some list of files and folders Opening a terminal opening notebooks opening notebooks in different tabs You can see that it's quite a dynamic environment It's only it's a pip install away. It seems pretty stable Actually seems totally stable from I've seen but it is pre-alpha. So you don't have the same functionality yet That's pretty nice Personally as I've done demos using the notebook. I like running demos in the notebook and everything is controlled But it's nice as well to be able to pop out into a terminal And I like it's the Jupiter lab environment I'll be able to have my my notebook running and in the same window a terminal To see what's going on. Okay. I was just given idea about Things that happening in the Jupiter project There's a lot of interesting stuff, especially on the user interface side of things I'll just mention that there are a lot of Projects using Jupiter For blogging there have been examples of nature and scientific American Doing blog posts or people doing complete books. I think it's Python for data analysis is available As a complete set of Python notebooks Jupiter notebooks And in education there are various MOOCs online courses that are using Jupiter online Okay, and I just mentioned that O'Reilly blog article actually I've been having trouble running it recently But the idea is it's just a standard blog Looks like any other blog page on the O'Reilly site But within the article they have these blackened out cells on the right You can't see it, but there's actually a run button bottom right and so as you read the blog article You press run that's going to launch actually a docker container running As you can know notebook instance and you can actually run but of sleep modify the code as you go Okay, so if I look at what I've been doing with Jupiter everything else Of course with Jupiter as I say you have this rich media Through markdown or through code you can inject the HTML CSS JavaScript SVG into your notebook It's a very rich environment I've been looking at the use of the bash kernel for command line work. Why not? And then I wanted to see well, I Like the idea like the idea of having The unique shell has a first-class citizen itself You know why shouldn't you just be able to do command line stuff as well in the notebook? But why not supplement it as well with with graphics I've been playing with with that a bit You could say why of course because you could just do Python and then escape out into bash What are several reasons for one escaping out into bash? Can be slow you're launching a sub process and there's no persistence between cells when you do that But the the other reason was I felt what I want to do a tutorial for example, which is command line stuff I want it to look just like the shell and Then with the graphics as well. I don't want to be breaking out into magics. Okay? I've been looking at So I mentioned the rise extension. I'm running here. I've been looking at publishing live blog posts. In fact But it's not quite that people are using Jupiter to create blog posts. That's not what I've been doing I've been in blogging and then linking to serve a binder where you can run a live instance of a tutorial And I find that pretty cool And then another thing is I've been doing is being Out at work so I work in telecom. We manage about six open stack platforms Open stack is a cloud management platform And each of those systems is between probably about Pete and 30 physical nodes. So there's a lot that can go wrong So I just I've done some just simple stuff and the crown just monitoring a bit those systems I'm not saying that is how you should manage those systems It was just some play quick and dirty and it decided to extend that I thought well, why not use the notebook? I saw people were talking in the mailing list about using Facility called NP convert So I gave that a shot. So we'll look at that as well Okay, I won't talk much about Web technologies as such. I look at it within My command line stuff. Just one thing I'll say is I mean you have access To the browser so you have enough rope to hang yourself by I won't hang myself with this normally, but just here's an example where I can I can't see very well But I changed the theme of my presentation. So that's just a bit of JavaScript a bit of CSS HTML And you have full access, but you can do lots of things that will Break your notebook if you if you play too much with CSS Okay, so looking at what I've been doing with the command line There are actually two bash kernels available and I went for this one the Egypto account Callisto, and they've got a set of meta kernels And I like these basically because well first of all I could get it working on my windows with just a one line fix But also because they have a family of meta kernels. The idea was There are magics and I Python, but those are specific to the I Python kernel and here There are a set of magics that are available for all their kernels So I mentioned that I have the scheme the prologue another thing called coconut. I think a Set of kernels have the same common magic. So I like that and it's also on the active development So why well floats my boat As I said, I started out actually doing some demos I was actually doing some Docker demos at work doing presentations and then demo I started out doing them in Python escaping into bash very nice Started doing them in command line and then also I run a docker build lab in February And so I developed the whole lab in a notebook and that was that was pretty nice And so I was I was pretty late preparing and at times when you've changed things You know make sure everything works. It's nice to be able to do to just run all And your whole lab just runs from start to finish and then you can see if there are any problems Okay, and I wanted to look at Can I make bash a sort of first-class notebook citizen and adding some magics? okay, and As we've seen a moment, I looked at some command line tutorials, which I blogged about and that you can run live on binder So let me just Just go to an example notebook with some of the things I've been playing around with in bash Hopefully this is stable wasn't quite the case yesterday, but Okay, so I just wanted to play around with the different things HTML JavaScript and so on See how I could make some of those things available in bash These first examples might seem like a waste of time But just showing you can do it you can echo some HTML into this bash function I created and it will actually be rendered as HTML So the principle was not to use the magics which I could have used but to be able to really Stay in the bash paradigm Okay, similarly, I can create an HTML table Okay, a more practical example. So a work Okay, this is an example of running a the open stack command line client and seeing a couple of VMs that are running So I captured that although I'm connected to work. I wasn't going to depend on the network connection So I can echo that same text Pipe it through this thing HTML table and display it as HTML Okay, nothing amazing Then if I define a function highlight say I want to highlight all the VMs or actually all of them are active there Then I can do that same Same pipeline deciding to to highlight the active stuff or whatever. Okay, so it's not rocket science But the there could be uses for that Or similarly just doing some basic stuff I don't know why that was empty Okay curious Okay, so function definition. So I just defined a function HTML find just be able to Output the results of that as an HTML table again. It's nothing nothing amazing itself Okay, one of the magics available in the metacernal bash This thing percent percent dot. It's basically graph few Graphics so it allows you this is line magic where you can just create a graph on the fly So I wanted to I wanted to create a Bash function to do the same thing Okay this is really just Playing seeing how we can make something which is essentially command line make it more visible if you want to run a tutorial some other examples and this one Okay In this one all I'm doing here is just creating a fire hierarchy and then I created a function to display that graphically Which is broken. Okay, never mind. No, it isn't broken. I'm broken Try again. Okay. There you go so nothing amazing just Just trying to mix the paradigms again can do just basic JavaScript do a pop-up Amazing. I'll skip the Python one. Let's go Piping Python into Python from back bash is a bit weird. I don't have a use case for that one I Did an Example of just doing some graphics with d3gs not very pretty, but I just wanted to push see what I could do This is fairly Empty example really, but I'm actually calling lightning lightning is a graphics library with publicly hosted server So the content of this is just Not something useful afraid, but it was just to show the example Similarly, I did the same sort of thing with bokeh Okay The intention would be to create a minimal API around a bash function to invoke bokeh So I've done some stuff like converting CSV files into HTML and this is the thing I'd like to do the same thing of doing CSV to bokeh and then maybe with a Minus pie chart minus bar chart this sort of option okay, I literally copied some Some code from the other bash kernel Don't buy someone from the Jupiter team just fail to do image display and all my functions I tried to make them work in different ways of Display of a file name Display on the standard input Okay, I'm gonna skip the SVG is just some magic. Okay that occurred um Okay, I mentioned so I wrote some Python So here it's just in the cell just to show you That would do conversion from CSV file to HTML See what I could do with that. So See, I'm just using DU taking the first five lines. Just see what we get Doing that for the minus H human option Then sorting that. Okay, so we got the largest files first 210 megabytes for demos directory Now I'm going to pipe it into that Python script Okay, and this will output HTML. I'm doing ahead. Just take the beginning so you can see that it constructs the HTML now they're the same thing but piping it into the HTML function Okay, so, you know, that's very much in the Unix command line spirit of piping stuff into into different functions Okay, I'm gonna skip the rest Okay, there was one other thing I did because I started to do just you know like lists of files and this sort of thing I thought it could be interesting. It reminded me of project called XML term and days of my youth Well, actually, I wasn't that young but back in 2000. This was like pre Firefox Motsila was the browser and some guy added this capability called XML term and Well, basically that allowed you within the browser to have a command line interface and it would do things like thumbnails and so on And you could it would actually have utilities would communicate XML and I believe text mail at the time I thought this was going to be really great But unfortunately Motsila was a was a big thing and it's a good thing that it got replaced by a Firefox and so it got It got called nothing ever became of it. I was surprised a few years ago to come across a thing called graph term Actually developed by the same guy. So this time it was all in Python So I was just wondering well, can I do something similar? Okay, so I created a function XLS. I see that's a tribute to XML term because he had a function called XLS So the idea was It's a bit slurks is a rent lot of rendering, but it will look Different files in the directory and it will render them in different ways. So folder icons and folder Jupiter icon for notebooks The CSV it's rendered as an HTML table Docker file there's a nice whale Okay So I had fun with that Personally what I find more interesting is binder So binder is a resource that for moment is free So I shouldn't be talking to you about it because everyone runs out and uses it and maybe it won't be free anymore This is the interface to binder. Basically you give it URL of a jitub repository It will look to see if there's a docker file and it will build the docker image corresponding If there isn't then it will look for a pip requirements file or a conda YAML file, but basically it will on the floor. Well, it will build a docker image and Then you or anyone else if you share the link with them Who'll be able to to run instances of that environment? So you have instant runnable environments that have all the dependencies necessary for running a Jupiter notebook So let me see Here's an example actually this example is the dashboard when you give it a repository This is actually my jitub repo which has this presentation in it. So if I If I click on launch here, which is rather daring mid-conference It will actually launch an instance of The docker image that I'd already built from from my jitub repo Okay, so in the dashboard And I can actually go Okay, I was just hoping for one thing that I would have the icon of the rise presentation Okay, there you go. This is the icon to go into the slideshow mode Okay, and that Is my presentation at this time running out in the cloud on binder. So I find binder really cool I'll kill that But more interesting for me So I created a jitub repo with some command line tutorials And I linked to that So you go to that jitub repo basically when you build an image on binder they They propose that you put this launch binder icon on there and if you click on that It will take you in it to an image running that repository so again, this is Live in the cloud So I there I just Initially as I blog posted and I thought well, I'll do loads of these before Europe. I done great And so I thought I'll start at the bottom and I did an LS tutorial. Yes And they took me another two months to get around to doing anything else But thankfully I did a week or so ago I added a few few also especially for you. I added a pill six one liner tutorial It's just a shell. It's an idea, but now something more interesting So here this is an index page the different tutorials I created and one of them. I like the idea quite a bit So I did a TCP dump tutorial. It's pretty short, but you know, just get the idea So I always start off by Just doing a check of what environment we're running And so running the first cell we can see that we're running a a Debian Debian Jesse. This is running on on binder What bash version we're running what TCP dump version we're running now We're not going to be capturing with TCP dump Out there in the cloud. We probably could do if I wanted to I'd have to do things in the docker file To enable that but it's more interesting in the tutorial anyway to have a Pre-captured file and go through that So I describe in the markdown how you would capture your own packets And then we see I have three examples of packet capture files And we can rerun the code To look at those so let's say I want to look at just the first two examples up at a minus C2 in there I rerun we can see that no such final a director. Yeah, well if you try and type a conferences this happens You know, okay, if I put that in the right order you can see okay shows just the first two so the point I'm trying to say is you know, this is a Live tutorial I linked to it for my blog and anyone can run that and play with something like TCP dump in the notebook environment I find that I Like that concept I really do Okay, let's get through So I've talked about rise for doing slide shows. I won't go into more detail right here Something else I did then as I mentioned we managed about six open stack platforms and four of them I Basically I wanted to look at managing them see how they're doing the disk space and so on It's actually after seeing someone talk about in be convert on the Jupiter mailing list But I'll give that a try then be converts it can execute a notebook And it can obviously convert to another format so you can call it from cron So I have a cron job that nightly Runs a notebook and a cron Converts it to HTML and then I send that HTML to me as email Now There were several problems along the way. I thought great. Okay. I'm now supplemented with graphics except that email readers will Render HTML, but they won't render graphics unless it's actually a Link to an actual external graphics file And But okay, well, I'll try CSS so I developed some CSS stuff and ah they don't they won't do the animation of CSS either Same thing with obviously the JavaScript and even SVG. So okay, never mind So I looked at what I could do and obviously HTML tables So if I look at okay here we have a live example I'm actually gonna clear all the output And rerun this The idea is okay. I developed this in the notebook then periodically abstracted stuff out so that the There would be relative little code showing in the notebook as this will end up in an email Let me just I Initially I did these basically one per platform. I know to create HTML table showing the status This is Physical nodes if they respond to being this is the VMs on the active or not This is checking where the ports are open and this is checking some of the API endpoints and then I was also getting some discusses stuff that one I've actually abandoned because The end of the day You know you want a dashboard you want something that will give you a heads up is everything. Okay, please So I basically refactored it. I'm going to run all that now It's a bit slow Okay, so this is connected to our lab in Grenoble I Mean that this is just showing the progress is pinging physical nodes and it will go through the four platforms At one point when I only wanted to do an initial status Table at the top for the multiple platforms. I thought well, hey, yeah With the notebook and do some JavaScript and just insert that up in the top of the notebook But I forgot of course that that won't be rendered in my HTML email So I just had to refactor the code Each time I got a platform status. I'd return some status codes to me and the HTML and then afterwards Once all the status have been done from just coming through to the fourth platform now I have another cell where I Construct a table of the overall status the different platforms and then I loop through the different platforms Printing the HTML detail of what happened Okay, just let that finish Okay, it's very quickly So I get this platform status summary and of course it's some HTML so I can link Link through so if I look at the second line there platform platform called NFE 5 what I've done is Shown on the right the status of the ports and so on everything's fine But we can see here the disk usage, but this platform has about I don't know 30 odd physical nodes and this is the node which has the worst case disk usage If I link down to that platform see those details on the disk space So I've ordered the disk space of lots of different nodes So this one was the worst one and here I built a bar graph showing the trend over the last 10 days or so So at the end of the day this space is high is a problem What's worse is if it's going up like 10% a day and so on Okay, so I just wanted to show that as an example of how we can use and be convert To do status reports I'm not saying this is the way I should monitor our platforms, but it was more of an experiment with Jupiter Okay, and this is the sort of command that I used another little gotcha with it was with Gmail in particular It would clip the message saying I have to go on another link Basically the standard template used there's too much CSS and so on and so using template basic you can cut down the HTML file size Okay, so to finish off I'd like to go further with this using Jupiter hub and MB grader trying to do some online So classroom type labs Experiment more of the metacurnal bash stuff I'd like to Have a more sort of consistent API set of functions that can use so I can do things like bokeh graphics and so on But from some real data I want to do some pull requests against the metacurnal bash. I'm out on the limb at the moment Propose this stuff outside the Python community will actually got rejected for Linux con Europe never mind But that's fine. There's this conch kernel. I mentioned Like to take a look at that though. I suspect It will be a weird mix of command line and Python. It's probably not what I'm looking for and also look at there's a C plus plus interpreter kernel which has a It's a kernel for the notebook. Okay, that's that's it. I think I ran over