 Joseph Heinrich is the head of the scientific IT systems group at the Forschungszentrum Jülich in Germany and he will be giving a talk on the visualization system of the G.R.S. visualization system. Okay, thanks, have welcome. Okay, first of all thank you for coming here this morning. My name is Joseph Heinrich and together with my colleagues I'm working on different projects at Forschungszentrum Jülich. Most of them are visualization systems and I'm proud to have the opportunity to give this talk here at EuroPython about G.R., a framework for visualization systems. So let me start with a question. Who is already using some scientific software with Python such like Medplotlib, Mayavi, VTK or okay that's a big number. As mentioned I'm working at a research company and in the past years it turned out that they are growing, that there's a growing need for better and faster visualization software especially scientists need easy-to-use methods for visualizing two and three-dimensional data sets possibly with a dynamic component and they want to create publication quality graphics and videos for their publications probably in the internet and they want to make glossy figures for high-impact journals or press releases. At first glance those methods don't seem to be very challenging but we are talking about that later. There are a lot of scientific plotting methods we need such as line bar graphs, curve plots, scatter plots, all these things you see on this slide here. In principle this is nothing challenging and there are dozens of solutions for all these kinds of plotting methods. There are also powerful software libraries for scientific applications in Python. Those listed here on this slide are the most popular ones I think maybe I've forgotten one but we all know Medplotlib which is the workhorse and the de facto standard concerning graphics in in Python and there's even Mayavi for three-dimensional applications. Mayavi which is very powerful and based on VTK and it offers an application interface called MLab which can be used in your own scripts. There's VTK it's very versatile but it's difficult to learn because it's a very low-level system and we have tools like Wispy and OpenGL which are both very fast and which are limited to 3D and which are really the lowest level APIs for graphics with Python and there are also some GUI tools like just like QWT with its corresponding 3D equivalents and the problem with this is that they are currently unmaintained at least for what my information is. So there are some problems so far and the main problem I think is that the 2D world and the accelerated 3D world are separated. You won't find a tool which provides services for both 3D and 3D graphics and another problem is that some graphics backends only produce kind of figures so it's not possible to present continuous data streams from life sources and also I've made the experience that there's only a bare minimum level of interoperability so user interaction is somehow limited with these tools. Also if we are talking about analyzing large datasets we often see that there's only a poor performance and also these APIs are partly device and platform independent so your own scripts will suffer from some system dependencies after the time. Okay so let's Python get up and running and push for Python. There is a very nice distribution which has been introduced to you in the keynote this morning it's called Anaconda and I would really like to recommend this distribution as it's very easy to install a complete scientific Python stack but I think we need something more for example we need some more performance and this can also be achieved by Anaconda at once for example number which is also be mentioned this morning which is capable of accelerating number or your Python applications based which contain NumPy code even on GPU hardware or multicore processors and I will give some example later. There's something more we also want to achieve more graphics performance and interoperability and for this purpose I would like to introduce our GR framework which is a universal framework for cross-platform visualization and the main key points are that it has a procedural graphics back end so you can really present continuous data streams and it has built-in device drivers so you can visualize both 2D and 3D scenes in one canvas and there's a very good interoperability with GUI toolkits so you can establish a very good user interaction and as you can see in the bottom part of the slide it's also very easy to install so this would be our complete scientific Python distribution I think we have everything we need especially we have more performance and interoperability so let me give some examples how this looks live you can see here numerics simulation of a damp pendulum the calculation is done in the aka4 function you can see which is simply a numerical integration of this differential equation and you can see that you can mix graphics with test formulas and you can do all these things live while the script is running you don't have to produce figures or something like this the same works for 3D you can see it here in this case visualization is done with a API which has been written by a colleague of mine Florian Riem and he has written a open GL layer for GR which is called GR3 and you see it's very performant and it does its job you can even okay you can even visualize live signals from from ray files or from the microphone and with lock x-axis and this all runs in real time these are all things which are very hard to realize with other tools you can do this also in 3D I just pushed the audio away so you can focus on the on the graphics so the frequency spectrum is in this case visualized by by a surface plot which is realized with open GL and it's that fast as you could see you can also produce graphics with user interaction you can see here an MRI MRI application which renders some MRI data through a marching cubes algorithm which is part of our software and which can be rendered very very fast and moved with a mouse so let's talk again about performance we only have we not only have some some needs for for more graphics performance but also for for numerical performance and as mentioned before there's something called number which is part of anaconda but you also can install it for your own and there's number pro which has some additional features it's a part of anaconda accelerate which costs a few bucks I don't know the the actual price and it's capable of calculating number expressions on the GPU so you can write your own GPU kernels in Python and it's a very nice tool and it was to look at this software and there are even other tools like like cool blast cool FFT cool rent but those tools are just dedicated to cutar hardware so in this case you can see how you can profit from such software you see particle simulation which is very slow currently running at three frames per second and just by adding some decorators and an import statement for sure you can increase the performance by times 15 I think so it's you don't have to change your code and you can speed up your application enormously this is calculated in real time this would not be possible with this poor Python if you run the simulation in Python I think each frame is about three seconds in this case it was paralyzed or vectorized vectorized this is I have several examples in our demo suite just take a look at the website and you will see how the different optimizations work so let me introduce some of our success stories we have integrated our software in several of our applications we are working blow both for experimental physicists and for and for theoretical physicists and this is something for our instruments it's a live display for a small-angle neutron diffractor matter and as you could see you can set the region of interest and the surface is generated in real time you can rotate it you can flip the axis and there's even more and all this can be done in real time so this is another example here we are processing a huge data set and it's also done in real time and which was formally done by a proprietary solution and with a geopharmic we could embed this into a Qt4 by Qt4 application which was a replace replacement for the existing solution and which is much much faster which can produce movies and all these funny things there's another example here Nikos it's a very complex network-based control system which is used in at Froschungsreaktor München in Munich for all the instruments which do a neutron scattering and in this case we replaced by QVT, QWT application with a QTGR application and it was much faster it was more responsive and it had some additional features which we didn't have before so this is a case study to see how fast you can simulate data. Born is a software for simulating neutron and x-ray scattering to compare it it was a replacement for Metplotlib at this point and it uses a single call it's the line just in the bottom of the left side and if you look at the old code if you compare the old source code with a new one well that's only one line and an export statement to generate a movie for example so it's not such complicated to produce movies with a GR framework so what are the conclusions the use of Python with our GR framework and number and perhaps number pro extensions allows the realization of high performance visualization applications both in scientific and technical environments and the GR framework can seamlessly be integrated into any Python environment I would suggest to use Anaconda the integration is simply done by a C types mechanism so you can also use it in your own Python distribution and the combination Conda and Anaconda provide a very easy to manage and ready to use Python distribution that can be enhanced by the use of our GR framework especially with its functions for real-time or 3d visualization functions so what's next we are not far from implementing a molecular dynamics package and which will produce such results we have already all this stuff written in C and simply have to write some simple wrappers which will then be integrated into our GR framework and with this framework you will be able to do things like this here the simulation which has been calculated on a very big machine and the data is read with a simple Python script and then rendered with a GR3 library you can then export this scene to for example Povrey and produce a high-quality graphics like shown in the right side of the slide and you can even do this in highest resolution if you give the correct parameters to those routines and you can see here it's a very realistic presentation of a DNA so what are our future plans well we have thought to combine the power of metplotlib and GR and we think it should be possible and the basic idea is to use GR as a metplotlib back-end so this would speed up metplotlib and all your metplotlib scripts would profit from this speed up I think it's possible we didn't start this development but I think we there's a good chance that we get this things running and there are even more challenges you learned about bouquet this morning and I think this should also be possible once we have the metplotlib integration it should also be possible to connect those scripts to the bouquet back-end which Travis mentioned this morning in the keynote at this point I think we should talk to Travis to cooperate on this slide you find some resources there's a website for our framework there's a git repu it's hosted on the pie package in the index we even have first bean star other binary distributions for the GR framework and the talk should be online on this link which you can find out later so some closing words maybe you hate me after this but I think that it's important I think that visualization visualization software could be even better if the prerequisites of foreign application would be described in terms of usability responsiveness and interoperability instead of a list of software with module dependencies we should use native APIs on the different systems instead of GUI toolkits and release updates should not break version compatibility this is something that I have observed very often so let me end up and thank you for your attention thanks for me for this great talk any questions are there any questions one of the features of matplotlib that I find very convenient is its integration with ipython notebook because I can play with the visualization before I integrate it into some application or save a high resolution copy for publication or something like that so I wonder if is GR framework compatible with ipython notebook and I use it from there right now it's not but because there's some discrepancy on the others on the one side we are talking about immediate mode graphics and with ipython notebook there's just a sequence of commands and maybe if you get our matplotlib back and running as we we consider that it could work then it might be possible to use an ipython but I'm not sure about this thank you we'll do our best thanks I'm doing a lot of training of neural networks inside CYTHON very almost completely outside of global interpreter lock would it be possible for me to bind to see apis or CYTHON apis that are excluding global interpreter lock or do I have to bind back through python then as I want to visualize the training of the network during the training process I think this would be really cool for that oh I'm not sure about this I think we have to talk about this after the session I'm sorry are there any more questions okay I have a question when you use this vectorize decorator and you use the same code in the limited or the basic number version will this just do nothing or will it complain with the name error or something or import error is this vectorize decorator available I mean even if it does nothing in the basic number version so you mean if it's not present on your machine or yeah okay I mean I mean for example you have this vectorize which I think I understood only works for the pro version and if you have only the basic version and so the difference is that the pro version is capable of pushing those LLVM code on your GPU and the public version is only capable of paralyzing on your own CPU yeah so pro version is only needed if you want to use your GPU for the computing of number operations yeah what I meant is if I have if I get code from you which has this vectorize decorator and I only have the basic version installed will it just not vectorize but otherwise ignore the vectorize decorator no this would not work okay so if you use want to try those demos you really have to to purchase the pro version but there are a lot of other demos which don't depend on the pro version okay thanks a lot yeah okay if there are no okay meanwhile no questions have been added or okay so thanks again yeah