 In na podium, da so pogledali, ko so tudi všite, če so vse izgledali, ki se je zelo, in ko je začali. Vesli vši vse pošli, kiSA je naša izgleda, če ne skupio, in ki se so pošli, ki so naša izgleda, nekaj zelo, ki so pošli, ki so pošli, ki so pošli, ki so pošli, ki so pošli, ki so pošli, ki so pošli, tako če so mi prik modu, nače sem pravazalo adviski o Fondation in Pavia, da je tudi se njih jem, jemče tudi glabno, kalni, stačnите modu in zelo smo neko stačnimi. Pajte, neko je to, če je to težkost vse projekti z NR, se je vso občasil v zrobjenju sezmik na Evropa, hier. It was a shared collaboration at ETH Zurich. And actually the engine was developed first to make it possible to plot these kind of maps. So we do bought the other, means real get waste, but also the risk meaning we can give you a map saying how much damage there will be in this region, how many people will die for earthquakes, things like that, and even more stuff. Tako, to je pravda kvalitva, ki smo počutili. In včasno, to je veliko velika kvalitva. Prejdeš klaster, da je tukaj. Zelo smo počutili nekaj data, da je nekaj data počutili nekaj 100, 200 gigabijtih data. tudi nekaj, da je nekaj so velik, ali je, da se očasim, je to, da ne bilo odlično zelo, vsega vsega vsega vsega vsega vsega vsega vsega, nekaj je bilo zelo infrastruktur. In potem, da se nekaj posproste, po vzpe, nekaj je zelo, nekaj je zelo, nekaj je, da so vzpe, je to zelo, je to zelo, zelo, da se je zelo, da se je zelo, Namazno, da je vse objevak tudi vš件, vse je izgleda na mjelji, kako je zelo mene. Načo je, da je zelo mene, da je vse načo jaz vse vse vse, pa je tudi opravil vse vse. Po svojosti vse je, da je vse zelo poživljeno. Poživljena je bilo, da je vse vse, da je vse poživljeno. Zelo, da je vse vse vse vse vse vse vse vse. Tukaj transfer, da je spodobnje vseč, potrebno je zelo zelo zelo zelo zelo. Svala cipje, prikrančenje. Vseč sem vsem, da so prikrančenje. Tukaj. Tukaj je skrinskot, da se izgledaj na to, da se izgleda na vseč. There are more or less 500, a bit more than 5 other courses running and now these actually are two different calculations and two different classes, but we are also able to connect everything in single classes. So there is a lot of CPU going on. What's behind this kind of picture. So what I am going to talk now is mostly about what are the libraries that we are used and how was our experience with these So if I say its labarist bad, it does not mean its labarist bad. It was bad for our use case. Or maybe it was good but we are not able to use it properly, this is also possible. So don't think this as a criticism, or something. Or some lab is specific. Anyways let me start from NumPy's. Say PI, which I guess everybody knows in the Python world. They are the standard. They also are the ones with less problems. je to nemajstvena rejabilitva, v každje je zato da včasno je zelo vroč. Znači, sem boš ampak prič, za vso všeč nemajstvena rejabilitva, bilo je delat in jaz sem nebezpojejno, da je to pričal, da je nemajstvena rejabilitva in jaz sem nebezpojejno. Najnega je, da je to pričal, But the problem I had here was we structured arrays. We used the structural arrays which are like records. It is a future numpad which has been there for years, but this did not play well with picole and picole. So when I had this structural array, it was returning back from the workers to the controller node because of course we have distributed situations and we had the controller and then workers. vse. Got in trouble in means that all my tests were green, everything was working, demos were working, but once you started the real big calculation, you were getting back bogus numbers essentially, for some strange reason that I never discovered. At the end I solved because instead of using structural arrays I used plain regular arrays and then it was going on. One of the things that may happen that we have a library that works perfectly well in small cases, then you try in the big case and something strange happens. Something strange can be a segmentation fault, can be an error, but can even be and unfortunately often is that you get bogus numbers, like all zeros instead of the right numbers or even random numbers. So it's a bit tricky and scaling can give some surprises. So NumPyCyPy are the core. Then we also use a lot H5py, which is a Python wrapper built on top of the HDF5 library, which is in C. Raise your hands if you know about H5py, HDF5. I would say at least half of you knows. It's a very good library if you have to store arrays and more than arrays, you can also store attributes of these arrays. It's also hierarchical, you can have a hierarchy of arrays. It's very good, it's very efficient. It is more than 20 years old, the C part, the H5py, why the Python part is newer, but still has been there for a long time. Still I would say is not perfect, and this certainly is on a level of maturity much less than NumPy. So we have issues, in particular this structural arrays that we use a lot. Also issue with this variable length arrays, variable length arrays as a feature, when you have an array with rows and columns, let's say that for each row the number of columns can be different. It's not a real array, it's a list with a variable number of columns. This kind of structure is very convenient, especially if you have arrays which are, let's say, mostly empty. So maybe this row you have only 10 columns and the other we have 100 columns, but you don't want to fill everything. So they are convenient, but in the past, in particular, even in the present, they gave you some problem. Another problem that I did not expect was the byte string issues, because you can say, this is a library for arrays, where do we have the strings? The strings are in the attributes, because in the HDF5 word, you can store also attributes to this for the arrays and these attributes can contain strings. And they are never sure if they are strings, they are bytes, so I mean you are sure, but when you have to support at the same time both Python 2 and Python 3, it was a bit tricky to get it right. More tricky than I expected. Now we have migrated everything to Python 3, so now everything is fine, easy, no problem anymore, but to support both of the things, at the same time it was a bit tricky. So I am giving some caveats. And then there was this very recent debacle, I call it because it was really a big problem, because we were using until one month ago, the HDF5 1.8, which is now end of the line, so we got deprecated by, the last version was released and now they say now the 1.10 is the future. And we did this 1.10 upgrade and we lost feature essentially. We did some stuff. One feature we lost, because that before, while the calculation was running and writing on the file, you can open another process that was inside that file. Of course, another process was writing, you were not sure that the file was in a consistent state, so maybe once in a while you tried to read what was in there, you got an error in the reader and that was fine for us, okay, it's an error, somebody is writing, it's an error, but 99% of the time you were getting a good result, so you could see essentially the percentage of calculation, because in this HDFI file we save the data. So we can see, I wrote, I don't know, 12 megabytes out of 20, so I know how much stuff I stored, I can see what's in there, also the percentage of calculation is there. So I could see that, now I cannot, because now there is a lock, so 100% of the time is impossible the file because another process of exclusive lock. So in theory it's more secure, in practice it was a bit annoying for us. Also this new version of the library has a mode which is called single writer, multiple reader, that should solve the problem, because if you enable this mode, multiple reader can read the file while somebody is writing, but I was not able to get this mode running in our use case. There are examples there in the library, it works, but it did not work until now in our case, because it's more complicated, of course, than the textbook exercise, so I'm still fighting, maybe we will be able to fix this, but still it was not easy this transaction. Also the HDF view, if you have used it, HDF view is a tool that is provided by the H5 consortium, allows you to see what's inside the file. The old version is not able to read the new format, so we have to use the new version of the H5 3.0, it's called the reviewer, but this viewer has some regression, so I cannot read files that contain variable length arrays with structured data type, which was what we were using, but before it was working, now it's not working, it's kind of annoying. There is another tool, which is called SILKS, SILKS view is a Python tool, while HDF view is in Java, SILKS is in Python, and it works better, it works better in the sense that he had the same bug, the structured arrays, but I opened a pull request in the data repository, the SILKS view, and after two hours they fix the issue. So in this sense I like it, they're very responsive, so if you have trouble, maybe you can consider using SILKS. Oh, no, wait. By far the most difficult for us, the most problematic issue in HDF5 was in situational parallelism. So there is what I call a fork badass to do with a fork, so if you follow the workflow that I have written here, so first create the file, close the file, then fork, fork when you call a multiprocessing, multiprocessing is doing a fork on Linux. So if you close the file, then fork and you read the data, it works. Of course you have to save on a different file, because you can read and write at the same time, at least in the older, now there is the single writer mode, but it doesn't work very well, so we have to do different files. So the problem is that if you forget to close the file, for instance, or you remember to close the file, but you have another piece of your code which is opening the file and you don't know that, and this happens, of course. So what happens that sometimes you get the right result, sometimes works, test green, sometimes you get an error, like the file is corrupted or something like that, sometimes you get bogus numbers, very, very annoying, very, very annoying, so. And this H5py, I say, is evil in the sense is not really MPU much, for instance, this code that if you go outside the limit, there is a side limit on the attributes, if you go out of that limit, you get a segmentation fault and it's not clear what caused that in the old version that it was working in the new version, it's not worth all. It's not so easy to use this library if you have very large calculation, things going on, because we are writing 100 megabytes of data. On the other hand, it's really, really, really fast. So we are not going to abandon HDFI, it's really I couldn't find anything in the point of view the performance. It's very good at the performance. And in every time we had the problems, I was able to find the workaround essentially, so this is also good. Then we are using a lot of geospatial libraries and on this point maybe I will be faster, I will go faster and you can ask me after the talk. There are issues even in the geospatial libraries, not everything is easy. But probably most difficult things are the installation so particularly because shapely is doing some monkey patching inside, so if you don't have the right version, maybe works on Linux but does not work on macOS. There are all kind of problems with these libraries in a bit tricky. We also used QGIS, we had a problem, QGIS was using a version of LibJOS, it was incompatible that the version of LibJOS we were using the engine, so you cannot use the same code base. At the end we had to do a client, several approaches, so we put web interface on top the engine, QGIS was going through the web interface and not importing directly because otherwise you have problem incompatible version of libraries. So there are issues here but maybe ask me later. Then instead what I want to talk a bit more is this rabbit, mq and salary combination because the way we are doing concurrency on a cluster now is using mq, which is not the ideal technology to use. It was not my choice, let's say it was inherited from the past and we were able in these six or seven years that we are using, we were always able to found work around and get this work in so we are still using it. It's not the fault of salary at your app, mq that I think are very good libraries. Problem is they are meant for a different use case because our use case we have to transfer larger race, we have to transfer gigabytes of a race and this is optimized for small messages not for this. Also rabbit mq has a database which is called Mnesia where you store intermediate results and messages essentially which is useful for lesions because in theory if you do calculation you start but we don't use that feature for us if a calculation dies instead what happens that we have a very large calculation typically you run out of disk space because rabbit mq is trying to store everything in Mnesia in certain moment run out of disk space which for us is bad for us would be better to run out of memory immediately and then we can find a way to change the calculation for hours maybe you run out of disk space it's still not the fault of the technologies but it's not optimized for our use case also it's very complex rabbit mq, there is configuration there are users, there are touching parameters that they should not touch then they have strange errors so if we add something simpler we will be happy also we thought about using instead of salary rabbit mq using salary redis which is lightweight, everything is in memory thought the performance should be better but then the revoke functionality the revoke functionality functionality you say ah, there is something went wrong please kill all the tasks which are running that revoke functionality did not work with redis and this was something 2 years ago did not try again recently maybe now it works we don't know but still this is reason why at the time we tried but didn't work to use redis or something different as rabbit mq with salary also the default configuration is not the ideal one there are some parameters to tune that we have to change otherwise we run out so well in particular at the beginning 5 years ago we had this let's call memory leak so that all the results so I am producing 1 terabyte of data all this terabyte of data was kept in memory because let's say I had 1000 tasks each task 1 gigabyte 1000, 1 terabyte, all terabyte was kept in memory and we had to monkey patch library at the time now probably there is a flag to avoid the monkey patch of us we asked on many list and they gave us the answer so there is good support it's a good library it's not meant for this kind of very large calculations the worst problem we had with salary was I don't know last year this we had salary waiting for a ready completed task so we don't know why and for some calculation only for some not for all was reproducible what happened was that everything was finished but salary was waiting for something that was already returned so the calculation was hanging 0% CPU on all the cluster so we did for instance we reduced the number of cores because we had 256 cores problem 192 cores problem 128 cores was working so it's a scale issue but if it was inside we don't know if it was inside salary was really ugly so this is why of course salary was more than 50,000 lines of course not so easy to go there see what's happening and the part in Erlang is even worse what you do as a Python programmer and also when everything works there are still limits on the data amount of data you can transfer so if I want to transfer 10 gigabytes in 10 hours not a problem you can transfer works fine but if you want to transfer this 10 gigabytes in one minute what happens that you will get an error in the salary I mean the Q side you will get an error like it's too much data transfer there are problems in scale so what we needed was a plan B to progress and the plan B for us was 0MQ so I knew about 0MQ for a long time for years probably but was never really interesting going there at the end last year during the summer we had these problems we had this salary hanging so I said let's study a bit 0MQ and see if we can do something so during my vacation I studied a 0MQ book very nice, did some experiment works there was some problem essentially I'm very happy with 0MQ I can recommend it I would like to have 0MQ inside the Python standard library if it were me so in the first pages of the book there is this push-pull pattern I follow that essentially after some attempts etc it worked so I can now transfer 10 gigabytes in one minute without any problem still you have a problem at the Python level because you cannot pick or unpick a ray which is too large 4 gigabytes and that's you cannot do anything but from the point of view the 0MQ it was really really well we discovered one issue that the packets keep circulating what does mean this if you have a calculation you run calculation number 23 and this gives back to the controller node data then something goes wrong like out of memory so the calculation dies and then you start another one calculation 24 this calculation 24 we see packets coming from the old calculation from the number 23 because they are not lost they live there so what we found it was relatively easy to fix because we have the calculation number if you get a packet the calculation you are running so from the old one so there was something to look a bit but essentially we are really happy with the 0MQ and now we are using it in conjunction with celery so we are still using celery but we are returning back the big arrays with 0MQ not with celery why because we don't want to reinvent the wheel for instance we also have a way to run everything without celery using only 0MQ except that the revoke functionality does not work so I should be there and try to reinvent the revoke functionality so killing all the tasks I did not want to reinvent this also because I am physicist by originally so I am not a system programmer I don't want to reinvent this kind of stuff I did what was possible to do in 2-3 days one week for the rest probably we should use a different technology which does everything for us this different technology maybe dusk maybe dusk we are experimenting with it I am here to talk with you guys if you have experience using dusk please after the talk talk with me what I can say that years ago I know about dusk since the beginning but we are conservative we said it is too recent let's wait, let's see the documentation was not clear at least to me I thought dusk was all about task dependencies, graphs, complicated stuff instead there is just this dusk distributed client map which is exactly what we were doing I need a replace for a map instead of doing the map in salary let's do the map with dusk and essentially I tried out in 2 hours I got something working on a single machine for the cluster our assistant tried to install it and says that it works everything is low so we are at the beginning we will see if we will use it or not then I want to talk a bit about what we are not using which can also be of interest to you but we are not using our C extensions, Saito, Numba or this Intel Python that I mentioned because Intel Python is a sponsor of this conference yesterday there was a talk a training so not a talk, a training made the Intel guys very nice talk, I was there I attended so I want to mention why we are not using it so C extension we actually use them in the past this was before the invention of Python wheels and before the invention of Python wheels it means that we want to install a software that depends on the C extension you need to have a compiler on the client machine and this was an issue particularly in Windows because our users we would not expect our users to have the ability to install a GCC compiler so so what we were doing we had a fallback we had this part which is compiled if you have the compiler etc ok, if you don't have it there was a Python replacement for it this part was computed essentially distances distances on the sphere but it was bad because essentially we had duplication of logic the logic was kind of complex more that you would expect to see this part in Python and if you change something you have to change it to places then there was also the big issue of the migration to Python 3 because the part in C was written for Python 2 and we wanted to migrate and also there was another issue that I am not a C programmer and also I don't want to be a C programmer and the one who wrote this part left the company years ago so this was going to maintain etc so essentially the problem was solved because I was able to speed up the Python part and just drop the dependency ok and our codebase now is 100% pure Python except the part in AMPI for our point of view 100% Python is really really nice to have that Sighton is good, I mean I was here at this conference followed some Sighton training 2 years ago would be better than plain C at least it would solve the problem Python 2, Python 3 because I wrote a single code but when I tried it on that code for the distance calculations the speed up I got was I don't know 20% not really significant so if there is an order of magnitude difference then ok but if it is 20% then it is not worth the effort also a lot of our code is not suitable for Sightonization in the sense that it is code that we don't write because we have a library of ground motion prediction equation ok the ground motion prediction equation is the equation that predicts the earthquake it predicts how big will be the ground motion and this part can be written by our users which are typically our scientists and these scientists want to write Python so we don't want them to require to install Sighton it would be everything more complicated I am at the end and we said no for a moment tried also Numba tried Numba always on this distance computation part the part that was in C originally and even here I could not get a substantial speed up and by using the parallel option because Numba has an option that you can parallelize your code for you but there is a trick because we were already calculating everything in parallel so if you are already calculating everything in parallel then you use a library which is also parallelized we have probably nested parallelism which is, as it is called, oversubscription essentially are too many, there is too much contention and in the end you get that your calculation is lower than before so and also Numba use more memory we have a problem because we easily run out of memory and in the end what they discovered that Cypi special distance which is defined in Cypi which is already our dependency was always faster than the approach I was using always sometimes even one order magnitude faster and so I said let's switch that and now distances are no more bottleneck for us not a problem for what concern inter python there is some concerns about vendor locking because we don't want to force people to use one specific technology but it's not really that it's really that even if yesterday they trained me so I speed up 100 times when I tried on my code it was slower so I will talk with the guys and tell in the moment my point is I will wait another year one year ago I tried and was slower and this year is still slower and then there is also this point that this is my last slide I want to close here I also don't want to spend too much time in technologies because typically the best use of your time is to think about the algorithm when you are using you go to the scientist say but do you really need to do all of this calculation because 99% of the time there is a work around there is another way you don't really need to do that and you can get impressive speed ups like for instance one week ago no no two weeks ago where the calculation for Canada was taking weeks to run for 11 cities in Canada in those 11 cities you could use another approximation and doing things 100 times faster because we are doing more calculation than needed actually the difference was within the error that you have so this is also one of the biggest reason why I don't want to try all the technology in the world want to spend my time try to find solution on the scientific side and that's it I'm open for questions we're not going to be passing the microphone so if you have questions can you go downstairs here and ask your question please I was wondering if you have any experience using the x-array library instead of bringing together NumPy, Pandas, Dask no, didn't try everything so sorry this is rather new I guess I'm not sure how new it is actually but seems to be bringing together the technologies that you mentioned I heard it but I never tried that's your problem fitting to the MapReduce paradigm for example can have you tried for example PySpark or something like that but PySpark is a big technology stack it's also the Java part et cetera so we want to have our constraint to have something which simple you can install your laptop most of your users are also scientists that maybe they are not doing this large stuff so we need something that works as small maybe you can scale up also on the cluster in the let's say easiest way possible so you mentioned one of the tools from Dask to actually use the immigration part we mentioned explain that part a bit the immigration for Python 2, Python 3 no, no, no, the Dask like yes, yes, understand what you mean yes essentially what we are doing here we have the map functionality which if you are using multiprocessing it just pull map actually star map if we are using salary there is something similar and if you are using Dask I use this client map I use this map functionality everywhere so since now Dask is exposing this interface for me it was 5 lines of code to change to dispatch it because we have a wrapper over different parallelization mechanism and by specifying a configuration you can choose which parallelization to use so if I have a map framework everything is easy for me I can support in the open coin technology that technology I want to thank you Mikail there is one question, sorry you talk about H5P do you have by any chance have time to test pandas a lot we are not using it in the engine because pandas mostly to process data for my job is to generate the data more so it is no medical simulation so it is not our and it depends on us so I try with but I cannot say much about pandas we don't have time for one question sorry so again I want to thank you here if you want to ask questions coffee break or later was a great talk I really like how do you finish the talk I agree with you that improvements on algorithms is more important than on tools the next session is going to start in 5 minutes and it's going to be about reinforcement learning and thank you very much