 Je vse zelo način, da sem zelo v EU. Zato sem vse zelo v zelo vzivnih, zato sem tudi na 16. 15. o 16. Vzveč, zelo vzivnih. Abonan, sem skupaj na Zem, na Pavija. Zem je gledan, prezervna model. In nekaj we poček... Oh, nekaj! Takoč. Svet. Vse. Zvaločujem je kot zelo, da je začilo vse. Vse, ki svoje dobro ležimo, je to je nekaj glasba, tako v ljudi, lahko in však u vse, nekaj začeli se dobro zelo. kjer tako finančne tuklice, kot nekaj mali nekone, nal NPT je izgledan, kjer je katalitina na všim zelo, kjel je moroči iditi. Na miri je zelo sa Úvodom, ko je dolgoje magazi, prišljamo nogod kakji ni nekako, tako je zelo, kjel je pokrito, kako je bilo, da je bilo, da je vse boh, da je hrana. To je napovo, da je še in dezember, kaj je bilo, in je bilo, da je vse vse, da je vse napovo, ker je gen vse napovo, da je vse napovo, in da je vse napovo, da je vse napovo, in da smo v Python'ju konferenci, oče, da se je ahaj napravil. Ne znašljate, kako to je pače. Vsačo je pače kaj je kaj na Python, saj, na pi, v svoj delovih, in začali po počke, 2-3 monthi kaj je komputčnje na njom skupnega, ležitko 500 kors, plus in počke, in ljudi odsleda in problemi, možno, da smo počkali z memori, da bo začal. To je pravda, zelo. Zelo sem da pa dopril nekaj, ker vzelo, da svej preformacijski komunitet je biti vsega, je biti vzelo, zelo je biti jezivno vzelo. Svej nekaj, ker je bilo dobro preformacijski komputer. So je bilo nekaj nekaj na kvalitku svoj laptop, bo nekaj je ni 4, c++. Je prič Python. So don't be scared, we are very welcoming here. And probably lots of you, I expect in the ocean, there are lots of people that are all scientists, all work in computational fields, could be finance, I have been working for seven years in finance as well. So actually I'm originally a physicist. So I think most of you feel comfortable to programming in Python and doing something, that they want to do something that is more than a laptop. So this talk is for you. So this talk will not be much about performance, will not be much performance oriented. So it's true that sometimes I do really, I need to profile code and find issues. For instance, last month some legacy code was written before my time. It was low that I had to start the Python profiler, see exactly which line was low, why, and there was a function which was called too many times in the loops and moved outside the loops. So this is a skill that you need, is important. But honestly I do this one or two times per year. It's not so common that I have to go this low level of detail. And this is a bit surprising for me because I would have expected to be spending all my time in the profiler, but it's not in the kind of middle performance computer that we are doing. Actually it's very rarely that we go to this level of detail when it's the Python profile. Essentially what you are interested in is the total runtime in your computation and more or less where are the points where is low. Memorial lens means that you don't go to the database which Python line. What you have to do is to instrument your code. So typically you have some context manager, so you say weave and you have this context manager and inside the blocker code and weave this context, computing like start time and time and save some places, how much long it took to run that blocker code, how much memory was allocated and that's instrumental. As your computation goes you store this information and at the end you have a report, you got a report, you understand where the problems are and you don't essentially need to go to low level except one or two times per year. So this talk is not about micro positionizations. It's not like the talk that receded mine about the site that was very good talk but this is not what I'm going to talk about. Instead I want to point out that what is more important for this kind of middle level computing is the architecture that you are using, the data structure that they are using, the library that they are using, for instance. An example, we have to compute distances, okay. In the after you want to know this is the hypocenter of the earthquake, how far it is from the cities. Look in these distances, in some calculations it was low. So I tried to use a site and I tried to use a different number of various things. We, no avail essentially was, I was get some improvement but if I get 20% improvement is not enough essentially to justify to have a big dependency from some external library. Instead at the end they found out that in sci-pi there is a module which is called sci-pi distance which is able to compute Euclidean distances and it was an order of magnitude faster than non-pi. So what they did was to convert, we have long left, convert in 3D coordinates x, y, z and then use the Euclidean distance. It was an order of magnitude faster, that's that. So essentially right now in the first century is probably more important to know which is the right library to use because it's already done more than going low level, depends on all the things that you are doing but for the thing that we are doing, you should read the documentation know well your non-pi supply. Okay. Having said that, I want to start, so from the input-output formats because I suppose you are writing some scientific code and maybe you want other people use your code. So really, one thing that is really important, maybe people don't think enough about is how you will use and will interact with your application which are the inputs, which are the outputs. Okay. And one thing that they discovered here is that once you decide that this is my input, you are stuck with it essentially forever. Why? Because there will be always somebody coming and said, ah, I ran this calculation 10 years ago and now I want to repeat, I want to reproduce that result. Or it was typically a scientist or engineer or users and they want to reproduce something that was published in an article even a long time ago. So you really need to support, you need to be able to run all the calculation but also for myself if I want to compare an older version of the engine with a new version of the engine with a regression, I must be sure the engine, when I say the engine means our software, it's called OpenQuake Engine and this is software that you give in input, things like the seismic sources, like the exposure and in output you get maps or region where there are more earthquakes, more damages, things like that. So the engine is, this tool is a command line tool, there is also a web interface and there is also a QGIS interface that run via a plugin so you can run calculation, display the results in that way too. So I want to say, you really need to keep your input formats working forever because originally I thought okay, but maybe I can give a converter to the user, so now the old version, this was the format, you convert and then after a few releases, nobody will use the old format and I can remove the old parser, so no, really it does not work because if you need to provide the converter, you can as well that converter put inside your source code and then have internally done a conversion every time and the user will not need to do anything and will be done internally, that's the right way if you want to do that. So, and you must do that because I'm pretty convinced that it's impossible to get the right format at the beginning because what you, for instance we decide to use XML for the sources, XML maybe is not the best format but there were reasons at the time was done in this way because other software in the seismic community were using that, the other software were done written in Java so XML for them was meaningful, so we use that, there were some standards. So there are reasons, things that they say, okay, this format probably is good but after 10 years is not good anymore, so you must prepare to do that and what you can do instead is to have an internal format so what you should do is to have some internal conversion from the input that must stay the same forever to the internal format that you can change in such a way that is a format, okay. The output formats are a bit better so we were able to change the output formats because, okay, some user may protest because, but usually they are happy because before we were giving XML as output so they had to write their own parsers for the XML, now it gives CSV, they are more happy with the CSV so it's easier, especially if the format is simple. Okay, so let me talk a bit more about in practice what we are using, so there is a 90 file where you set the parameters that they are using, I don't know, like the investigation time, you want to know how many at waste there will be in the next 100 years, that's in the investigation time, or parameters like the discretizations, things like that. And the INE format is good. Right now I would use the TOML format, I don't know if you, do you know of the TOML format? Never heard about this format, somebody is a new, no, not so new, relatively new format, so at the time there was not that format so you could not, it's used especially in the IT, there is a Python PEP to use TOML format for distribution tools. It's a nice format, it's essentially an extension of the dotini, which is also hierarchical, so it's in the middle between a JSON and then a dotini. So that's format at the time, we are happy to use that, but we are still supporting the time and we will forever use the dotini, but we use it in TAML. The XML, there was a period that there really was against the XML, not so much against, the big mistake I think was using the XSD, which is XML schema, that was really not needed because the XML schema is a way to constrain the XML, but was not really working now, it was making everything more complex, by the end you need to do additional constraints which are specific to the earthquakes, so you have to write your own validator in Python anyway, you cannot trust it, it's not enough, the validator at the XML level, so this was being removed, the XSD schema are no more, fortunately the XML could have been a bit simpler, but okay. CSV, we can use CSV now, we are using input CSV more and more, especially the engineers are very happy with this CSV because typically they have Excel sheets, so they have input CSV, so it's good for them. We are using HDF-5 in some cases, because the other scientists, there is a big split in this earthquake community, there is ASRD, which means geologist essentially, other scientists typically geologists, geophysicist, et cetera, and then there is the risk guys, which are typically engineers, so they know about structural engineering, buildings, things like that, so they come from different backgrounds, they have a different mindset, so and have them in the middle because they have to support both of them, so let's say more ASR guys, geologists, et cetera, they like to code, to program, so they use formats like HDF-5, for instance, it's okay, so they ask us, but for instance, the model for California, which is this USERF-3, is the seismic model for California, the model is written inside this big HDF-5 file, so we need to support that format because this is the format that they give us, or in Canada, they use this ground motion prediction equations, which are HDF-5 files, so there are tables and numbers inside this area, so we have to support because they use that. And then also we support the ZIP, because if you have lots of files, the ANI file, the HDF file, put everything in the ZIP, and you are done, so it's very convenient to ship in a calculation from 200 people, et cetera. No problem with that. In output, we are using XML, and when I write NRML, means Natural Risk Markup Language, is a version of XML specific for this earthquake stuff, and we are removing this output because people have to parse it and it's very inconvenient. Apparently nobody wants to write an XML of normal parse in our community, so we are trying to export instead CSVs, CSV, everybody is happy, they are CSV where we have an header with the name of the fields, and before the header you have a sharp character, like comment in Python, and inside the header you can put some metadata, like this file was generated and the date by this version, the engine and maybe some parameters that are relevant. HDF-5 sometimes we generate etch, and there is also NPZ. Do you know what NPZ is? NPZ is a zippet in an pie array, essentially. Let's say if you have more than one an pie array, you can zip them together, it looks like a dictionary where the name and the array, name array and you zip everything. This format we use it by necessity, necessity essentially we need to communicate from the engine server to the QGIS plug-in, I mentioned before that we have also QGIS interface, and so we need to transfer this large arrays from the engine server to the QGIS client, and we wanted to, originally the idea was we will transfer those as HDF-5 files, but it didn't work, actually it worked for us, but we were using Ubuntu and the version of the libraries were consistent between the engine and the QGIS plug-in, but then we discovered that in any other platform they were inconsistencies like QGIS was using its own version of HDF-5, the ZIP part, and if you want to try to read the file generated by us with another version, essentially segmentation falls, total disasters was very, very difficult, so we said, okay, let's transfer these arrays as in NPZ, as NPZ files, because the NPZ is a format that NUMPI introduced 10 years ago, I don't know, 15 years ago, it was very stable and works everywhere, so we did that by necessity. This is an example of the web interface that we have, and this kind of outputs you can have for, this is a classical calculation, produce other clues, other clues are essentially the probability of having earthquake bigger than that, at some value in each point, so you can download lots of different formats, ESP, XML, NPZ, this full report is in a restructural text, it's also another format I did not mention, but this is useful because inside the report you get information, which was the seismic source slower, or how many ruptures there were in this source model, things that are useful to understand if there are problems, or things are slower. Various things, this is very interesting, the input, one output is called input, is actually the input files that we have used to do that calculation, this is really invaluable, because if you have another guy, another because this is a multi-user situation, we have a cluster, several users, so one of my users run a calculation and has a problem, calls me and says, ah, it doesn't work, I have an error, so I can go here, download the input output, I don't know that, unzip, and I have all the input files that he used, so he can run again the calculation of my machine and debug and see what's happening, so it's useful in this way that everything, and then at the end there is this download data store, and we call data store this HDF file file, which is the internal format that contains everything about the calculation, contains another copy of the input, and all the outputs, and when somebody has a problem, can tell him, please send me via Skype or something in the HDF file file, the data store, and from that I can repeat, understand everything, so that's a good tip, have a way to get the input that your user are using, because typically they say, this happened, they say, ah, by run these files, and last year they were working, now they don't work, and then you discover the files they run, they are not the same as before, they change some parameters, so it's very important, and there is a way to get really what they run. Tenant formats that we are using, as I said HD5 is the main, because everything, the real data, the arrays are here, Tommel we are starting to use here internally, and it's convenient, because it's a readable hierarchical format, SQLite, because we have a database that store information, like who submitted that calculation, when it was submitted, and then you can do statistics, like who was the user that used the cluster more, more runtime, longer calculation, things like that, so you subscribe database for this kind of stuff, but this is, let's say metadata accessory stuff, the real data are inside HD5, and they are good, we are happy with this format. Of course, the choice of the format has a big performance impact, this is clear, but it's clear to me, but probably not to the one that originally thought that exporting everything in XML was a good idea, because we had cases where the export time, the time to export, the result of the calculation, was longer than the calculation, because we were trying to export in XML, totally foolish now, and then we removed the XML export, the export CSE, which is a lot faster, and again, if what you want to export is too big, there are other options like the export in HD5, or just give the data store where there is already information without need to export empty. Also the importers, very recently, for instance, I changed the exporters because we had the exporters reading the imports, sorry, importers. We have a format for the exposure, so the buildings, in XML, and also we introduced the format later in CSV. But for legacy reasons, the CSV format was actually reading the records, converting these records into node objects, these node objects are similar to the nodes that you will get from the XML, and then reuse the XML importer logic, which made sense at the time, because you have always had the line, so you try to reuse what you have already. But at the end, the importer was low because it was doing all this conversion, so what were recently changed, now the importer just read the CSV, and the XML importer instead reused the CSV importer, so I did the opposite thing than before, and now they are five times faster, so you can read two million assets, which is, for instance, the exposure of Canada or California, do you have millions and millions of buildings? You can read these millions and millions in two, three minutes, so they are not initial now. And for instance, normal format, they say, HD5, we are happy with that. Really, I would recommend to just that. Okay, let me, if you have a question on this, because now I am half of my talk, the other half will be, but if you have a question, that's the right moment, on the input format, that's the right moment. Otherwise, I'm going to talk about task distribution. Okay, which is very, also I think that's probably more interesting than the first part, but I wanted to start with the input outputs because they are more important than people realize. Task distribution, what we are doing, but originally we started from salary for historical reasons because the people that were there before my time, they liked the salary, they knew salary. We started originally, there was only salary. And salary has, we were a bit in a queue. Salary is very nice, queue mechanism, but is not meant for computing, for high performance, it's not meant for that. So a bit in a queue works very well if you have a lot of small messages, but on the contrary, we have few messages that are larger raised for gigabytes of data in one message. So, and then rabbit in a queue as things like fault tolerance, keeps everything in a database, which is cause Mnesia. So if the calculation dies, you can restart it, but these features are very nice, but we do not use those features. For us, if a calculation dies, runs out of memory, there is some problem, dies, you don't restart. Okay, that means that you made some mistake, you have to change the parameter, repeat everything. So it has features that we don't use and these features have a cost in the sense that, for instance, the data transfer is really slow and there is a lot of conversions between Python and that there is a airline, sorry, inside rabbit in a queue than in airline. It's complicated, but still we kept using salary essentially because of this revoke functionality, which works very nicely. So the revoke functionality is the ability to kill a calculation without killing the calculation, the other users, because we have a cluster, multiple user, I run the calculation, after 10 hours I discover that it was low, I'm still at 1%, so I said, let me kill, I will retry, but if another user, in the meantime, started something, I cannot kill all the cluster, I need to kill my calculation, I kill the other, and this is the revoke functionality that works well in salary, and we try it with other things, they don't have this revoke functionality, like DASK does not have this functionality. I asked the main developer of DASK about what about that, and he said, don't use DASK, we don't plan to have this functionality ever, so we don't use it, we start selling, in the sense. But we solved the big problem of the data transfer by using 0MQ, essentially. The trick was to use 0MQ to receive the outputs, so if your calculation produces a lot of outputs that you want to collect, by transferring those by 0MQ you can transfer one terabyte of data and without big issues, essentially. On the contrary, if you try with salary, you immediately run out of memory, out of this space, everything becomes low, it's a disaster. So 0MQ is the reason it was introduced just to solve the problem of the data transfer from the outputs back in the state. For the inputs, there are less inputs and outputs in our case, because what we are doing is generating, for instance, ground motion fields of the earthquakes, they are bigger rays, we are generating more than what. So we are using 0MQ to receive the outputs and we are using both in single machine but in the test situation. The biggest issue that we have with the test distribution is not much the technology or the libraries that we are using, but it is related to the fact that when you do a seismic simulation, you have some objects which are called seismic sources, essentially they are containers for earthquakes. So seismic source contains all the potential earthquakes can be generated. So they are in Python speak, they are iterators, there are iterators. And you can have a seismic source that contains one million ruptures inside, so it's very big and another that contains three. So it's very inomogeneous, when you have to send the task to the workers, you may end up with a task that contains big source and other task contains more sources and the task that contains the big source will take forever to run. So the problem is the slow task, essentially. So this is a case where we have a mean time or around 600 second, 10 minutes. So in average, a task running 10 minutes but this one running more than 50 minutes. And this is a good situation because now the situation has improved a lot. But in the past, we had the calculation that was taken, I don't know. 10 hours, and actually 10 hours was the lowest task, without the lowest idea, all the rest in one hour. So we're spending, waiting 10 hours for something that could have been done in one hour. This is the biggest problem we had with the task distribution. And recently we were able to, I would not say totally solid, but improve a lot on that by changing essentially the task distribution mechanism. So the idea was that we can split the big task in two smaller tasks. Out is done internally, and we'll try to explain a bit. When you receive the output of a task from Xeromeq essentially, you can, what you receive typically is the diction of arrays, arrays, or things like that. But if I receive a tuple, which first argument is a callable, and then some arguments, this tuple, this callable, where arguments actually I can think this as a task, a sub task, so I send back the information in which I can start a new task. And in this way I can essentially introduce a task glitter, which is able to split the big calculation to small calculation, those smaller calculation. Here as an example I was implemented, so this task glitter, the first element are the seismic sources, then other parameters. And this seismic sources, you can give a weight to the sources, so how many earthquakes they contain essentially. You can have a max weight parameter somewhere, so you can decide, ah, this was a big task, big weight, so I will split in blocks, and yield, when you yield this tuple, first argument is a function, then the other arguments. When you yield this, the receiver will resubmit that task function with this block of parameters. So in this way the sub task works, and we are able to split the big task. A trick is that it is difficult to associate the weight, so how many ruptures there are inside that source, with how much time it will take to perform the calculation. Because it depends on various things. I will give you one example. I could have a complex fault source that is long 10,000 km, 5,000 km, so affects 10,000 size, affects a lot of size. Even if there is only one rupture, this kind, but affects a lot of size, lots of size means you will have a large non pi array. Okay, non pi array will be 10,000 slots. And this non pi array will be so large that you will be essentially out of the, what is the, that is the process, the cache for the processor, and if the array is small, you are within the cache of the processor, and everything is fast, if the array is big, you are outside of the cache, and everything becomes ultra slow, so. And this depends on the processor you have, how big is your cache, that details, limitation details makes it very difficult to estimate, knowing the number rupture, how long it will take. So what we did, this was very recent breakthrough on, two months ago, one month ago. So what we did was inside each task, and you have, I don't know, one million ruptures to compute, you take a sample, so you make a sample, so the bigger calculation, you measure the time it takes to compute this sample, and then you estimate how long it will take to run the entire calculation. Then when you have that, you can essentially have an association between the weight of the calculation, how long it takes, so you can decide how to split, and the user can say, I want to do tasks that are 10 minutes long, or half an hour long, or two hours long, and the engine will understand, more or less, how long they will take, it will split accordingly. And this is really what solved the problem of the slow task. Then we made another improvement, so now the engine is so smart that does not require the user to set the daskudation parameter, but is able to figure out what is the right task duration. Of course, the engine is not perfect, so maybe sometimes it does not figure out the right task duration, so that parameter is still user configurable, and it's very important that you give the user control in some cases, because now essentially what I did, I have a system that works well for all the model that we have in the global mosaic. So global mosaic, I mean we split essentially the world in 30 regions, I don't know, North and Africa, South America, Canada, Australia, these 30 regions, and we have source models for all these 30 models, and we have also automatic tests that run during the night, and run samples of them up, because of course the model is very large, it will take weeks to run one, so run all that, no, weeks, months to run all of them, so we run simplified version, but we have tests to make sure that the software is able to run what we need to run, and we keep working even in the future. So, now I am nearly at the end. As I said, the data transfer issue was solved in the output using zero-mic Hugo, which is very good, so I recommend it. In the input instead, the trick, we are using a shared file system, NFS for instance, but you can use the shared file system you want in a cluster. So, the idea is that if you store your input in, for instance, in the HDF5, in a location, in a directory, which is visible to the workers, the worker can read directly from that directory instead of transferring via a bitmq the data and it's solved the problem with the input that the transfer. And finally, there is a note with the, if you have a taskplating procedure that produce a lot of tasks, you might be careful because if you produce too many, it will run out of memory anyway, so you must be careful. Problem we had with the memory occupation in the past and even now we always have memory issues. Memory issues are not necessarily always bad, sometimes it's good if you run out of memory early because what happens is that the scientist try to run a calculation for the first time, he probably is not the person that made the model in the first time, doesn't know which are the right parameters, try to run it and doesn't work. It's better if you run out of memory after 10 minutes rather than keep working without running on memory before 10 days and then the end you discover that it can't complete because it's just too big. So it could be good to run out of memory early. Yeah, memory location is very important actually also for the performance point of view because I had this idea that if you do complicated mathematical computation like sinus, cosinus, square roots, this complex, instead no, typically that is ultra fast what is law is allocating the arrays. So, be careful about that. How to reduce the required memory? There are few tricks which are well known and I would say try to use an umpire arrays as much as possible because if you have an umpire arrays you know how much memory will take just to do the multiplication shape of the array and if it is a 32-bit or protein point, 64-bit protein point, you know so use as much as possible umpire arrays. Also for instance recently we were running out of memory when computing statistics with the other course and so I solved by using site by site algorithm so for each other I don't know 100,000 sites for each site I do the computation. This is not the most efficient way actually before it was done in blocks, blocks of sites which is faster but at the end doing it in blocks of sites or I had two options or I ran out of memory or I used too many tasks because of course if you have more tasks the blocks are smaller but there were so many tasks that at the end everywhere was ultra slow so at the end went back to use a site by site algorithm so you have to see case by case. At the end another way to solve the memory issue is if you have a parallelization framework which is able to yield partial results this is an example you want to accumulate results from okay you have a processor for each source if you do the result of this source then you accumulate for instance on a list and if you are able to yield the partial results first then you avoid to accumulate everything in a huge list. In order to do that you need a parallelization approach that is able to do that unfortunately we don't have it so easily so I had to implement it in a with a zero MQ but if somebody with a core developer in Python and as can touch this I think the multiprocessing modules should work in this way so multiprocessing if there is a return return one result if there is yield returns partial result that's an idea they had 10 years ago they feel very natural to me I don't know why nobody is using it but anyway it worked out a lot for the memory and with this I'm done, I'm ready for questions. We have four minutes for questions so we have some microphones there and there so please line up behind them and fire your question away. Thank you for your talk. You mentioned that Dask was missing a feature compared to another framework. You mentioned that you talked to the main developer and they have no problem. Can you repeat, I missed the subject. Which is missing the feature? When you were talking about zero MQ I think you said that Dask was missing a feature. No, no, no, no zero MQ is the solution as the feature that is the multiprocessing module essentially in the standard library that does not have the feature or yielding partial results that I think would be useful. You did not speak about Dask, the framework. Dask, yes, I spoke. The feature of Dask that was missing was the revoke functionality, the ability of skill. Revoke, the ability of killing my calculation without killing calculation by another guy. And that was available in zero MQ. Ah, it's now available. It was one year ago that it failed. Okay, it was in zero. Okay, thank you. Okay, sorry if you misunderstood, it was by Dask. Any other question? Did you ever look into using Apache Spark for a similar framework? And if so, why did you choose to use your own system? It's too big Apache Spark for us. For me it would be more than the high performance we have the middle level. It would be too big also from the point of view of infrastructure. Our system is very busy and would be a big change for us. So we try to do whatever we can do in the small. Okay, any other question? Well, one question I have is, have you tried to use MPI at MPI for Pi? Yeah, I'm aware about MPI. We do not have a cluster with MPI, so we cannot test it essentially. Okay. And again, it would be too difficult for us. We are trying to do middle. Okay. Middle level. And... Yeah, yeah, I have another question. Did you have a look at Apache Parkit? Parkit, I know about Apache Parkit, but we did not have a look. We did not. Actually, you should also consider that I spending 99% of my time fixing bugs and the mailing list, the training issue. And the time we have for changing the distribution mechanism is very little. So I made my talk on that because it's an interesting talk for the developers, but it's not really... Most of my time is not spent there. Okay. Any other question? Well, what should you do if you realize that you live in a city which is on one of your maps, like the fatalities map, that kind of stuff? If you are curious, we have... Sorry. Or that building over there. And there is this application that is in the GEM website where you can go there and see your city. You look in Bern and see what is the hazard on... You live there. I live there, okay. And also we have actually risk profiles for each country in the world, more than 200 countries where we... We give you information like which was the biggest earthquake recent... Okay, maybe I don't want to know. How many people are at risk? There is everything in the GEM site. Nepal. And Nepal is one of the more at risk countries. Okay, let's leave it like this because we get depressed. So let's thank Mikael again. And the next speaker can come.