 Hello everyone, I'm very pleased to be here today and I would like to thank the organizers for inviting me and giving this talk about the reduced Google metrics analysis of Wikipedia networks. So, I'm not used to this format of presentation and I'm sorry not to be with you today. This is a last minute problem that avoided me to come. So, today the work I'm going to present has been made in collaboration with Samer and Zand, Dimash Epeglonsky and Klaus Fram. This is most of the results I'm going to present come from the PhD of Samer. And I think you already had a very nice introduction this morning of Klaus about fundamentals of Google metrics and mostly of the, I guess, of the reduced Google metrics analysis. So, this talk will mostly focus on how to live a life. You don't see my slides? Oh, sorry. Let me try and leave it, so let's try again, I'm sorry, share a screen, share a partage. So, that was better now. Cool. Thank you. Thank you. So, most of the talk will discuss on how to use Google metrics tools to extract some really macroscopic and interesting information from the Wikipedia encyclopedia. So, as you all know, Wikipedia is a very large free collaborative read in the encyclopedia. You, it's now very interesting and we have a lot of very good information in it. And what is really interesting for us is that it relies on hyperlink structures for all articles. All articles are linked together with hyperlinks. And so, for instance, the web page of France directly points to the web page of Western Europe, which might as well point to the web page of England or to any other web page. So, we are going to leverage this hyperlink structure and build the so-called directed network, where the vertices are the articles of Wikipedia, each article has a topic and the name and the edges that interconnect these links, the edges that interconnect these vertices are called, are given by the hyperlinks. So, Wikipedia can just be directly mapped to a directed network of topics that has a shape which is scale-free. Here we will concentrate on a couple of Wikipedia editions. You all know, I think, that Wikipedia has been written in various language editions. So, you have English version, which is the most commonly used and which has the largest set of contributors worldwide. And as you can see, the English version, the English edition of 2017 has over five million nodes and 122 million links. So, this is getting a very large network. You have other editions that are pretty interesting and more or less large, depending on the set of contributors. So, you have the French edition, the German edition, the Arabic editions. And we have looked at all of these and how we could extract some meaningful information from these editions. Okay, so one, of course, like we are today in this very nice conference, we will leverage the Google Matrix analysis of this large, direct network of Wikipedia. So, for these Wikipedia networks, we can build a Google Matrix. As you all know, this Google Matrix represents a mark of transitions of a random surfer that will go with probability of, let's say, half an alpha percent of the time used the hyperlink structure to travel on this hyperlink network. And with one minus alpha proportion of time, we go randomly as well. So, from this Google Matrix, it is, of course, well known that you can use the eigenvectors corresponding to the largest eigenvalue, which is called the PageRank probability vector, to capture interesting nodes. These interesting nodes are called central nodes in Wikipedia. And several studies in the past have really looked at how Wikipedia can be understood using PageRank metrics and Google Matrix analysis. So, a couple of works exist. Some very interesting ones are the ones where you have, where the researchers have extracted the ranking of historical figures over 35 centuries. And another one, which is in good agreement with the heart ranking. And another one, where the ranking of world-wide universities has been captured, and which is really in good agreement as well with the Shanghai academic ranking. So, of course, here we only look at the hyperlink structure and extract the ranking only for the subset of nodes which look at either universities of historical figures. You can as well look at Wikipedia by creating an inverted network, directed network, where you do a transport version of the agency matrix. And therefore, and then you can compute on this the largest eigenvalue that we will call the ChaiRank in this thing. And in this case, you will extract good diffusion nodes and the nodes which are good for diffusion capabilities. So, in terms of a small PageRank example, I'm going to start introducing the various studies we have done. And one of the studies really looked at geopolitics interactions among countries, either worldwide or either at the scale of Europe. So, here on the map, we have colored various countries, which are the top 40 countries when looking at the PageRank index for the English version of Wikipedia. Here, the first, the largest, the most important country in English Wikipedia is the United States. Then the second is France, then you have the UK, then you have Germany, Canada, et cetera. So, we have grouped all these countries into sets that are represented by the colors. So, you have the English-speaking countries, which are in orange. You have the former USSR block in blue. You have in red, Europe. You have Arabic countries in the top 40, which are in yellow. And you have Southeast in pink and in purple. You have Chinese block. And, of course, you have Latin America in green. So, here you can have a good ranking of all these countries with the PageRank. But, of course, as you may already know, the PageRank index and the PageRank probabilities really depend on the language editions of Wikipedia. There is some cultural bias here. And the PageRank index varies depending on the English or Russian or Arabic edition you are looking at. For instance, in the top table, you will see that, as given before, for the English Wikipedia, United States is ranked first, while for the Russian edition, it is, of course, Russia. So, in order to find some cross-edition ranking, we have built a TTA score, and previous authors have worked on this as well, and to offer a global ranking across several editions of Wikipedia. So, this global ranking is looking at the top 100 nodes in one edition, E of Wikipedia. And we are capturing this sum of 101 minus this rank to, and we sum that over all editions. This gives us a ranking measure, which offers where the largest values give us the most important node across all editions. We can, as well, of course, compute an average PageRank probability and build from that a different ranking. But from our experience, it was much more interesting to have this TTA ranking than this K average rank that is pictured here for a set of 40 pages. So, in this slide, I'll show you, we have selected as well another subset of nodes in Wikipedia for our research, and we have calculated a set of the TTA score for all the painters that are enlisted in the English Wikipedia. And we have created the TTA score over those seven editions of Wikipedia. And we have picked the 40 most important ones according to this TTA score. So, you will see in this list of 40 painters, very famous painters such as Vinci, Picasso, Van Gogh, Ambran, Rubens, and others. I have shown you the two sets, the set of 40 countries, the set of 40 painters we have extracted, which this extract versions of 40 painters is more related to the seven editions that we are looking at. And based on these subsets, we will try to find a better representation of the interactions among all these painters, all these countries, and how they interact together using the theory which is called Reduce Google Matrix. So why? So Klaus, I think, has already introduced the Reduce Google Matrix theoretical foundation. The idea is to use this powerful tool to create a subnetwork vision, like a thematic view of the full Google Matrix for a set of articles of Wikipedia. For instance, I'm interested in painting and I want to know what are the, from all those 40 very important painters, how do they influence each other? This is a question I can ask myself and how they are related maybe to world countries, like the 40 to the 40 top countries I've identified before. So is everything clear or did I miss? I don't see the audience, so it's difficult for me to have any feedback. Yeah, we are getting you perfectly. Okay, thank you. So the little red circles here represent simply the sub-set of articles that I'm looking at, and then from that we are building what Klaus has introduced earlier, which is the Reduce Google Matrix. So this Reduce Google Matrix, I'm just going to quickly go over it again. So we consider here Reduce networks of NN nodes. Those NN nodes are the ones that we are, which we are focusing on. So for instance, those 40 painters or those 40 world countries, and we investigate the construction of a reduced matrix, Google Matrix, and therefore we have to reorder the regular matrix, the regular Google Matrix G, where we put on the top left corner all the elements of the NN time NN matrix GRR. On the lower right-hand side, we will put the NS times NS elements that represent the scattering matrix. This scattering matrix is the matrix that represents all the other nodes of the network, which are not the NN nodes of interest. And then we have on top right and top left corner, the different probabilities, which made us go from, which help us travel to the other nodes which help us travel from the reduced set to the scattering set and back from the scattering set to the reduced set. And accordingly we will rank, from G we of course can compute P, which is the page rank probability vector, and that way we order the same way to have on the top the NR first probabilities related to the NR nodes of the reduced network and on the bottom we will have the NS nodes of the rest of nodes. And from that, since we want that to calculate a new reduced Google matrix GRR, such as of course the use of GRR produces the same steady state page rank PR. And from that, and from the definition of G and the reordering that was made before, we can define GRR as being the sum of GRR plus GRS times 1 minus GSS to the power minus 1 times GSR. Close to you I think that NS is of course too large for a direct evaluation of 1 minus GSS to the power minus 1 because we are just looking at a few as a really small subset in our nodes. So he has proposed the following numerical evaluation by saying that it is possible to invert 1 minus GSS, assuming of course that GSS is not singular, by saying that we can extract lambda C which is the leading eigenvalue of GSS using a power iteration. And from that, extract as well PC the projector onto the eigenspace of lambda C and its complementary projector QC. And from that it is possible to compute those two parts of the inversion. So the first part, the left part, PC over 1 minus lambda C represents the projector component. And on the right hand side you have the complementary projector component. So when you integrate this derivation into the definition of the reduced Google matrix you can see that GR is simply a sum of three components. The first component is still GRR. The second is the projector component which is the multiplication of PC over 1 minus lambda C times GRS and GSR. So this is a part which is a lot related to the page rank projector as you will see. And you have GQR which represents the indirect interactions through the rest of nodes and where you have no impact of the largest eigenvalue of GSS. Now we illustrate these components of GR. And the first illustration I'm going to give you is to show you what is the reduced matrix we get for a set of 27 European countries selected in Wikipedia. So here we have selected the 27 European countries that were composing Europe for the 2013 Wikipedia edition and we have selected them and looked at what is GR in the Wikipedia network of the English edition. So on the left hand side I've plot the reduced matrix GR and on the right hand side the projector component GPR. As you can see both matrices are really dominated by the projector component and among almost 95 to 97 percent of the total column sum of GR is given by this projector component GPR. So this GPR as I didn't tell you but on the left hand side all the rows are ordered by increasing page rank index so the largest page rank probability is on the top for France, Britain, Germany, Italy and disorder and the bottom as well on the X axis for all the columns are ordered with the same ranking. So you can see that this representation is really that the projector component doesn't give us any really new information compared to the regular page rank probability vector and all the interesting information for us is captured mostly in those three to five percent of the total column sum which is given by GR and GQR. So if you look a little bit at those matrices the sum of all weights of these matrices for this 27 EU network and for as well the 40 top worldwide set of countries so the set of countries I've given you before which is worldwide you can really see that the projector component offers a big chunk of the power of this matrix if you sum all elements in the matrix and you normalize by the number of columns and only a small part is captured by WRR and WQR which correspond to the sum for GR and GQR respectively. So now what can we see in GR and GQR for this 27 EU country network? Here we have innovative information I think really only in most of the innovative information is in the right-hand side figure and is in GQR. So GQR here we didn't picture the diagonal terms because they are pretty important and this blows a little bit the color scale. So as well like before the color scale red values are the maximum values and blue colors are the minimum values. On the left-hand side you see the direct links that are represented by GRR which is a direct view of the regular Google matrix and with this regular view of the Google matrix you cannot, since its column normalized compare what happens in between columns but on the right-hand side for GQR you can compare what happens with the different you can compare column-wise what happens for one line. GQR represents, I want to say again the scattering of represents all the path represents in the Markov chain the contribution of the random walks that go through the scattering matrix and so it represents not the direct interactions between the node of our network but the indirect interactions where all the possible travels that go through the scattering matrix and go back to the node of interest. So in order to understand this a little bit better what I can say is that, for instance if I'm interested in the, if I want to see a strong interaction that is indirect and that does not really exist in the direct links on the left-hand side I could capture what happens here for this red point. This red point means that when I am on Finland I have a high chance to use the scattering matrix and an indirect link to go to Sweden so there is a strong interaction between the Finland and Sweden. This is not as well captured with the direct interaction between Finland which is here and Sweden which is here it is not among the largest ones in the full matrix left but it's more visible on the right-hand side. Here as well you have a very strong interaction between Belgium and Luxembourg which is totally meaningful and as well between France and Luxembourg and you have as well a strong interaction between Sweden and Denmark and another between Sweden and EE. So there is a lot of information in both matrices and mostly on the right-hand side and on the one on the right-hand side to better... Of course these indirect links that you can see with the right-hand side here capture as well the cultural views of different language editions for instance if you compute a GQR for the English version of Wikipedia and on the right-hand side you capture for the same 27 new countries you build the matrix GQR and D for the French Wikipedia you see that the indirect links are not exactly the same and don't have the same magnitude. You still find a strong influence between Finland and Sweden and so the question we are asking ourselves is there a fundamental difference between those indirect networks for different language editions or are there some common traits that represent some common knowledge that is true in different editions of Wikipedia. So to investigate that we have created what we call networks of friends. So the network of friends is what? We say that a top friend of a country J is obtained by ranking all the countries in column J by descending value of column J in the matrix of interest and we pick the top four friends for instance to build a little directed network as represented on the right-hand side of the slide. So on the left-hand side we have pictured GR for the English Wikipedia and have created a ranked selected five countries in Europe which are Sweden, France, Great Britain, Spain and Poland these five countries are identified as the most influential ones in terms of page rank for the set of countries that have entered Europe at the same time. So for the founders we have friends then you had the Great Britain joining then you had a set of countries joining Europe with Spain and another set with Sweden etc. So for these five important countries pictured with the largest circles we have selected, we have looked into GR for the top four friends in GR as defined before and we can see that Sweden's top four friends are represented no, France's top four friends are given by Spain are given by Great Britain by Germany and Poland and this network is really really represented, is really dominated by the countries which have a high page rank this is completely true since GR is dominated by the page rank description and the top countries, the top friends of a country J are likely to be the top page rank countries in the set of 24, 27 new countries now when you build the same type of network of friends from GQR&D you see much more diversity because you have extracted this important eigenvector and you only see the contribution of the rest of the network and it is interesting to see that by building the same top four friends for English Wikipedia or for French Wikipedia by selecting the same set of important countries so we have selected as well Spain, France, Great Britain, Poland and Sweden and we have built exactly this network the same way than before we see with the black arrows that they capture first important notes that are not necessarily the ones that have a high page rank in general this is because GQR&D has no page rank contribution in there these plots have been plotted with an automatic tool from Jeffy an algorithm which is called force-direct layout which defines where the nodes are and groups together the nodes which are more highly interconnected what I didn't tell you yet is that the red arrows represent the edges that have been computed as the top four friends of the friends of those important nodes and we have added these recursive friendship edges until no new vertices added to the network it means that red interactions are friends-of-friends interactions and black edges are direct friendship interactions with the origin being one of those important countries so you see what is interesting is that even if you are looking at two different editions you can still see that there is a clear community that is represented by that you can find in both types of editions you can see that friends is most of the time related to Benelux countries and Neverland you can see that the red countries are in Poland as well as are most the same etc. and you always see that Portugal and Spain are grouped together so there is some common knowledge that you can find in both editions and this is true for other editions as well I'm not going to give you other different examples so a little twilight those cross-edition friendships we have counted how often in all the five editions we have for five editions we have investigated which are English, French, Russian, German and Arabic we have counted the friendship interactions that exist in all five editions or in four editions out of five or in three editions out of five there is always a friendship relation between France and Belgium or between France and Spain there is always as well a friendship relation between Great Britain and Ireland or between Poland and Czech Republic and you can read the rest of these results in this table so there is some really common knowledge in these structures of networks so I will show you as well another type of network of painters since I've introduced painters before so what we have done here we have selected 30 painters to create a reduced network for these 30 papers we have selected six that belong to different painting movements so we've looked at Cubism, Phobism, Impersonism Great Masters and Modern Painting we have associated a color code and then we have picked these 30 painters with a theta score ranking and we have identified for each category the most important page rank painter which is Picasso for Cubism, Mathis for Phobism Monet for Impressionists and Da Vinci for Great Masters and Dali for Modern from that we have created as well the network of painters and we have created the network of top four friends, the top three friends we have created the top three friends and similarly the black arrows represent the top four friends of the leading painters of each category and the red arrows represent the friends of friends interactions that can be obtained recursively until no new vertices added to the graph so what is interesting to see on the left hand side we have English Wikipedia on the right hand side we have French Wikipedia and we can clearly see that the chronological development of these painting movements is really coherent on the top you have an orange the Great Masters which come from the from the late middle age Renaissance century and they from this the set of orange Great Masters is densely interconnected with Da Vinci being influencing Douga and other leading painters of the Impressionist movement Impressionist movement painters as well very densely interconnected and they as well really influence the blue group which is Phobist and then you have a nice indication between Phobism, Modern and Cubism painters which of course are more closely interrelated and you can see the same type of developments on the right hand side for the French Wikipedia so it is nice to see that two different cultures that have both the same type of knowledge of course there are some local differences but the macroscopic view is really interesting to see then we have looked at building a friendship network that looks at the interactions between painters and countries of course to do that we have built a subnet of a reduced network which accounts for both the 40 set of painters and the set of 40 countries and we have extracted the top three country friends for each of the 40 painters we have identified before and to do that we have not used only GQR alone we have used the sum of G or R and GQR so we have some direct interactions within G or R and the contribution of the indirect path and this we've done it for English Wikipedia the black and rose represent interaction where the local where the direct interaction is more important than the indirect interaction while the red ones represent the opposite so you can see that France and Italy are really really really central central in this network and that a lot of those painters are related to art development in France, Italy or Spain we have built the same network for the French Wikipedia as well and the French Wikipedia you see as well as central position of French, Spain and Italy but Netherlands seems to be more central than it was in the previous one so these are interesting views as well so now is there any question in the audience related to this part because I'm going to move to another part for the questions no you can move on okay I can move on so the next part is mostly related now now that I have identified my subset of nodes my network I'm able to picture it and to capture the direct and indirect interactions I would like to know how does a relative link variation will impact the reduced network structure what happens if an interaction grows stronger in the network and what are the nodes that are going to suffer from it and what are the interactions that will develop based on this change so we have developed a sensitivity analysis on this reduced network and on the reduced Google metrics from that what we do we look at what happens if there is a local change on a given relationship for instance I take the relationship between nation J going to nation I in the reduced network this time I look at GR and I will modify with a slight variation the element of GR at location IJ and I will re-normalize again the column J to keep good properties for this new changed matrix and I would calculate the modified page rank value with this GR and then I will observe the change of importance of nodes in the network by calculating the logarithmic derivative of the page rank probability for a given node K it can be node I it can be node J it can be node K but here we mostly look at node K which is not necessarily the link that we have changed so this measures for us this is what we call the sensitivity of a nation K to the link variation J to I so looking at that we can have various examples of results that we have found having this type of analysis so for the 27 European countries dataset and for the for we have looked at what is the impact of an increase of the of the interaction between Italy to France so we have changed the link from Italy to France and we look at what happens for the other 25 countries in terms of ranking and what is the average sensitivity variation that we observe for three different Wikipedia editions so here we took the value of D and we have averaged it over the of the three editions for EN, English, French and German Wikipedia so what we can see is that for this analysis that the country which is the most affected by this increase of the cooperation between Italy to France is Slovenia it is true that Slovenia has a lot of economic exchanges with Italy and if Italy increases its communication and if Italy increases its relationship to France it is most probably Slovenia that will see its cooperation with Italy decrease and thus will not be able to we will lose in terms of importance in this global ranking in the network the second country which seems to suffer the most is Greece which has as well a cooperation with Italy in terms of economy or history or geography now we have looked as well at the set of 40 worldwide countries and we have looked at the impact of an increase of relationship between China and the United States and we have mapped on this figure the the change of the sensitivity that all the other countries will observe if this change happens so here the lower values are in red and larger values are in green and medium values are in blue so the countries which would be mostly affected by an increase of collaboration of flows going from China to US would be of course the border countries which have more exchanges with China like India and Russia and the ones that would benefit the more are the ones that are located close to the United States because they are of course cooperating with the United States more often what we can as well observe with this type of analysis the sensitivity analysis is that it's possible to identify countries, clusters of countries that really function together to capture the fact, of course it is well known that Sweden, Denmark, Finland work together and on the top line we have looked at the average sensitivity for link modification going from any of these Nordic countries to France or to Germany on the top it is going to France and on the bottom plots D and E it's going to Germany not the opposite, ABNC is going to Germany and D and E is going to France so you see that every time one of these countries increases its relationships to an important European countries the other ones will suffer because they will reduce their cooperation with this country that is a bit leaving this cluster of countries then we have looked as well as the average sensitivity of countries to painters that calculated what would be the sensitivity of the 40 countries if you increase the relationships from Van Gogh to the Netherlands so you all know that Van Gogh is originally coming from the Netherlands and that he has spent only the four last years of his life in France but he was very productive years and this is where he really didn't get famous of course at that time but most of his masterpieces were drawn there and if you would artificially increase the interaction between Van Gogh and the Netherlands in this reduced network you would scarry see that it's France that would suffer from this increase of interaction because it is highly tied to Van Gogh as well in Wikipedia this is when you increase sorry there is a little typo here it's the sensitivity from Van Gogh going to France I don't think this is the right picture, I'm sorry this is not the right picture this is I think the sensitivity, the caption is wrong this is Da Vinci to France you can find all these updated pictures figures in the paper that have cited at the end of the talk but here is the increase of interaction from Da Vinci to France so of course you will see that Italy is the country that will suffer from this change so here we have looked at like unidirectional sensitivity and it is possible of course to look at what happens when the link is changed from I to J and it's modified as well from J to I so we have simply calculated B-directional sensitivity which we call the two-way sensitivity to measure the sensibility of a nation I to the changes in both directions on the link I to J and link J to I and we have observed that for instance the relationship between painters and countries so here on this plot we have represented the diagonal sensitivity of the top 20 countries of top 20 painters so on the left hand side we have the top 20 painters of the 4D top category I presented earlier and on the bottom we have the 20 most important countries that we have identified worldwide and here how can you read that is that we have calculated the two-way sensitivity for the top 20 countries on the bottom when the interaction between this country and the painter on the left hand side is modified so how to interpret that is the following is to say for instance that France is mostly impacted by as well Vinci if there is a link between Vinci and France that is being changed the same way for Spain you have Pica which is very important as well in terms of leading painting figure and Michelangelo is as well pretty important for most of the countries that are presented here so this help us capture very easily in a synthetic way really the importance of some specific painters in different cultures and in different for different countries another different little analysis which follows from the two-way sensitivity is what we call the relationship imbalance between two nations here we would like to know on the relationship between two countries A and B which one is the strongest nation which one has the more influence on the other one when it is when there is a link change in both directions between both countries this relationship imbalance is calculated between the two-way sensitivity falling for country A observed at country A minus the two-way sensitivity calculated as well for a two-way variation between A and B and observed at point B observed at nation B and this relationship imbalance can be interpreted in the following way if this F between countries A and B is positive it's B that is the strongest nation and it's going to lead and if FAB is negative A is the strongest nation and we have plotted these outcomes here for the case of the 27 EU network what we can see in terms of relationship imbalance analysis so it is of course we have only presented half of it because it's a perfectly symmetric matrix X axis represents country A Y axis represents country B the blue values represent negative values red dish orange dish values represent positive values so here we can see that France is really a country which is dominating in the this is English Wikipedia France is really dominating all of the countries in this sensitivity analysis and Germany is another really important one Austria and Italy is an important one as well and Austria a little less we have done the same type of development for the world wide network and here you can see that US is really the strong and so strong that the variation of other countries influence on each other is pretty much them by this so you have US which is important in English Wikipedia France which has a strong impact on Germany so this concludes a little bit my talk so here I wanted to show you is how Google matrix can be nicely leveraged to analyze Wikipedia it really offers a nice framework to automatically learn really a embedded information and like having a macroscopic view of some elements which are really interesting most of the results have shown you are results that are of course pretty obvious we we wanted to check whether the type of information that we could obtain was meaningful and reasonable so what is nice to do with Google matrix analysis is that you can of course capture important results which page rank or their matrix across editions but you can as well exhibit interactions within a subnetwork like in this thematic view of this directed Google matrix network with the reduced Google matrix analysis and what is interesting as well is to understand the influence of links and nodes on the network with the sensitivity analysis which was in the last part of the talk. In terms of perspectives Google matrix has very nice properties to become for me a major tool for artificial intelligence and automatic information extraction for such large from such large and very large information networks but therefore I think we still have to be able to automatically extract the subset of articles the subset of nodes that we want to investigate further so an automatic procedure that could be able to extract the subset of articles that for a given study and create this subnetwork would be really interesting to go beyond and have an artificial intelligence leverage more efficiently this network and as well you may have seen Wikipedia is changing every year you are adding a lot of new articles new links etc and how to capture this variation into the reduced network at a reduced cost that would be very interesting and trying to understand what are the parts which really are important in this evolution or not. So most of the works I've presented here are to be found in this recent literature so if you need some further explanations more developments don't hesitate to go and look for this paper the two first are free access. I'm free for your questions. Ok I try to go back to not sharing my screen once again. I have two questions if nobody else in the audience is. Can you just go back to the slides up where you show the indirect interaction between countries? Yes direct interactions between countries. Is that Finland and Sweden? Can you see my slides now? Because I stopped the sharing. Ok this is what I'm thinking. I can't find, I'm sorry I have some technical matter I can't find this one here. Ah I can find it here again. Yes I do it right away. Ok. We are not seeing them. Oh no. So in the previous slide? Yes. Yes. So probably one back again. When you notice that Finland and Sweden have this stronger indirect relation. I notice that Finland and Sweden is here and here you have one of the strong terms of the column. Even in the previous one. I notice that most red notes are red elements are above the diagonal probably because of the way you sorted rows and columns. Do you have an interpretation for this? About the way we sorted rows and columns. Countries are sorted by importance right? Yes by importance I agree. So do you have an interpretation of why Finland has indirect relation to Sweden but not the other way around? It means that you my interpretation is the following is that you have an important an important number of paths linking Sweden and Finland indirectly and you have a reduced number of direct paths compared to the all possible paths that are going out of of Finland which one was it? I mean that here you have a lot of subjects which are integrated in Wikipedia you can have two articles that are interconnected related to issues related to politics related to agriculture related to movies and theaters and a lot of different things. These are cultural strengths that are captured by indirect links while direct links are the ones where you have a direct link on the page of Sweden going to Finland. So the fact of having a culture which is closely intertwined for the two countries I think increases the rate of indirect links that go through this gathering matrix. My observation was more that the indirect links tend to be more important from less important countries to more important countries the other way around because the red dots are more about the diagonal than below. And a sort of related question is about sensitivity when you measure the sensitivity of adding a link from Italy to France I think. Why are all the numbers negative? You mean that the only now there's one that is positive it's France you see and it's taxing most of the values. Well, France must be positive because when you add a link to a note because that note increases its score but you mean that all the other ones are negative or negative. But this is an average value as well over three editions but most of them are negative this is true. Do you have an interpretation for this? It's like the only country that gets more important is France. Yes, because it's the one that you are pushing you increase the probability and then you renormalize so it makes sense that the other ones are negative because you have to compensate for this increase. Okay, thank you. It's normalization that you can see here. Yes. Would it make sense to have relative values because we often see that small countries are underrepresented and it would maybe make sense to have a relation between the population and the quantity of links. You mean normalization by a number of inhabitants. Did you get this? No, I didn't hear it, sorry. So someone was commenting that it may make sense to normalize the values by population because the countries have different sizes so it may make sense to have a normalize. Did you try this? No, no. So I think it's a good suggestion. More questions? No, let's thank the speaker again. Thank you.