 So before starting, I would like to say that in a couple of slides I'm going to use menti.com so that we can have a little bit of interaction and I will also have the chance to get some feedback from you and not only share the information for a BIP scholar. So just take this into consideration and maybe have menti.com tab open in your browser. So as you can see in the title of the presentation, the service that I'm going to discuss is related to academic profiles. So why do we need them at the first place? So in general, we know that researchers while they are working, they are producing a lot of outputs. For example, research papers, data sets, software, and it is very common that they need to be evaluated based on this output for various reasons. For example, for career advancements when they are candidates for a particular open position in a research organization, when someone is going to evaluate them for some professional achievements, for example, to give them a prize or something like this, an award. So it is a very common activity, a very common task for someone to go through the various items that belong to the output that the researcher is producing and then try to evaluate their productivity, the impact of their work or very similar aspects of their career. And of course, an academic profile is something like a summary that tries to provide an easy way for evaluators to go through the various items of researchers, the various activities that the researcher had participated. But even if we have academic profiles, the work is not that easy. And a very important problem is that the scientific output is being produced with an exponential growth. So we have more researchers than ever in the history of humanity. We have also this culture around the notorious publish of our Paris trend. So a lot of researchers have the pressure to publish more. And also we have other actors in this landscape that are trying to grab the opportunity, let's say, like the predatory publishers, that are offering quick and easy ways for everyone to publish, even if sometimes the publications that are produced in the journals are of questionable quality. But in any case, all these are just factors, just some of the factors that contribute to this increase in the rate with which the scientific output is growing. And of course, this exponential growth is related to many problems. For example, it is hindering very useful tasks, which are related to both research assessment, which is the subject that I will discuss today, but also other tasks like those related to scientific knowledge discovery. And one of the ways that some developers tried to, and researchers tried to alleviate problems like this was to try to use to exploit impact indicators for the research work. So the main idea is that in theory, it is possible to estimate to some extent the scientific impact of an article if you analyze how many other articles are talking about this article. And of course, the more articles that are talking about this, the most the expected impact would be. And based on that idea, various indicators have been proposed and have been used by relevant platforms. We all know, for example, citation count, which is a simple and popular indicator of scientific impact based on citations. And we know that it's being used from many academic search engines like Google Scholar, for example, to help researchers prioritize their reading. But also going to the researcher level, there are some indicators, some researcher level indicators that are based on citation count like the notorious age index or the item index that are provided again by Google Scholar. And these indicators in many times are used both from the researchers themselves to self-monitor their performance, but also from people that are using this to facilitate the process of research assessment, not always in a good way, because sometimes they are using them as a short spot for the evaluation and they over-reliant them. So based on this approach, a common structure for academic profiles that you can find in different platforms would be something like this. You have sometimes the effort of the researcher, the name, some contact details and details about the affiliations, a list of publications, sorry, I'm getting some noise, yeah. And also a list of citation count-based indicators usually on the profile page. So this is more or less the concept that you can see if you search around for platforms that support this kind of concept. So before going to why this is not good enough and some problems that could someone think with this approach, I would like for all of you to go to menta.com and use the code that you can see on the screen, 24, 15, 51, 19, so that we can have a first quiz. Let me start. So first of all, I would like to see which are the platforms that you know that can, but they offer a researcher profile, maybe some platforms that you already use, for example, or you have heard about them, not only open ones, but as some of you already provided some proprietary. I will give you some time. I know we are a lot of people, so I will write. Okay, it seems that we have various that you already know. Of course, a lot of you mentioned Google Scholar, Research Gate, Web of Science, Orchid, a very important one, but also Kudos that is focusing on contributions. So yeah, okay, there is, I mean, here the heterogeneity in the responses also is an indication that this particular field has a lot of options already, and has been developed heavily. So then I have some questions. So we mentioned that one very common content of this profile is a series of a group of citation count-based indicators. So I have a couple of questions that I would like to see. How much do you agree with these statements and how much you disagree? So the first question is, so is there any problem that I don't see anywhere? So yeah, I can see now the first responses. So the first statement is, is the citation count-based indicators a convenient way to get some insights about the impact of a researcher's work? Let's see how this goes. The second one is about if you think that they live important aspects of a researcher's work uncovered, then if these indicators can always with clear semantics. And finally, if you think that these indicators are often not often, often misused during research assessment. For example, when evaluators are very lying, then okay, I think that we have already collected a lot of responses about 30 responses, and it seems like a lot of people, maybe almost most of the people believe that indicators for impact that are based on citation count provide some useful insights. But on the other hand, it is also evident that they cannot cover the full spectrum of researchers' activities. And also most people agree that the semantics of these indicators are not always clear. For example, what does it mean for me to have an age index about 15? Is it something related to my productivity, to the impact of my work, both of them? How is this different from the I-10? Are they capturing the same thing or maybe something slightly different? Or these are not always evident based on your answers. And until now, I agree with everything. And of course, it is more or less agreed from by anyone here that these indicators are often misused during research assessment tasks, activities. Okay, so it seems that currently there are some problems with these academic profiles. And of course, if we would like to focus on the problems, we will soon realize that these problems are either related to these indicators that the profiles are including, either on the fact that most of these profiles focus a lot on publications and not other activities that are related to the work of researchers. So I will try to analyze in more detail some known problems related to these aspects. Then I will try to describe how some of the functionalities that we are building for BIP Scholar are trying to alleviate this type of issues. And finally, I will give you also a quick demo and explain how we are extending this platform and improving it significantly in the context of the graspos project. So first of all, when we are using citation count-based indicators, we should always keep in mind that citation count is an indicator that has a lot of known issues. And these issues can affect the ability of people that are using it to discover valuable research. One such problem, there are also others that I will not mention, but one such problem is that it is possible for a work to have a lot of merit and also a lot of impact. But this work that has a lot of impact could have an indirect impact to the community. So it may not be highly cited from the articles of a particular domain, but because it is cited from a couple of important articles in that domain that those are well cited, you cannot understand if you just count citations that this is an important work that has influenced a lot the respective domain. For example, here we have this hidden gem. It has only one citation, but you can see from the paper that is citing this publication that it is an important paper. So since it has influenced this important paper, someone should be able to also acknowledge the fact that this work has contributed to the field, something that is not easy and is not captured by citation count-based indicators. Then another pitfall is that often people forget that even scientific impact has multiple aspects. So scientific impact is not something that you can easily capture using one indicator like citation count. And also you may capture one particular aspect of scientific impact with that indicator. There are also other aspects that are not captured and based on the application, based on the use case, it may be important, these additional aspects may be of particular importance. Here, for example, I have a very indicative example for people that are searching for important publications in a particular field, for example, in machine learning. And we have two cases. We have an experienced researcher that is visiting the field. So this researcher is interested for recent papers, for recent advances, I mean works that are currently popular in the domain. While we have a second researcher, maybe a student that is trying to draft a survey of the same field for this particular use case, it is of special importance to find foundational articles, those that has been well established in the domain. And of course, although some articles may be also popular and foundational, it is not always the case. And these two properties are not always correlated, 100% correlated. Of course, another thing that we should consider is that when someone is using indicators, for example, for a profile, and these indicators are used in practice to make decisions, for example, to hire people or to give awards, then very quickly, this indicator will start to get a lot of attacks. So people will try to gain the indicator so that their own work, their own profile appear to be more important than they are. And this phenomenon is well known. It has multiple names, one name is the good first law, another one is Campbell's law, but also there is this cobre effect term that describes the same phenomenon. So in general, we should always keep in mind that when you have only one indicator or a couple of indicators that are capturing the same thing, these indicators can be gained very easily and this of course will affect the decisions that you make based on them. If you don't also consider additional context and additional properties of the work of the people that you are assessing. But then, okay, the previous point somehow argues that you need more indicators because you capture more aspects of research impact and also it is more difficult for people to attack a set of indicators than one particular indicator. But on the other hand, you have also the opposite problem. If you are included in a profile, a lot of different indicators and the semantics are not always very clear and also you don't provide a lot of provenance about how you're calculating these indicators on which data, which are the common pitfalls, the limitations that you know about them, on which data you have calculated what is the coverage, things like that, then it is possible that you create more confusion than the good that you bring. And also, you cannot help people avoid improper uses and getting misconceptions about the researchers and their respective research work. Another thing is that impact is not everything. A lot of people sometimes confuse impact with scientific merit, but as we said, for example, with this example about the hidden gems, it's not always that the impact is correlated, highly correlated with scientific merit. So, there are aspects of academic performance that may be difficult to quantify, but these aspects may be also important. And of course, publications is not everything, as we said, most of these profiles are focusing on a list of applications, but we all know that there are a lot of important research activities, like software development, data set production, peer review, teaching that are not related to the scientific articles someone is writing, are very important, are vital for research itself and for science, but they are not always properly acknowledged. And finally, you should always keep in mind that you should not over rely on indicators, because when people are doing this, we know that a lot of problems cannot occur. And if you are interested to learn more about the types of problems, not only in academia, but also in other domains. I strongly suggest for you to read this book, The Tyranny of Metrics, it has a lot of examples, it includes a lot of examples where, when the use of indicators for assessment created problems. A final point that I would like to make is that when we are, even if we don't have all these problems, when a platform is providing a generic profile that tries to summarize the whole scientific track, the whole work of the whole career of a researcher, this profile will not be always easy to scrutinize and to get insights about the performance, the impact and all this that are related to a particular researcher. So, speaking of all these problems, let me describe how we are working towards alleviating some of these issues, because of course we have started focusing on some of them and not all of them. So, let me first discuss our approach about indicators, about impact indicators in particular. This started us a research interest for us, so based on the work that we have done together with some PhD students that I had co-supervised. We made this large survey and experimental study on different indicators for scientific impact based on citations, and we tried to see if we could find that all these are measuring the same thing or something slightly different. And it happened that not all of them were focusing on the same aspect of impact and not all of them were good in identifying different aspects of impact. So based on this work and some others that we have published after that, we had identified a couple of interesting aspects of scientific impact. And this, we have given them some names. The first one is the traditional impact. That is the well known, the well known citation count is captured in this particular, the impacts from this particular perspective. We have something else that we call it influence, and it is very related and sometimes very correlated to the traditional impact, but it alleviates some problems that the citation count has. For example, it considers also in direct influence and not only the direct one, bringing, giving good scores for publications that are hidden gems. And the algorithm that we are using to calculate the score for that is page rank is the same algorithm that Google is using for web pages to rank web pages and bring to the top those that are more important. And then we have another aspect that is popularity. This is trying to alleviate the problem that when you're trying to measure something like a centrality measure on top of citation network. Then you are getting a lot of bias against those publications that are recently published, because these publications do not have enough time to accumulate a lot of citations. They may have already attracted a lot of interest, but because people need a lot of time to write some papers and then for the papers to be to get published after a peer review and all this. Most of the publications need at least a couple of months or in some domains even a couple of years to start getting an adequate number of citations. So if you are trying to calculate citation based measures, then you are getting a lot of bias against recently published papers and the previous indicators that I mentioned are biased against the recent publications. So we felt that we should introduce another indicator we call the popularity and we have an algorithm that based on the experiments is the best one to identify this particular aspect of thing but and we have also another one of the impulse that is just trying to identify how quickly how fast a publication attracted attract attracting the interest after its publication. And based on the results of all this tab is, we created a workflow that gets data from the opener graph creates a large citation network that contains more than 150 million works publications data sets and other research products. And there are billions of citations that exist between them, and that we get from the graph again this citations and the graph gets them from cross from open citations from a lot of different sources. And on this citation network, we have some distributed codes written in Apache Spark, and that are calculating these four indicators that I mentioned, and we are then publishing them on Zenodo as an open data set it is called btb. We are also using them to create some added value services to provide to our users. One of them is the big scholar that I'm going to mention today. And of course, we're giving back the data to the opener graph we include them. So everyone that wants to use the opener graph data also has access to this indicators. And what are worth mentioning is that we don't only provide this course, and because sometimes the scores are not very easy to understand. We also provide classes for each of the research products. For example, something like if this article is in the top 1% or 0.01% of a particular domain or of the whole data set that we have. So we make sure that we do not double count citations that are made from the same article, but from different versions. For example, if we have the preprint or multiple versions of a preprint of an article, we only count once the citations. We take the advantage of the opener. We also have an application algorithm to achieve this. And this is very important. And finally, for all the indicators that we are calculating, we offer detailed explanations we have a particular section in our website that tries to explain, not only the semantics of this indicators, but the proper uses, there are no misuses and documentation about how they have been produced and on which data. So, more or less, this is how we are trying to alleviate problems with indicators. So I think that different types of contributions as I implied, we are currently collecting the publications and the data sets of the researchers we get them from their orchid profile so anyone that has an up to date orchid profile and has all these items publicly available can create an account in our platform synchronize their account with orchid and get everything in their own profiles in our system. And as you know, okay publications and data sets are not enough. We acknowledge that and we plan to extend to cover more types of contributions of researchers. So to add entries from for research software peer reviews that they can also come from the orchid profiles, the involvement in projects, and also we are trying to identify teaching activities. We are not at a very good shape right now regarding this we but we are trying to add additional activities to be acknowledged inside the profiles that we are building. Then, our system also identifies the topics of the works and this topics appear in its entry of the that represents a work and also supports and more importantly also supports the declaration of contribution roles so the researcher can provide the credit classes that describe better the contribution that they have in a particular work. So it is possible to give more context about what they did for this particular work, and also the system supports views of the profile that focus on particular aspects that can show the profile from different perspectives for example according to particular topics or according to particular roles but I will show you more about this in a quick demo in a couple of seconds. So before that, let's go to make again. There is another code 250001507. So if you go there, I have a second quiz for you, which is about the next thing that I would like to mention. So, let me see. Oh, here you are. The question is, if you have heard of the term narratives of this. So if you know what it is, if it is the first time that you hear about that, because this is something that I'm going to mention. In the next slide. So we already have a lot of responses about 20. It seems like, like most of you have already heard about narratives of this. This is very good, because it seems that the artists can fix some problems that I mentioned. There are some of you that have not heard about them, but also some of you that maybe experts because they have used. So going back to my presentation. So what we are trying to also support in the big scholar profiles is to offer a researcher the chance to put more context on the research work that they are doing so we plan to support narratives. And it is already possible for a researcher that creates a profile to write the narrative describing, for example, a line of work that they have created and connect this narrative to party to a particular set of publications, so that they can later connect with someone if they are interested to know more about this particular line of work. And this is also a work in progress in the context of grasp was of the grasp was project. We are trying to extend this functionalities in a way that we support some well known templates for narratives of this and we can facilitate the creation of narratives based on this template in our platform directive. So, just to summarize, the big scholar profiles is a platform service that is trying to help researchers emphasize what matters in the research work and to put it into context. And in general, what it is provided. It is an orchid based profile for each researcher that is willing to create one. You can connect your own orchid profile, get everything inside the profile, and then you can have an enrichment of this profile with additional information like credit rolls that you provide, like the indicators that I mentioned, like narratives that you can provide. Of course, we focus a lot and we are going to extend the functionalities around narratives, because we believe that it is a nice way for describing lines of work and providing valuable information about this works about the impact that we have about related activities and the skills of the researchers that were needed for that. And finally, an important aspect of these profiles is that you can explore each profile in a more interactive way than the most the most platforms are providing, because you can have, let's say tailored views based on your interests. And just to give you a quick demo, you can see here my own profile. I have included a lot of works in my orchid profile and then this works come here in the big scholar profile. And as you can see, you can find the list of works on the top, you can sort them based on different, for example, publication here, which is the default or based on a particular indicator, and then you get a summary of all the topics that these works have here on the top. You can get also a summary of the different roles that the researcher had in these works, if these roles have been provided by the researcher. So for example, here I have provided that to this paper I have contributed in the conceptualization, methodology, but also in writing. And of course, I cannot any of the terms from the credit terminology that it is well adopted by various publishers and has been ISO standard recently. And here you can see some values for different indicators that we are grouping them together based on their purpose, the aspect that they are trying to capture. You can see these scores based on the whole profile that I have, but if you are interested to see how I am doing in a particular domain, for example, data science, you can use these topics here as facets, and then you get only those publications that are related to the topic. And if you are especially interested to see only those works that in which I have contributed to writing the original draft, you can also do this in a particular domain, and with a particular role how I am doing. So this is something that you cannot find in other platforms, you can see, you can find tailored views of a researcher profiles, or you can see for your own profile for self monitoring purposes, the same thing, you can investigate, which are, for example, the strong skills of yours. So every calculator is calculated on the fly based on the list of works that the filters keep in the list of works. And of course you can then clear the filters and see again the whole picture. And going to the narratives. Here I have created one public narrative that describes a particular line of work, you can see the indicators based on this particular narrative, based on the works that contribute to this particular narrative. And I have created a narrative that describes how all these are related. And of course, I'm trying to also discuss about my motivation, the impact that this works hard, works hard and things like that. And everything that you can see here can be public or private so we don't want to stress etc to publish this type of profiles. Interactive profiles. Currently we provide two options for them to make them public as it is mine, or to make them private and you can use the same functionalities just for self monitoring purposes. And going back, the important next steps, the work that we are doing in the context of Graspers but also other project. We are trying to improve impact indicators for data sets for research data, because currently we only consider the direct citations that the data sets are getting from publications. We also collect, we consider on collecting us at data from us at scouts, for example, a platform that open that is providing to give insights about how many downloads how many views the data sets are collecting. But also, we plan to consider indirect mentions or indirect acknowledgement of the use of the data set for example in many cases a data set is introduced in the publication and in that cases in some domains, they prefer to cite the paper and not the data set so we try to consider this type of acknowledgement. We are going to base soon support the creation of these views that I saw you the CV views, let's say, based on particular assessment protocols that are that has been created in the context of a particular framework, for example, Graspers is building one such framework the also framework. We will try to translate the guidelines of the framework and its contents into ways to represent the data that we collect in a way that is compliant to, to the suggestion that the recommendations of the framework for particular uses because every user is different it is different for you want to quantify the compliance to open science practices or if you want to quantify productivity or something else. According to the use case, each framework is possible to provide different protocols, we plan to find practical ways to translate these protocols into ways to represent these profiles, standard ways to represent these profiles. We also are trying to add functionalities that will facilitate the creation and analysis of narratives of this. As I said, we are going to very soon support widely known templates that have been created for narratives of this. We are going to provide some structure to the narratives, and this will help both people to better write these narratives, they will understand what to provide there, and also for assessors to, for evaluators to understand. The narrative structure and find information that need more easily. And finally, we are going to some support multiple ways to download and share the profile views. And so it will be possible for example to download them locally and maybe share them if you wish. Before closing and going to the question section, I would like to ask you, if you found this interesting to go and create your own profile, you can use the QR code that they have in this slide. And this will redirect you to the URL of the scholar, and you can create, you can register there you can create a user we have more than 500 users already in the system, and then you can synchronize your profile and you can even keep it private. When you create this profile or make it public it does not matter but if you want to play with that you can keep it private and you can give us a useful feedback. So that was from my side. Thank you all for the interest. I would like to hear from you the questions that you may have. And in general, apart from the discussion that we're going to have right now, you can also find me via email, or via social media. We can also discuss their offline. And thank you again for the interest. Thank you. Thank you. Thank you for delivering a truly informative and inside for presentation. Now I'd like to invite our audience to participate in the discussion so if you have any question please feel free to raise your hand or submit it to the chat. Thank you. Okay, so if there are any questions already in the top just let me know because I cannot see the top. So we have a question about the topics links to publication and other results I would like to know if they're manually added or machine generated. Yes, this is a good question. Currently they are automatically generated. And we get them from the relevant week data classes that has been added to the publications from open Alex. So currently we get them from open Alex, but very soon in the next period we also consider to include topics based on different for different taxonomy for fields. So hopefully we know that there is an active work in open air and the opener graph to provide for the majority of the works that they cover the fields of science, and we will try to also support this. But to be honest, because we already get everything from the opener graph, we will seriously consider to only use these because it was then our workflow will be more sustainable and easy to extend and maintain. Thank you. Thank you very much. Any other questions. I will give some more time. Yes. In terms of managing less account, it is going to be possible to sign in with our kitty. Yes, that's a good point. Currently, we haven't done. We haven't worked towards this direction, but I agree that would be helpful. At least we tried to avoid creating overhead for data entry for researchers. We were elected to get everything for for market profiles and just to help people in having only one place to provide this information. Also, while when we started to collecting credit roles. We started getting them locally. So the credit rolls, we keep them inside our own database. But since it is possible to push this role back to orchid. We plan to build this extension. And every time that the researcher is adding some contribution roles in their own publications, it will be possible to push these changes back to the orchid profiles. But if there are already determined orchid roles because sometimes, for example, this can come from publishers. This will be automatically loaded to the profile something that is not happened right now. I mean, right now the credit role should be provided by the researcher inside our platform, but we plan to change that. I agree. Although we tried to minimize the overhead for researchers, this would be an extra step and we will consider to do this in the future as well. Thank you. Thank you. Any other questions that you may have. No, okay, so that's such as we have a little bit. I think there is another question. Yes, yes, yes, yes. To measure the success of the new metric, do you have some test samples, I mean with a genetic age index method. These researchers cannot get the credits. They, I don't see all the questions. Sorry about that. But with a new metric, this is not the case. Do you have this kind of test samples? Let me see if I totally understand the question. Yes. I mean, we are not the first of all age index has been connected to various problems that we all know. And I mentioned earlier, and it was for us, the decision why we have included it, for example, it was because it is widely used. So we would like to keep it so that it is possible for someone to see that whatever we are calculating is very close to what other platforms is. So they can see, for example, the coverage that we have, it is very close to other platforms and the age index that we are calculating because of course we are using the open air graph data which are excellent in coverage are very close to that. So we don't believe that the age index should be used by its own. We are providing some additional indicators that are research level and we think that this bring different aspects that are not covered from the age index. So let me tell you this to them. If we have tested, if they work, if they make sense or something like this, all these indicators are based on some article level versions of these indicators that we have tested very thoroughly in the, in the past. And we have confirmed for example that whatever we are calculating for popularity has good results in capturing the popularity of a particular article we have some ground truths that we have used, which are based on the idea that you can understand the impact of a publication if you wait for a couple of years and then you measure again the citations in case I will not go into very a lot of technical details but the idea is that regarding the article level indicators we have done a lot of tests regarding the research level we take from granted that they capture the respective aspect of impact based on the test that we have done for the article level indicators, and then we summarize them for the article level. And also, despite which is the current set that we support for the research level indicators, we have just selected those that we thought would be useful and we have included them in the profiles, but what we are doing is not a working in a vacuum. We are trying to follow the latest developments in the field of research assessment, and if a very important indicator has been proposed we will try to include it. And also, if a particular indicator is very problematic, is found to be very problematic, we are going to highlight and emphasize in order to interface that fact. And finally, it's not everything as I said around indicators. We don't consider this as the most important contributions that we have. We provide indicators as a way to supplement the processes for research assessment, so someone can use them in a way to help them get the first glimpse and then they should focus on the other more quantitative, qualitative aspects that we provide like which are the contribution roles of the researchers in particular works, which are the narratives try to understand the line of work that is related to particular narratives. And this is the way that we plan to use the platform. And again, everything as I said, we're going to follow the outputs and the last developments of the research assessment community where not the experts in that we are building the services, and we are following their activities in trying to build something as useful as possible. And don't forget that even all of these aspects that we mentioned, when they become established, a lot of problems will be identified, a lot of biases will be introduced, a lot of attacks will happen. For example, even for a narratives, we already know of some problems we know for example that based on the characteristics of a particular researcher, they may exaggerate or not in their narratives. And this will create a different impression based on how much someone is exaggerating on advertising their work. So it's not that all these concepts by their own are bulletproof, nothing is bulletproof. We are going to provide to give more ways for people to have a chance to create profiles that better emphasize hidden contributions that they have, and putting the correct context, and of course this is a continuous work and we will stop, we will not stop and say that, okay, now this is ready. This is something that we should continuously improve. Another question, I have the chance to also answer this. Yes, yes, yes, yes, of course. About Beep, that is connected to Openair. Yes, okay, so keep in mind that this tool, yes, of course, since it is using Openair as its main source, every data inside the Openair graph can be visualized and summarized here, analyzed here in this platform. This is not the only researcher level monitor that we are planning to build in the grasp of project. I know that Openair also plans to have a similar platform, taking some key ideas from Beep scholar, but also from other recommendations that are around. They are trying to create something similar. So in general, either this platform or the Openair one, yes, they can be used from the AC for various purposes. Okay, thank you. You guys, something last, you can just consider the Beep scholar as the most more experimental platform regarding all this. So, and then the Openair services always are mature and production ready. Okay, thank you. Is there any other question? You can raise also your hands. Okay, thank you all for participating in this session and especially for your valuable contribution in this session. Thank you all for participating. Thank you for the invitation and for the interest. Thank you. Bye bye.