 We move on to our next speaker, Basil Mahfouz. And his talk is case study of education policy during COVID-19. Thank you very much. Thank you. So my name is Basil Mahfouz. And I'm a second-year PhD student at University College London. Generally, my research focuses on using quantitative methods to try to assess how research is used in public policy. And today, I'm going to share with you the results of a case study that we did on education policy during COVID-19. So there was 450,000 papers published on COVID-19 during the pandemic. To put that in context, that's almost as much as all the research on climate change, but only in two years. So given that vast amount of literature and scientific research out there, how well did policymakers take advantage of what they had? So what we did is we went to Overton, and we extracted about 2,452 government policies dealing with education during COVID. They represented about 50 countries, although the majority of them were from the US, the UK, and the EU. Collectively, they cited about 24,000 scholarly citations, which about 12,000 were unique papers. So what we did is we then extracted the DOIs of each of these citations and matched them on the database of Elsevier's International Center for the Study of Research, where we got all their metadata and began our analysis. That's what we found. 75% of the citations were to papers published before 2020, which was rather surprising. So we wanted to ask why and how this happened. So our first step was breaking it down per field. And what we found was that you'll see in blue is the total papers cited and then read the fraction of which were published before 2020. So it's a bit of a discrepancy there. The ones that were medical research, you can't really see them very clearly, but their pediatrics, general medicine, or health-related topics tended to have a better ratio. So they tended to cite newer research versus, for example, social science research, for example, in education. So what we wanted to do first was figure out for each paper that was cited but was published before 2020, where there are similar papers that were published during the pandemic, something a bit more relevant to the public policy elements. So we took about the 8,000 papers that we matched in the ICSR lab, and we matched each of these papers with their corresponding topic IDs. So just for context, there's about 85 million papers indexed on Scopus, and they're classified into 97,000 topics. So for each paper that was cited, we identified the topic, and we wanted to see within that topic which papers they were published after 2020 or during the pandemic to make that match. The second step is we took the abstracts and we performed some basic natural language processing on them. We vectorized them using term frequency, inverse document frequency, to try to match each cited abstract with a newer abstract for a paper that wasn't published. It wasn't, sorry, it wasn't cited. And we wanted to see how many we got. So 50%. About 50% of the papers that were cited before 2020 had at least one paper that was published during the pandemic but was not cited. This creates a discrepancy between fields. So we'll see, for example, pediatrics, about 80%, whereas for education, which is the bulk of the research, about 40%. So there's a discrepancy here between how some governments or are able to extract knowledge per field. So I think we can conclude early-ish from here that for some extent, governments were better at using medical research, sorry, we're better at using certain fields than others. It's a bit more, analysis needs to be done here. The second step was we wanted to understand to what extent this was linked to quality. So this is a bit, I understand it's a meta-science conference so I'm gonna get some feedback on this. So we use quality indicators as the kind of conventional metrics for academic quality, right? So it's the field-weighted citation index for the paper. It's the average age index of all the authors who are co-authored on the paper. It's the site score of the journal that was where the study was published. And as a control, we used open access just to see if it was blocked by certain accessibility issues. So when we took the papers that were cited and the papers that were matched but not cited, we ran two separate regressions. One was a linear logistic regression and the other one was an SVR regression to see how these different variables interact with whether or not a paper gets cited in policy. We found early on some differences in quality here. So in orange are the papers that were old but cited and sorry, in orange, yes, the paper that were old but cited and in blue the papers that were newer but not cited. So we find that, on average, they had a higher field-weighted citation index which is not surprising because newer papers have less time to accrue citations. So we expected that. They tended to come with slightly better journal impact factor site score data. They also tended to have better age indexes for authors but this is possibly due to the fact that there were some new researchers who were venturing into new fields that might have influenced that. But interestingly also we found that the papers that were cited predominantly came from non-open access journals and there's multiple reasons for that, mainly because while the newer research was on COVID was mainly all open access and the second reason was the ratio of open access papers 10 years ago was not as prevalent as it is today. So if it's an older paper, odds are it's going to be not open access. Here were the results of our regression. In total, most of these indicators did not have any statistical significance. The only two that did were the age index of the authors and the field-weighted citation index. But again, the field-weighted citation index would make sense to have an inflated coefficient because mainly older papers had more time to get those citations. Now when we trained the machine learning model to try to predict papers whether based on this data, for the most part they failed. So based on this data we had a recall rate of pretty much less than anything significant. So what we can learn here is that despite that there's no at least observable relationship between these quality indicators and the likelihood of being cited in policy there. So key findings, there was a large number of new research that was relevant to policymakers, that was paper newer, but was not cited. There was no observable relationship between the quality of a paper and policy impact and that there were some discrepancies between different fields with what decision makers or what policy makers in education were able to do. So with that, thank you very much for your attention. This is my email if you'd like to get in touch. I welcome your questions. Thank you so much. Thank you so much. I invite questions. We have four and a half minutes. So plenty of time for this session. So if you have any questions, please just get up and go to the microphone. Thank you very much for a very interesting talk. Wolfgang Kirsner of Computational Metascience and Astrophysics, Michigan State University. So I was wondering when you looked at the citations, you only looked at, so we have a post-drop as well on a similar issue, when you looked at the citation you only looked at very recent papers that only have been out for a small amount of time. So the amount of gathering citations is very small versus your control group which is older or do they misunderstand something there? Okay, so when you mean by citation, I'm assuming you're talking about the citations in policy documents. So for those, we didn't exclude any of them. So if you were cited, you entered into the database and then we matched you with newer papers if they existed. And so for those is where we found that about half of them had a newer paper and probably I should add that about 70%, sorry, about the average was about 70 papers, newer papers for each cited paper that was older. I see, thank you very much. Thank you for the question. Hello, my name is Han. Thank you for your great presentation. I do have a question about, you compare the policy documents, you use TFIDF and you compare the old policy and the new policy, could you go back to that slide? I have one question. I don't know if this goes back. Oh no, I think I pressed something wrong. I don't think I can go back on this. Okay, so my question, maybe I misunderstand your answer but I was wondering, you use TFIDF and you decide if the new document is similar to the old document but how do you set the threshold to compare two metrics? So thank you for that question. I don't know if I can go back to my presentation. I had a slide on that but if I can, I have two abstracts, I'll tell you about the technical. So first step there is the first matching was on topic level. So basically you had 97,000 topics so the fact that two papers already are in the same topic means they're already in the same area of research. So then when we use TFIDF to match them we tried multiple different coefficients of cosine similarity. So the first one was we started with about 0.5 so if I had 0.5 cosine similarity then we considered it very accurate. Oh great, so thank you so much for putting those up. So this is how the topics work. So they work on co-citation analysis, right? So if papers cite each other, they're in one topic so that's how Syva's topics work. And here are two abstracts that have a cosine similarity of 0.3. So abstract one, you can see impact of principal turnover, it's about what has Missouri data but I'll give you a second to read through it. And then abstract two which was newer, relatively similar topic but you can see that one is about with Texas so there's a bit of a domain difference. So I'm not necessarily saying these two are alike and one should have cited one over the other but it's just to demonstrate how close these two papers are given the method that we use to try to match them. Okay, thank you. Hi, Rhys Richardson, Northwestern University, great talk. Maybe I missed this but did you look at the frequency of retractions among the cited documents in these policy briefs? So thank you for that. No, I'll take a look. Having said that my first impression would be these are fairly new policy documents and I mean, I'll take a look at the old cited papers to see if any of those are retracted. That'll be a very interesting thing. So thank you for that point. I'll definitely take a look. Hi, congratulations. Did you compare cited and non-cited in terms of risk of bias or at least in a sample more detailed quality evaluation of the research? So thank you. I think the only metrics used were the ones there. I had experimented with other kind of more altmetrics based ones. So how many news mentions, how many social media tweets, et cetera they had. They also didn't really play a big factor and the data became smaller and smaller. So I decided to kind of make the cut-off at these three main indicators. But I definitely agree. I think there's a lot of questions about how we could use, what are better indicators to use. So if any of you are gonna come up and let me know of a good database where I can kind of assign some other value to those papers, I'd very much appreciate it. I think we're out of time. Thank you so much. Thank you very much for listening. Thank you.