 So I'm going to go ahead and introduce our first speaker. One thing I just wanted to remind folks of our speakers specifically is, you know, talk for about 12 to 15 minutes. If you start to get pretty close to that 15 minute mark, you know, two minutes or so, I'll just give a quick verbal. Hey, you know two minutes left so be expecting that. But yeah, so our first speaker is Richard Sever and Richard is the assistant director of Cold Spring Harbor Laboratory Press at Cold Spring Harbor Laboratory in New York and co founder of the pre print servers bio archive and med archive. Richard you can feel free to go ahead and share your slides and get started and I'll be timing it so you don't have to worry about timing yourself if that helps. You can so you can hear me okay. Absolutely. My screen so I think we're good. Yep. So yeah. Thanks very much for the invitation. Excited to be here virtually, if not in real life. So I want to talk a little bit about pre print specifically bio archive met archive today, but I just wanted to start by saying that all the things I'm going to talk about come from Cold Spring Harbor lab. Cold Spring Harbor lab is an academic context. Cold Spring Harbor lab is a mainly molecular biology and genetics lab on the north shore of Long Island. But we also have a very, very long history of scientific communication and the first meetings were held in Cold Spring Harbor more than 100 years ago this is a slide from the first course help that. We spent a lot of time in a number of different initiatives I like to say we have hundreds of scientists, thousands of visitors and millions of readers we have a conference program. We have a graduate school do residential lab lecture courses and we also involved in dissemination of science with producing various books journals and pre print servers most recently. So kind of right to explain the importance of pre print servers I like to show this slide which is a slide I show the students every year when I teach them about how the publishing process works. And the point here is that in traditional publication, every academic who is listening will be familiar with this, you submit to a journal. It's a decent journal most likely outcome is that you're paid immediately rejected you have to start again. You know if you make it past the editor then you have to run the golf corner of referees. You know if the editor still thinks your papers halfway decent the most likely outcome is that you have to revise the paper which can take months, you undergo my multiple cycles in this. Before anybody can actually read the paper, and then when the papers published most of the time it's behind the paywall. So the question is, is in the age of the web whether we should do it this way and I think that would argue the answer is no. And then what we should do is decouple the dissemination of the work from its subsequent certification by a journal by posting a pre print. And so that's what I mean by pre print and unpublished manuscript. Yes, but you have to go through the formal process of being certified through parrot peer review by a journal. And obviously there's a precedent for this. The physicists have been doing this for nearly 30 years archive was founded in 1991 and now has more than a million people in computational science, physics, and math. And it's so inspired by this we launched bio archive in 2013. And it's also an academic nonprofit service, completely free focuses on biological sciences isn't really intended to complement archive is funded by Chanzaq about the initiative and cost and part of that and as I say it's all it's free to read and free to post. And interestingly, since we've also seen 2013. And other academic communities decided that this was a good idea so there's been a real explosion of pre print servers in largely nonprofit discipline specific manner, chemistry chemistry, chemistry, agriculture, agriculture, social science, many of these and we ourselves. Last year launched met archive as a collaboration with Yale and BMJ group. Provide a home for clinical pre prints and the reason for doing make creating separate server was that you know for that there are additional concerns about clinical information so very much complement archive but it has some enhanced screening procedures. There are some additional declarations that authors have to make really tailored to dissemination of this rather more sensitive information. One of the benefits of white white white do this as an author. As I say the main main reason for posting pretty quickly transmit the results. It's also an opportunity to get pre publication feedback from colleagues in the wider scientific community so that you can improve your papers are ultimately better when you send it to journal and and that process is as quicker. It's also a way to increase visibility, especially career sciences. A lot of the time the early career sciences the sort of time course with that fellowship will be at odds with the time course of publication with a pre print you control the timing of dissemination so you can say here's the work that I've just done you can read it right now. And this is useful for grand hiring committee committees who want evidence of recent work in their examples like Nikolai Slavos that I saw in the bottom right here where he got his first faculty position before the work actually appeared in the journal in time on the basis of the pre print and the cell report paper that was his was his key work once he was already in the job. So this slide. If you look at the blue curve just shows what I mean by this delay blue curve is the submission to publication times for, excuse me for journals in public, and you'll see that the median is about six or seven months, but the range goes out. Beyond two years, three years is not unheard of for people to spend three years in pyramid. And of course, if you can disseminate it immediately by a pre print you can see on the far left where bio archives which is the same mention within 48 hours, which is a major time saving. So what this means for an individual is exemplified by this curve by by former post of Coltsman Harbour now as his own lab, and he described some work he did recently where he basically he wrote a paper. He started to on Twitter was on bio archive midway through 2017 started a collaboration with the author of that paper. They started sharing resources doing experiments and by January of 2018, they'd actually done a whole series of subsequent follow up on with this with this great collaboration, and we're actually getting results from that collaboration in January of 2018. Now in January of 2018, the paper that he first read in midway through 2017 as a pre print was finally published in a journal. And in the same week that he finished the follow on experiments. So for him, that was a seven month head start he's, you know, he's a 10 year pack individual he's got five years to make it seven months is a really big head start. And our hope is that you imagine if you aggregate all those seven months together that we really could speed up science. And so Steve Quaker, the biohub in San Francisco has done some back of an envelope calculations that suggests if you make various assumptions for what that delay means in terms of spread of ideas and how many ideas it takes to make a discovery. You can actually get to a point where everybody posted that papers in pre prints in 10 years you could speed up science five by fault which I think everybody agrees would be a good idea. So this is just how bioarchive works when an author submits a paper and grows some administrative checks that check it kind of looks like a paper, it's plagiarism screen, it then goes to group of affiliates, or essentially faculty members who take a quick look and say that looks like it's science to me. The very important stresses is in peer review they're just saying it kind of looks like science, the paper goes live, and then an author can resubmit a new version at any time. It's important to stress why we do have the screening process and don't put up anything. We do have concerns about ethic, we don't want to publish things that are non science garbage to the science or play drives were very concerned in the clinical sphere for things like to ensure the patient's own reveal so you need to have a quick look to check on that. And on the health side that we don't want to post in research that could be dangerous like, you know, disputed vaccine safety disease transmission rule familiar with HIV denialism etc. And toxicity, carcinogenicity, you know, we're not going to post a paper that says cigarettes and cancer so it's important to have that screen. Assuming you make it through the screen which 90% of, more than 90% actually papers do. The paper goes online. It's date stamped on the top right so you get you know you get a priority signal as an author you can define the article type. We define also as new results confirmatory results or contradictory results so that allows people to do reproducibility studies, the manuscript gets a DIY as there's a, there's a, the author can define the subject area and there's a download link to the PDF. So, as you can see here, this is an author's PDF it's not typeset like a journal it's just the PDF that the author sent in. But we do augment this with full text HTML after a couple of days so people can see in line figures. I would say an author can revise a paper anytime that's a big difference compared to the journal and in the history tab, you see a list of all the prior versions of the article so you can go back and look at them as well as your vision summary. And where an author can say that this is what's different out about the new version of my paper and that will inform your choices of reader whether or not you want to read the new version or not. We have metrics so we have article level metrics so you can see how often a paper is downloaded and all metrics for how often it's being thought about other social media appearances news mentions that type of thing which I think causes tend to be interested in. We have, we also have on site comments to increase discussion of work. As I said we have the on site comments but we also aggregate discussions that are occurring elsewhere so we have a scrolling list of tweets about the article. Links to blog posts about the article news mentions and a number of dedicated discussion sites for people to appearing so I'll talk a little bit about those later. So in the feature list is we also provide links to a journal when the article is published so you know after a few months of the article published in a, in the journal then we put a big from the red link which basically this article is now being published in general you can go into the version of record here. So how we're doing. We actually this weekend just passed 100,000 papers. We're getting about three and a half thousand papers per month. So I think that scientific community are really inviting with their feet on bio archive on that archive. It's obviously a much younger server. The interesting thing here is, you look at the first half of this plot so the black line is total papers. The dark bars are submissions and the lighter bars are revised submissions and what you see is we've got sort of slow growth as expected through through last year. And that's like, we had only bio archive seven years ago and then January 19 brings exploded as the COVID-19 pandemic and we start we've got a huge influx of papers. And before long in April, March April we were getting as many getting as many papers on that archive is bio archive about 100 a day and really that it was, it was very timely that this was launched because it was incredibly useful for COVID-19 dissemination. So as I said, 100,000 papers in bio archive now about 10,000 on that archive, a pretty consistent 30% of these papers are revised and sometimes multiple times, but most of the time it's just once. And we're seeing around 30 million views a month across the two servers. And an interesting figure is that more than 70% of these papers end up subsequently being published in journals that that's a different bio archive it's too early to calculate for that archive because we give a time lag about two years before we calculate it because it would be artificially low for them for many papers but still being reviewed. So over this period of the last seven years it's been great to see these behavioral changes, many more biologists posting reading sites and preprints. Now I think it's very rare to find a basic science journal that doesn't allow authors to post a preprint. Funders are encouraging that their grantees to cite preprints and grants. It was a big move was when NIH said you can include preprints in your biosketch. And we're seeing places like Rockefeller University encouraging applicants to include preprints in their applications. And, you know, one of the things that has been nice is the fucking of the recent trends to only talk about old data of meetings preprints allow people to put the stake in the ground so a lot of time people can talk about new research because they're confident that they've kind of put their mark on the ground by posting preprint around the time they give a talk at a meeting. One of the things that we want to do with bio-archive and meta-archive is have these functions as a hub to promote and allow evolution of the communication ecosystem. So, you know, obviously, there's various discovery initiatives, bio-archive and meta-archive indexed by Meta, Google Swift, and Microsoft academics. So they're all these discovery mechanisms. We link out to journals and other means of evaluating content. And as I say, we aggregate third-party discussions across the web and have provided home for reproducibility studies with this confirmatory and contradictory results categories, things that often journals in the past have not been interested in. So as I said, the most obvious point of integration with journals, we have these two technologies, J2B and V2J, that allow authors to wrap to very, very easily transfer papers from bio-archive to a journal or vice versa. So more than 100 journals participate in V2J, and what this means is that an author can submit to bio-archive, when they've completed the submission process, we then say, now would you like to submit to a journal? There's a big list of journals. They just click on one of those journals and we will send all that information to the journal. So that means they don't have to go through the tortures process and re-uploading all the files again when they go to a journal website. We also do the converse, J2B, so a number of journals like Elias, Embo and Development. When you submit to the journal, they will say, you know, it's going to take us a while to evaluate this paper. Do you want to put your paper on bio-archive right away? You click a button, that happens automatically. They don't even have to go to bio-archive, and both of these are very popular amongst authors. Another collaboration we have with journals is this project called TRIP, Transparency Review and Preprim, which allows journals to post peer reviews of papers they're assessing alongside a preprim on bio-archive. And this is really interesting because although the display of the purity of next to the preprim, the decoupling is preserved because that pain that I've circled there is a hypothesis window. It's an annotation engine and that's actually controlled by Elias. It's not anything to do with bio-archive. Bio-archive is just allowing it to be displayed there. So it's an interesting example of how something can be decoupled but very easily discoverable. And I think it's interesting to talk about this decoupling and the possibilities for innovation. And we're already seeing this be promoted by preprim. So here are just a few examples. PreLites is a service from the company of biologists, which is essentially a news and views for preprim. Pre-review is a sort of journal club for early career researchers that allows them to learn how to peer review by looking at preprim, by looking at papers that have yet to undergo. Peer review, so that's kind of interesting. Review commons and PCI are portable peer review initiatives. So you can post your preprim on bio-archive, review commons and PCI, complete the independent services. They will review your paper. Then you can take those reviews to a journal and say, well, what do you think? I've already had this peer review. It's been interesting to see initiative self-organizing among the community amid COVID-19. So Mount Sinai, Johns Hopkins and University of Oxford have all created these networks where they're rapidly reviewing papers on med archive about COVID-19. Because they think it's really important with millions of people looking at these papers that there is some context for readers, particularly those who may not be experts. MIT Press have taken this a step further by launching RRC-19, which is essentially a journal whose entire purpose is to review COVID-19 papers on bio-archive or med archive. So it'll be interesting to see how that experiment works. And I think ultimately in a sort of post-COVID world, it will allow us to really answer the question about what peer review should look like in the 21st century. So, you know, we can ask the question, when should peer review take place? And I would argue that from 99.9% of papers, the peer review should take place after they've been disseminated as a pre-prim. So that people can start reading them. How do you peer review them? If you've already accomplished the dissemination, then maybe the pressure is off and you can think about what's the best way to peer review paper depending on the subject. Maybe if it's a clinical trial or synthetic biology paper or developmental biology paper, maybe you do something different. And because of the decoupling of the dissemination of the work and the peer review, that is enabled. Who should do the peer review? Right now we have a system where everybody talks about review of fatigue and how difficult it is to find peer reviewers, but simultaneously there are complaints that the Global South and Early Career Researchers have hugely underrepresented in the peer review process. Again, does the decoupling, will that allow us to rethink peer review and allow that second problem, all these people who want to peer review papers and aren't, solve the first problem, which is that we're going to have enough peer reviews, we'll see. And then finally, which papers should be peer reviewed? There's a good argument that there are some papers, archival papers in a very, very niche area aimed at a small number of specialists, all of whom are perfectly capable of assessing themselves. So, I mean, one question is, do we need to bother to peer review those papers at all? And maybe the peer review is in the reading in those niche fields. That's a question that we can ask and answer. And I think it is important to think about peer review. It's often mistaken, you felt that somehow posting a peer review means that you're not for peer review or if you're in favour of peer review, that you really don't think much of peer review. I don't think that's the case. The decoupling doesn't mean that one doesn't take place. In this day and age in particular, and that's the kind of, you know, the experience of COVID-19 means that we really do need forms of review. I mean, it's a signal of trustworthiness of science. And actually, maybe we can start thinking about how we can do this better. And so this is an article that Kathleen Hall and Jameson, who wrote Cyber War, wrote with myself, Bairon McKim from PLOS, and Martyn McNutt, who's president of National Academy of Science. And in it, we talk about the different ways in which we can signal trustworthy of science and trustworthiness of science and articles, including things like badges that show ethical adherence to various things, or the checklist that people can go through, greater transparency, forward linking to other discussions and events that take place after the article is available online, ID verification, et cetera, and all of these things that could be enabled by the decoupling. So I think there's a lot of experiments that can be done, you know, in a post-frequent world. And just finally, I'd like to thank everybody who's been involved in the work on archive and that archive. It's quite common to think that, you know, it's just a website and nothing in peer review. So you just sling things up there. But actually, it's taken a huge amount of work. And it's been incredible with the deluge of papers on COVID-19. So there's a lot of people at Cold Spring Harbor to be basically working seven days a week for months. We have great collaborations with the Yale and BMJ team. Fantastic support from China Superbowl Initiative. Technology partners, partnerships with Haiva Press in Silicon Valley, Google and Hypothesis. And again, great collaborations with publishers like Kloss, Embo, Elive, and the organizations like the period of science. So that's all I have to say today. Thank you very much. Thank you so much. That was fantastic, Richard. And we do have one. So we had two questions. One of them was retracted because he answered it in the talk. So the second one here is, do you have a sense of citation patterns for bioarchive preprints versus their traditionally published versions? So, oh, so well, so there's a couple of things there. So people are people are looking at citation numbers. And what you see is that the majority of the majority of citations accrue to the published version. But that will probably vary. I mean, when you look at journal citations, most of the most of the most of the citations happen in the second year after papers published. So if you think the most preprints in about two years again and be a published in the journal paper, when people make a citation choice, they will choose the journal version, which I think is the right thing to do. So I think the numbers are somewhere between one and about 5% of citations to the preprint version. What is interesting is that there's a few signs now that if you compare papers that are posted on bioarchive and you try and find equivalent papers that were not posted as a preprint, then the ones that were posted on bioarchive get more citations in total. They definitely get more readers. And that's interesting because it suggests that maybe two bites of the cherry, the preprint and the published version get you more citations. But there's a lot of confounding variables in that so I'm a bit hesitant to make any strong conclusions. Sounds good. Okay, so we do actually have another question but we do have to move on so I'm going to be sure to flag that question and during the panel I'll bring that back up. Thank you so much Richard this is fantastic. This is excellent excellent. Hi. So now I am going to queue up our second speaker. This is Kathleen Carly. And Dr. Carly is a professor of computer science in the Institute for software research here at CMU and I triple a fellow and director of the Center for computational analysis of social and organizational systems. CASOS is the abbreviation and director of the Center for informed democracy and social cyber security. And that's here at CMU as well. And very excited for this talk. And Dr. Carly again once you have 15 minutes or so once you start to get to about the two minute mark I'll send you a private message in chat just to let you know. I'll send you a private message. Yeah, feel free go ahead. All right. So, hello everyone and thank you so much for having me. So I'm going to be talking about disinformation and social influence, and in particular trying to show you how that is currently being used to drive and anti science bias, and how it's actually being used to drive non-science as though it were science. So, in the course disinformation is not new it's always been around. On the left you'll see a picture of a bass relief of Ramses II, where Ramses is claiming that he won this battle over the Hittite people. In fact he really lost. And what did succeed though was his propaganda. Today disinformation is much more elaborate and cost much less to develop, and probably doesn't last quite as long as that bass relief. But for example, a common one that is very science oriented in during COVID-19 is that Bill Gates actually created the COVID-19 virus, the SARS 19, and that he did that in order to create a new world order, because once the virus is out there and spread worldwide it would cause everybody to go into lockdown. Once they were all in lockdown, they would then go cashless because no one can go to stores and everything has to be done without hands. And at the same time they would install 5G towers all over the world, and those would allow you to control RFID chips. Now that's true. Okay, and the RFID chips though it could be implanted in bodies. And so the vaccine which he also created would be injected as it was injected into you would contain an RFID chip that allowed the governments or the new world order to control you through these 5G towers. And this particular piece of pseudo science and there is even posting it as science is actually all over the internet and is actually part of the campaign to convince people not to get vaccinated if and when the vaccine were to come out for COVID-19. Of course, it's simply not true. So how does disinformation spread and what is it like disinformation actually has many faces, you might think it's just inaccurate facts. In fact, that's not the dominant part of the problem. That's a plate there are there are fact checkers out there and so on and you can look that deep fakes is everyone's talking about today. Again, those are extremely rare. But the real problem is a lot of what's called disinformation is actually spread through the use of illogic through unsupported opinions through misleading anecdotes and through pseudo science. It's often spread by bots and trolls and memes and cyborgs conducting campaigns to both create groups who are receptive to these messages and then just spread those messages to them. The reason it works this way is because social media, much like science is organized into topic groups. That is you have a set of people who are interested in a particular issue, like we're all interested in DNA or we're all interested in network science, or we're all interested in the network. Okay, and those people talk to each other but they also talk about the same things. These groups online can become very pathological, just as academic departments can, because they not only do they talk to each other they only talk to each own and they get into what's called an echo chamber where they're all talk very, very tightly connected and very much talking about the same thing. And once they are in those forms, it is very easy to send excite messages or dismay messages, and actually topple those groups over through the psychological process called a medulla hijack into responding to the world emotionally, rather than rationally. And at that point, if you can do that, you've lost them to scientific and critical thinking. So what is happening is that messages are crafted and people are adjusting things through the digital airways to actually take advantage of the way things like Google search works, or the prioritization rules and Twitter, etc. So that their messages always get to you first. It is much easier to find pseudoscience on Google, for example, then it is to find real science. They also exploit the way your mind works and the kind of confirmation biases you have your escalation commitment to make you think that once you found this piece of information, it is true. And studies now show that people do are not able to generally discern what information is false, particularly if it comes if it appears to be part of their groups norm. Okay, so for example, Democrats can't tell that Democratic falsehoods are false, Republicans cannot tell that Republican falsehoods are false, for example. In addition, all these things are exploiting something called social cognition, which is these heuristics that our brain uses to make sense in terms of the world in terms, then the vast quantities of information in terms of generalizations, and in terms of groups. So when you hear people say well everybody knows so if everybody knows that the moon is made of green cheese, then you'll say oh yeah I guess I did. And social cognition actually leads you through an influence process to start believing and thinking with those that you interact with lead and believe. So on social media we just get you to interact with the right people or non people, and then you will start believing and thinking about those kinds of things. Similar kind of processes underlie why there are camps and science that fight about the different approaches to solving a particular problem. So one of the ideas of course is that there are these now with social media with online archives. There are these super spreaders who have extreme capability of getting their message out there. That doesn't mean anybody believes that it just means it gets out to a lot of people. So one of the things that you see happening is you see super spreaders out there spouting pseudo science and it gets the information gets out to tons of people. They're super friends who are involved in two way dialogues, the talk you back and forth they, well you know, this is kind of true. Yeah, I really think it's kind of well my cousin said, and they get involved in dialogues with you, and can give me two things that aren't true. And finally there's the echo chambers, which on the surface look like a lot of these super friends. What is being manipulated is actually both the content the stories that are being told the narrative, the words that are being used and the way things are described, as well as the groups groups in in the online forms and it doesn't matter if you're talking about Reddit or, you know, Google groups or anything like that are being manipulated so that they are constructing who is actually talking to whom and who the opinion leaders are. And this can all be done of course by just simple messaging. The kinds of tools that are used are bots which are fully automated accounts cyborgs that are part human part bots trolls that are humans hiding under pseudonyms, and so on, as well as utilizing memes utilizing the subconscious cues and messages, such as like colors and happy words to make you think about things a certain way and of course the likes and videos. All of these things are out there. Things are used together to create what are called fake persona fake persona are things that you that you might think is a real person like a real scientist, but they don't really exist. And these fake persona are actually have underlying them teams of people who write fake science papers, some of which actually have gotten into archiving. I don't know about the medical one. They've gotten into archiving the computer science one. In terms of understanding the influence campaigns we often look at who is communicating what to whom with what impact and a lot what we find is that in fact, the way you communicate is by getting people to engage in particular discussions, creating excitement, enhancing messages dismissing certain messages, creating distractions there's a set of tactics that are actually used to actually spread disinformation or even regular information. All of these tactics involve shaping the content but other tactics, such as building a group or bridging between two groups, or nuking an opinion leader, are actually shaped or actually designed to actually affect who is talking to whom, and to actually create, like I said, local opinion leaders for particular things. You can actually develop measures for all of these things and can actually measure them on posts and those posts could be social media posts. They could also be abstracts and authorship lists from various articles online or in other archives. These techniques are actually used to conduct influence operations to actually change how you think about things. For example, during trying juncture, which is a NATO exercise, they have this thing called the Viking warrior campaign to try to convince people to do that NATO was good and to join NATO because it was filled with, you know, lots of good looking guys who are out there doing their bit for humanity. This is a very, very expensive campaign. Disinformation can be very cheap. For example, Russia put out this dismay campaign, where they showed simply a meme of the Defense Minister of Russia, coupled with a meme of the Defense Ministers of Sweden, Norway and the Netherlands. The implication is Russia is strong. NATO is weak because they have all these nice women who smile running the military, and they don't really understand the military. It had much higher impact. Disinformation can be done very, very, very cheaply, and therefore it can be spread very quickly. These are two more disinformation campaigns one used to build a group one used to nuke a group. On the left, what you're seeing is a campaign that was done with bots where you had a set of young men who did not know each other, but they have one thing in common. They all like to share light porn images of women. They share each other and so on. Bots came in and started sending out messages that actually told each of the guys about each other, which formed a group of these young guys. And then once they did that, they then told them where to go to get rifles and where to go start fighting in the Euro Maidan Revolution. That same tactic we're seeing today being used in the anti-vax campaigns to actually build groups that are anti-science who hadn't even thought about science prior to that. On the right, you're seeing what's called a typical geofence or a denial of service attack in social media, where they're just blanketing the airwaves and therefore covering all of Finland with just simply counting from zero to infinity in Finnish numbers. Okay, and therefore blanking out all the other information so you can't even go in and see what they're talking about. These kind of tactics have also been used during COVID-19 to blank out certain areas and to actually attack certain underrepresented groups. In the polarization campaigns, what you see is you have bots, trolls, cyborgs join groups like they've joined the pro-vax groups, they've joined the anti-vax groups. Then they boost the numbers of those, then they send out things that build more connections among the members, which is basically converting them to be more and more like an echo chamber. Then they wait for some message amount that is pro or that is either happy or angry on one side or the other, and then they counter it with messages on the other side, which are then amplified by bots. So for example, the story comes out that says, hey, there's a new vaccine for COVID, the anti-vax side will say, yeah, but it actually has the RFID chip in it so they're going to control you or it's going to sterilize you. So don't take it. So and which makes the groups angrier, then they engage with more backing and engaging community by building up the opinion leaders, getting these two groups to not talk to each other and getting them to be more and more upset, which actually makes it hard and will make it hard to actually deliver a vaccine once it actually comes out. So disinformation campaigns, you basically are done like this you find the controversial issues you embed bots and trolls on each side, you build the ties you foster fear on one side you call for protests, you spread disinformation so on. This tactic was actually used on the reopen America campaigns. So we saw new accounts created, you saw spike in those those new accounts came in the ones that were pro reopen were all coordinated and organized and started spreading, not just pro reopen messages, but at the same time, anti science messages at the same time anti backs messages, do anti mask messages, and they cited papers that had appeared in pre print series and linked to series that said that you don't really need masks, they won't won't necessarily save you from COVID-19. They also had stories about fake pandemics, and then that state job losses and people and hospitals being empty. In our own state in Pennsylvania, we saw as they formed an echo chamber that was focused around the pro, you know reopen side that was filled with a large number of bots, these bots not only pushed the anti science, the pro reopen side, but they also began some of them began to back Governor will one of the bots in particular that girl Sandra five was was actually backing up alternative opinion leaders and building up and was actually building up the group and was bringing in more excitement to them about where and when the various communities would be as part of this campaign. She is also one of the ones who is spreading an anti science message. So disinformation is all that is being spread is also spreading intolerance. In the Philippines and in many other countries the increase in disinformation during COVID-19 has led to hate communities forming these often rapidly go away, but in the US they have managed to stay. They have now become very stable, and although many of them started off as being, you know, negative toward the people on the ships and negative, you know, toward China, they have now become very stable and they are being redirected in their hate toward negativity toward science and negativity toward Democrats and negativity toward, you know, the African American community and negativity toward the LGBTQ community and so on, and they are being directed at a variety of things. At the same time, bots are basically being used to link more and more these hate groups together and make them larger and larger and larger. So that has been a growing anti science trend in the social media, and in particular we saw a surge in April with a surge of new attacks against expert advisors, and of those spreading that information on Twitter 69% had bought like behavior. So we saw this increase in attacks, and an increase in memes that were directed at trying to reduce the scientific influence. So some of the things I in my group study we can find out more in our website and so I'm here, but with that I'll stop. Wonderful, you did perfect on time. Absolutely perfect. Let's see. Yeah, Dr. Carly I didn't see any questions come in at least at this stage I'm sure at the panel we will. In the next 30 seconds has a question, feel free to. There's one. Sorry, I'm just. Okay. Yeah, maybe Alex, would you like to unmute yourself. Sure. My name's Lex Kravitz I was just wondering if there's effective ways to reduce the influence of these bot networks and whether it requires the social media companies themselves or what your thoughts are on on solutions. Yeah, so let me begin by saying that there are a lot of bot networks that are actually doing good things. For example bot networks are also used to warn people that tsunamis are coming and so on. And some of the bot networks just simply rebroadcast news. So they're not all created equal. So there are things you can do to stop them. One is, is you yourself to quit. You know, don't just automatically follow anyone who follows you because you're actually giving them that a lot of those will be bots and you're giving them more power. Make sure you actually physically know people. The other the other thing is just because something's trending in your inbox don't go with what's at the top, actually scroll down, because the bots are getting it at the top and if you go down you'll get more information often. So those are a couple of things you can do you can also report bots, although Twitter and other places are very unlikely. They don't really like taking them down because it's against our business model. And it takes a lot to actually prove that some things a lot. So one of the best things you can do in that sense is just ignore them. Thank you. Thank you so much for the opportunity and we had we have we have time so we had one more that came into it is. Yeah, so the question is, is it good to differentiate between trolls and bots, or can you consider them as potential hybrid, hybrid troll bots. Is that a conversation that ever comes up. The way we use the term troll. It has to be a human being. It's a human being who's hiding under a fake persona, who is actually engaged in the hate speech and in what's called identity bashing. That means they pick like a subgroup like it may be police, or it may be, you know, an ethnic minority like Muslim or it may be a, you know, some other kind of group, and they will start using hate speech against those kinds of groups so that's the way we use trolls we recognize that in the media sometimes they'll talk about troll farms. Often they're talking about individuals running a set of bots and those are what we call side works. And one more very quick one again we're still doing great on time. This one you, I think you had started to address quite a bit. The question is how do we. Yeah, so the person just said you pretty much address this it was the idea of you know just becoming self aware and kind of ourselves becoming vaccinated against this, you know, yeah, against sort of, yeah, and yeah, you did a great job in addressing that. So, okay, great well if any others come in I'll will look at those in the panel. So, excellent thank you Dr. Carly. Okay, so I am going to introduce our third and final speaker of the session. I'm very happy to introduce Sarah kitten, and let's see I'm going to go ahead and get bio up. So Sarah kitten is a technologist and researcher working at the intersection of technology and society. She's a Marie Curie research fellow on open design of trusted things which is a joint program between Northumbria University and Mozilla. The research focuses on the impact of technology and the internet for grassroots communities and neighborhoods. So, yeah Sarah you can feel free to go ahead share screen and and again once you get to about the two minute mark I'll just send you a private message and chat. Great. Thank you I hope everyone can hear me. I should begin by apologizing that I do not have slides like the other panelists that I'll try my very best to explain my points. Thank you to the previous panelists for the excellent presentations. My name is Sarah kidden and I am a Marie Curie research fellow on open dot as I have already been introduced. Our project aims to explore how to build a more open secure and trust for the internet of things. So the project started at the University of Dandy in Scotland where I'm currently best and it has now moved to Northumbria University Newcastle. And my research is basically trying to explore possibilities for smaller scale local internet of things technology and how communities and neighborhoods can be supported in making the best use of them. And to the focus of my talk today. Also talking about other spaces that I have been involved in. I'll be talking about openness for internet policy and building equity. So I just wanted to start by saying that as human beings we are curious naturally if you see if you hide something or if you see words like confidential top secret. Suddenly you just want to know what's behind those words and it's just by nature that we want to be like this. And in fact a lot of misunderstandings happen because we are not being open even if sometimes there's nothing to hide. And I want to use the example of openness in internet policy and why I believe that such openness should be encouraged in other spaces. Let me take you back to 1969 to the advanced research projects agency network, commonly known as a planet which has now grown to be the internet as you know it. I think this information is known by many, but for the benefit of those who do not know a planet was run by the United States Department of Defense and it was the first network or internet related project geared towards research. So in September of 1969 the UCLA was the very first location to host the first node on the internet, and the first message or email if you will was sent to Stanford Research Institute which was not to in October of the same year. So by the end of 1969 upon it was able to connect for locations you had UCLA University of California and Santa Barbara Stanford Research Institute and Utah. Ten years later of course it was no longer just the four nodes and it was very hard to draw the geographic map of the network or the internet. Because of the complex connections and now it's even more complex if you add the layers of products and services that run on the internet from multiple locations. So it's interesting for me to note that something that started as two nodes or four nodes connecting with each other has become very powerful, very big, and in my opinion very hard to predict. And the fact that we are able to actually present today via zoom and other conferencing software is just a testament to this enormous growth. So, but of course this was something that happened just like that, there were many things that happened or that helped to make it happen. And one of the things is openness. So the open nature of the networks and architecture allowed people to contribute to add new protocols to add new services, and the internet today still allows for many platforms and services to be added because of the foundation that was set. So we attended sessions yesterday and today and open access is something that has come up frequently I think in almost all the sessions I've attended, and related to the growth of the internet is open access for request for comments, or RFCs if you've heard about them which are formal documents drafted by the internet engineering taskforce that describe the specifications for particular technology. So these RFCs are actually public documents, they are posted publicly, they are accessed by anyone, and anyone in the community is actually free to, to propose something to write a policy proposal and submit it to the community and I think this is actually a, it's a strength of the internet and the growth of the internet especially in technical standards making bodies. And so if you go to spaces like the internet cooperation for sign names and numbers where I participate in the at large advisory committee, where we represent interests of internet end users to icon and to other spaces. I find that participation is open and all policies are developed in what you say is a multi stakeholder approach. So basically it's not just the technical community but there are different groups you have internet service providers, governments, civil society, security people and different people just coming together to, you know, to project if you will the internet. So the internet has been defined as a network of networks, many small, medium and large networks connected to make the one big internet. And I like to use the analogy of the internet being some sort of incomplete puzzle that requires different stakeholders to complete. So it belongs to no one and yet it also belongs to, to everyone so everyone basically plays their part and you complete your piece of the puzzle. And I would like to recommend a book, Roads and Bridges, Unseen Labor Behind Our Digital Infrastructure by Nadia Eggball, I'll post a link on the chat if you've not read it I think it's a very good book that talks about interesting things basically like how you, you take care of your physical infrastructure like roads and bridges you should do the same with digital infrastructure as well and how, you know, medical records, banks and things like that are running on free and public code. It's a really good book that I like, I liked to read and so in my current research right now in Internet of Things of course netizens are growing in number. And I should say actually exploding because we are not just connecting people we are connecting things, all sorts of things, billions of things, and we need to understand and learn from the current standards and what has worked in order to build a future internet. And don't get me wrong. I'm not saying that the internet right now is perfect and great because of openness and the multi stakeholder approach as you've heard from the previous speaker there are so many issues surrounding the internet and technologies in general. There's a lot of bias in artificial intelligence systems, surveillance, misinformation, the spread of fake news, just so many issues generally. But I think what we need to learn from this is what has helped the internet to grow from just four nodes to billions of nodes right now. Moving on to building equity. I just want to say that adding the word open to anything does not make it open in real sense. I've seen like many people will add open to to make it feel like their project is open but there are so many issues to consider. Sometimes the space might be open so it's open software or it's open, whatever you call it, and even some sits on the table but that does not mean that everyone is welcome. Sometimes we have to extend an invitation to underrepresented voices to have them join us on the table. And once people are on the table, it also does not mean that they have a voice. There are so many power structures, even in open spaces and those with more power contribute more than those with very little or no power. And sometimes also you have issues like resources. So you have a seat on the table you have some voice but you don't you can't even afford the resources because to participate in these open movement things from coding to software to policy. Usually on a volunteer basis and many times you're either self funded or you have external funding to allow you participate in open spaces. And then I just want to highlight something that has always been with us but somehow either we were not aware or we chose to ignore or it has been made visible and in our faces because of the COVID-19 pandemic and that is the digital divide. I'd just like to remind you that the ITU, the International Telecommunications Union is still reporting that 53% of the world's population actually has access to the internet. That's slightly half, slightly more than half of the world's population, depending on who is counting. So three months ago, I mean three months into the lockdown, I wrote a blog for a project website and I was basically reflecting on things I had seen and observed. I grew up in East Africa, partly in Kenya and mostly in Uganda and I'm currently living in the UK so I was like while keeping in touch with my family and friends and living here physically I was just trying to compare things on both sides of the world. And so on the one hand, you had one part of the population that continued business as usual, they were working from home, attending classes online, you know, shopping online and things like that. But there was another part of the population that's completely cut off from information from work, from class content because they don't have access to technology and to the internet. And it's particularly sad for me because at the end of the day, both categories of people are required to perform and compete in the exact same way. So we need to think about these issues and we need to just constantly remember that as we fight for these things that as we build the tools that is another half of the world's population that actually has no access and yet they are affected by all the decisions that we make. So I watched as people posted pictures of virtual graduation ceremonies they were defending their PhD thesis and you know, I was just imagining that there's another part of the world where, you know, people had been cut off from everything. And this experience is not unique to developing countries alone in the UK, for example, I know the government putting something like one billion pounds to tackle the impact of lost teaching time because of the pandemic. And I think out of that about 100 million pounds was invested in remote education and delivering laptops and internet to people who need it the most. And what does this tell us it basically tells us that the way we've been doing things is not working well, and we have to find other ways basically of doing this, especially in a world that's becoming more technology driven. And I think I'm going to stop here. I'll be happy to take any questions. Thank you everyone. Thank you so much, Sarah. That was fantastic. And I am just, again, you can feel free to send me questions in the chat or can feel free to raise your hand and unmute yourself as well. If you have a question. Let's see. I had a question for you just to sort of, you know, started off here. I love the point that you made that, you know, just by having the word open, it doesn't automatically mean that it's, it's going to be this welcoming accepting environment there's a lot of things going on that can impact that and I think a lot of us on this call are working in these open science initiatives. We're all librarians, some of us is in industry so on and so forth. And I guess what would be, you know, some of your words of wisdom for us just like on a daily basis things that we can remember as we're having these conversations on open science and open technology and things like that. I would just love to hear your perspective on that. Yeah, so just to re echo what the previous panelists was saying is you can't do everything basically like you just have your parts to play and this is why I was talking about it as a puzzle basically so you get your piece and you contribute to it and if you're looking at it from a multi stakeholder point of view, everyone has a different role to play so if we start from government, their role will be in terms of policy and ensuring that the policy is actually effective and it's doing the right things. But if you look at developers, how responsible is the technology that you're building? You know, is it actually building equity? Is it ensuring, is it giving us social justice and things like that? If you look at us activists, like we just have to continue pushing, like we'll push the government, we'll push the technologies, we'll push everyone to do their part. So it's not like there's one solution to everything. I think everyone just puts their parts of the puzzle and then we'll all complete this puzzle that I was talking about. Excellent. Yeah, and so we have, we're doing pretty good on time so we have some questions coming in too. I'm going to go ahead and pose one of them to you and the rest hopefully we'll get to in the panel. The first is what are your thoughts on what we can do to try and equalize access to research data? Speaking to myself. So there are so many things. I want to start with the point I was talking about access to the internet so like the basics, because before we even talk about access to research data, if we're talking about online data, like right now with the COVID-19 pandemic, we need to give people access to the very basic things that's access to the internet. So there are many solutions. You can find low-cost solutions to giving people access to the internet. I don't know if you've heard about community networks where communities are actually building and maintaining their own networks using very low-cost solutions and things like that. So encourage things like that, encourage communities to be able to develop solutions that work for them and then after that we start talking about, you know, giving people access to resources. Though I think that has improved if you look maybe 10 years ago, the access to data that we had 10 years ago is not the same that we have right now. So for me, I would personally say that we start with the basics and then build on to that. Wonderful. Wonderful. Well, at this point, I want to invite our three speakers back. I guess that if we were in person, we'd have you all come to the front. But we've got so many great questions coming in and I'm trying to do what I can to sort of meld them. And so it's something that everybody could kind of speak towards. But apologies if some of these seem like they are again quite directed. So I'm going to go to my list. It's a very long list, which is great. So this is a question for Richard, but I really think that anybody could probably speak towards this because it's that theme of participation that keeps coming up, you know, just these open networks and participation. And the question is at bio archive and just in general, is there any contemplation like contemplation for a mechanism for flagging preprints that are never published. So they're put out there but then never published like is there a mechanism to track those. Well, so it's difficult. It's difficult. I guess the question is, is what does that mean. Right. So I mean the simple answer is that we, we flag preprints that are published. So if the preprint is not published. Then, you know, there's an absence of a signal, right. So in some respects, there is the mechanism is just not having the thing that says that it's been published. The question is, is what that means. And that's where it becomes tricky. So, you know, I mean, on a bigger longer talk, I give examples of some papers that are five years in here of you. So what does it mean what does it mean you wait five years and then say that I mean there are examples you saw that curve I showed there's lots of papers that are two. I mean I was talking to a friend of mine the other day was like, you know, I'm trying to get this paper into nature. It's been two years. So when, when would you do that. The other thing is another thing I didn't mention is that there's clearly a subgroup a small subgroup of papers within quite a few of these with COVID-19 where there's never any attempts to publish the work in the first place. So that's kind of so the reproducibility studies some of them are like that. But, but, you know, with COVID-19, you see an epidemiological study that's coming out of China in February. You know, in the normal course of events, you know, you would wait X months and then that would appear in the journal. But, but you know, X months and the state of play has changed a lot. So that's kind of like a progress report. So I think that's, I think the notion of flagging something as not published doesn't really make sense. But I do think this is where my last slide I talked about how a bullish and publishing could take place with badges and the kind of appending of symbol signals, different signals of trust to manuscripts and so you know the fact that it's not met archive itself means that it's gone through some sort of screening screening process. So people don't think it's non science or pseudoscience, but it might still be totally wrong. Then you can imagine these papers accumulate a variety of other things. And so I think that's that I suspect that's the way we'll go in the center for open science of doing some interesting things with badges and I think things like some ethics test test will will kind of sit in that in that ecosystem if that makes sense. It does. Yeah. So the next question is really for all three of you. Do the panelists see any conflict between solutions intended to have immediate impact on current problems and solutions intended to have a more long term impact on those foundational issues where those benefits might not be seen for some time. Do we have any thoughts on that. And the person who asked that question you can feel free to unmute yourself if you'd like if you'd like to add anything to that. Yeah, that was me but I think you asked it pretty faithfully. So, I'll start us off. We actually see that solutions that are designed to only have an immediate impact. When they are taken up by governments and by policymakers often have a long term impact and often outlive their usefulness. So if they rarely die an early death. Okay, we also see that things that would have a long term impact are less likely to get funded or less. You know it's not the glamour as glamorous of research, and we often don't see as many people who are. We often see more people who have a humanitarian interest and less technology interest involved in working on those. So we often. So those are some of the tensions that we often see in terms of the let me reverse it and turn it around and say one of the reasons why disinformation is weird and is that it has both a short and a long term impact, and people often only go after the short term impact. You know it's like, Okay, can we remove, you know, all the stories about, you know, drink, you know, drink, drink bleach and you'll get rid of COVID. Okay, and things like that they're disinformation. Without thinking about what that is doing to the long term you know it has impacts for who is actually communicating with you, they're not it doesn't it, and by. And it also doesn't get at the real intent of some of those messages, which was not necessary, which was, yes, people drinking bleach it's great if it's, you know if it's all the people we don't like to drink it but the real intent of those messages was not that the real impact was to signal that there was this white supremacy cult that was becoming a religion and to get people to join it, and they succeeded, and that is the long term impact. So often we don't even study longer impacts. So yeah I mean actually something that you just said, and bring true for me this that this, like there is a danger occasionally with some sort of short term as an approach, we have a similar phenomenon where, you know every now and then. You know an author decides that a paper is wrong or there's some ethical issue with the paper on bio archive. And so what we do is we an archive do the same thing is we execute what's called a withdrawal. But just like a journal retraction we don't actually remove the paper. What happens is that it is labeled is withdrawn, do I defaults to a withdrawal notice but you can still don't see the thing that was withdrawn. And sometimes we actually have disputes with authors about this and they say oh it was wrong there was a screw up my student wasn't supposed to do this. I just want you to take it down. We, what the response to that is, you know that would be the wrong thing to do because it might feel good in the short term, but the reality is that that you know that there will be digital footprints across the web. People would have index that you know we can't go and tell Google to unindex it somebody will download the PDF. People will have taken the DIY and if they go back and they go to a 404, they will say, Oh, there's something fishy, what the hell happened here, and they might actually think worse of you as an author. Maybe if we're transparent and we say there was a paper here, it has been withdrawn, you can go and see what was withdrawn, but the important thing is that the office say they don't want to be considered as part of citation record for these reasons, and everybody's transparent about it. Then actually in the long term, it will be better for you as an author, particularly for an innocent bystander. So you know so that that's true and I think there are there are a few that examples like that. So I would just like to add that of course technology advances very fast that sometimes by the time you have the solution, whatever you're developing the solution for is actually obsolete. Though the ultimate I think I should say that solutions should solve a particular need so you're developing solutions to solve that need that you have and not just great great. Okay, so I'm, I've got another question here and again, I would say this is probably focused more towards Richard and Sarah but again anybody can feel free to comment on this but it's the idea of, you know, so we have the internet. As Sarah mentioned, nobody owns it but everybody owns it it's sort of at that intersection and in terms of encouraging people to you know stakeholders to play that substantive role in providing comments and adding discourse to these kind of things. And the specific question that really came from this is, when we think about bio archive and how a lot of articles are, you know there's things on Twitter there's conversations happening. What the question or said is the discussion of a lot of articles seems to just be limited to retweets, where people are announcing here's this preprint. And does bio archive plan to increase or incentivize sort of a more substantive academic discussion of those posted. Again that's very directed towards Richard but I think in general for other two speakers it's sort of that idea of how do we encourage, you know, healthy discourse around any of these topics in open science open technology disinformation so on and so forth. So we, I mean we obviously want to encourage that we do have, you know, we get on site comments about 8%, which I will say seems pretty low until you compare it to journals which is much much lower and, and, and there's a lot of discussion on Twitter and third party venues I think that the challenges is that we can do this but it's not clear to me how much bio archive or met archive themselves and do I mean one of the one of the challenges we find for any kind of participation like this is academic is academic incentives, academic and I always say the most precious commodity to any academic is their time. And everybody has no time to do anything. And so when you're when academics think about how they apportion time they think about you know what benefit that they get from it. There may be situations in which there's a direct benefit for them to engage in that kind of discussion of public and open but actually right now there's not much of a real incentive to do so. I mean one of the things that I think would help is particularly for early career researchers. Right now the currency of career progression is solely the papers that you produce and early career researchers at that point in their careers have very few papers. But you know if we can get to a point where some of the contributions into discussions become more formalized and in some way recognizes a contribution. So I mean I always I always say that if I'm trying to hire somebody in scientific communication editing or something like that, they will typically see that CV which is like these are the two papers that I did in genetics that took me the last four years, but actually be really valuable to me for the sorts of things that I want to employ them to do if as well as those two things, they have like 15 other contributions where they examine somebody else's work. So we can start if but I think that will take institutions to start saying we recognize these things as valid academic contributions and so you know I'm all for it like I don't think there's that much beyond providing a venue. I don't think there's that much we can do. I mean it's significant to me that we had this, you know, explosive growth of meta-archives in COVID-19, the exponential growth of bio-archives from 2013 to 2020. It's indicated it was satisfying a need. We haven't seen the same kind of growth in the dialogue, but it's difficult to know what one should compare it against. I mean there is a hell of a lot of discussion going on on Twitter, which you know didn't exist free Twitter didn't exist free bio-archives. So, you know, hopefully it will come. So I wanted to follow up on that a little bit. In social media a lot of the discussion is driven by what's printed in blogs and by images and videos that you see in YouTube. You know, if there was if people in science got credit for writing blogs and blog posts and got credit for doing YouTube video, you'd see a higher discussion level about real science, because right now there are bloggers out there blogging about anti-non-scientific flags and writing videos about them, and a lot of the discussion is centering on that. But if we could somehow prioritize those as recognizable, acceptable things that people could get credit for, you would drive more discussion. I was just going to do one thing just to add though. One piece of data that may be interesting here for you is that when we surveyed several thousand bio-archive authors a year or so ago, and actually, you know, what we found was a substantial number of them were engaging in conversations based on their papers by email. So I think that the other thing you need to consider, we all want to have open discussion, but there are some kind of discussions that people either by the nature of what they're talking about, or by character want to have privately. And you know, that's a bit of a shift to happen, say take a conversation that you're having personally with an author about their results and saying, OK, now have that publicly, so six billion people can listen to it. And different people make different choices. And I think we're moving in that direction. Yeah, and yeah, Sarah, I was interested in hearing your perspective, especially with, you know, the open technology that you're talking about. And again, I keep going back to this point because it was so prolific, the idea that the internet's there, nobody owns it and everybody owns it. And I'm interested to hear, you know, from your perspective, the idea of encouraging, you know, healthy dialogue and encouraging stakeholders to really, I guess, play a part in this. What has your experience been like in that? Yeah, so if I think, generally, my experience has been very good. Maybe I've just been lucky that I've met people who are welcoming and if you go to icon internet cooperation for sign names and numbers, for example, they'll have a fellowship program to introduce you to internet policy development and they'll tell you this is how you get involved in internet policy. And generally, I think right now there are programs that help you to get involved in those spaces. The only thing is you need to know about them for you to be able to participate, which information I don't think is so much out there. But I think generally it's open, it's welcoming if you have the information and yeah, perhaps we need to do more outreach and creating awareness about people and telling them why they need to contribute to internet policy because I know when I started about four years ago, I didn't even think that I can contribute to internet policy. I was just like, oh, that's that thing there, it's for those guys over there. But if we know that the internet affects all of us, then we are more willing to actually participate. Yeah. That is an incredibly prolific way to end this session and I know I have at least a dozen more questions and I apologize to anybody that we didn't that we didn't get to during this. I'm going to throw these questions into Slack in our OSS channel. And, you know, we can feel free to have that conversation over there as well but I just wanted to thank the three of you so much this was such a fantastic, fantastic panel fantastic session all the talks seem to very you know compliment each other and, and all three of you are very very. Yeah, I'm humbled to listen to your talks. Okay, well, so this is some business more for the greater the broader OSS meeting today. We're going to go to our lunch break now and we're going to return I want it's 2pm I believe yes 2pm. In time, obviously hydrate be sure to have lunch stretch if possible. We also have gather town open available to you and I believe there's a gather town link earlier in the chat here. So we're going to go ahead and just repost this so you have it and gather town is a way that you can continue the conversations if you'd like that we had earlier today it's a it's a fun little fun little platform to have conversations and and meet different people. So it, yeah at that thank you so much everybody and we'll see you back here at 2 o'clock. Thank you. Thank you.