 Let's get started. So I want to think about science as the ultimate public good. And really the sort of mechanism designed for adjudicating science is one that is peer review, where we kind of use that as a primary mechanism today for both doing this sort of ex-anti-review of determining which proposals are likely to be impactful, as well as this sort of ex-post-review on manuscripts to determine which manuscripts are actually impactful. And so I think there is a lot of questions around whether this is the single most effective mechanism for us to allocate both capital and attention and research today. And so I kind of want to go through both a little bit of the history of peer review. I think Evan at Protocol Labs often has this framework of Chesterton's fence of trying to understand the sort of reason behind a fence before we take it down. I know there are a lot who want to reform peer review and then kind of go into the history first of peer review before thinking about some of the challenges and experiments around reforming it and perhaps some ideas for how maybe impact certificates might be an interesting primitive for tracking reviews and reforming review. And so this slide is actually from a recent talk by Karl Bergstrom, who's a really great meta scientist. I actually encourage you guys to check out his talk from last month. And I've included a little link there below. But kind of going into the origins of peer review, there's sort of this canonical story about the first journal, The Royal Society of Philosophical Transactions, from 1665 with this editor, Henry Oldenburg. And people often point to that being kind of the origin story of peer review, where he was sending out reports to his sort of scientific fellows to get opinions on some of the submissions they were getting and this being kind of the origins of peer review. And actually, the historian that Eugene had mentioned that I sort of wanted to attribute sort of impact certificates to has done a good job questioning whether this is actually a true origin story of peer review, given the sort of very ad hoc nature of how they were doing it. And in fact, the term peer review is actually not one that didn't really emerge in the lexicon until the 1970s. It was a very niche thing. And so I kind of want to push back on the notion that science has relied on peer review as this mechanism for adjudicating how projects work for a long time. In fact, very famously, Einstein, when he was submitting to papers, generally did not go through peer review. He himself actually had a very upset letter when he was reviewed in the physical review on this paper on Gravitational Waves in 1936, where the reviewer had sent several comments back and he had sent this letter back to the editor saying, I see no reason to address this, in any case, erroneous comments of your anonymous extrovert on the basis of this incident. I prefer to publish the paper elsewhere and he would never publish in physical review again. So in the 1930s, definitely peer review is not the sort of standardized mechanism that we see today. Another sort of case study that's kind of funny is the Watson-Crick nature paper, which also was accepted without any sort of peer review. The editor, John Maddox, at Nature of the Times, said it was just self-evident that was correct. And if they had sent it out to any referees, they would have blabbed and that would have sort of compromised the sort of sanctity and secrecy of the paper being submitted. And instead, he'd never even used real peer review at the time. He preferred to carry a bundle of manuscripts with him in the pocket of his great coat and pass them around among his chums taking coffee. So really this sort of formalized structured system of peer review is very much a modern one. And it begins to change really around the 1970s where the government begins demanding more accountability both on the grant review side, where the NSF had been funding very social science projects that various politicians had qualms about and so there was a question around who was actually rubber stamping these projects? Were they politically motivated or do we have unbiased expert reviewers providing a layer of sort of accountability? And then also trying to understand what were the outputs that were coming out of this funding? Were those outputs actually valid? And so this is sort of the inflection point where peer review starts becoming sort of established as the dominant mechanism for how we adjudicate science. And if you wanna learn more, as I mentioned Linda Baldwin has written about the fantastic book on sort of history of nature as well as the sort of rise of peer review kind of in Cold War America. So I encourage you guys to take a closer look at that. Let's start moving forward. I think the question is sort of what are the challenges with peer review today? And in many ways it's this a true public good where scientists kind of donate their time. There was a recent paper last year where it's estimated that there is sort of a billion dollar altruistic donation based on the sort of over 100 million hours that were donated by scientists as just sort of this duty to just journals on the manuscript review alone. And so there's a question of how do we actually reward and recognize the reviewers for doing that work? Part of the question is sort of what motivates these scientists to do all this unpaid labor for these journals? And this is a fantastic survey done just a few years ago by a company, Publons that was working on sort of doing review or recognition for peer review and they surveyed about 12,000 researchers. A large majority felt that there was just this sort of intrinsic duty as part of their job as a researcher to do it, even though there are sort of pretty weak incentives and recognition and rewards for doing it. And so there's sort of a question of how do we actually find ways, especially as editors and sort of grant committees struggle more and more to find qualified reviewers to do this labor of reviewing? How do we actually motivate them? And so I thought the survey was at least an interesting one to get a good sense of what motivates the reviewers today. There's also questions around how effective your review actually works. There's some interesting sort of case studies at least showing that peer review has sort of been a gatekeeper of a lot of innovative research. This was a piece from almost 30 years ago kind of going through lots of different case studies whether it's the discovery of B lymphocytes or the Krebs cycle or even the famous Cape Fainies Altamira where peer reviewers have been traditionally a very sort of adversarial process of rejecting sort of high-risk, high-reward, high-impact research. And so how do we think about, again, mechanisms perhaps to align reviewers more with wanting to select projects that are actually sort of prescient in nature? There's also this fantastic tweet that came out I think about a year and a half ago from one of the top researchers that was studying spike protein design for the vaccines. And he had previously been working on the MERS spike protein and had a huge amount of trouble getting this published, these two particular protein substitutions in the spike protein and sort of tweeted this kind of tongue-in-cheek remark that after over a year of trying to get this thing published in top journals, he was pleased at least that it came out in the form of the vaccines at the end. So there's this question of, again, how do we do this reviewer alignment? How do we actually recruit reviewers who are excited about and not necessarily just adversarial in terms of what sort of projects they're picking, either for publication or for funding? There's also a question of how much peer review works as a feedback mechanism for actually improving the kinds of papers that are being submitted. This was, I think, a really interesting study during the pandemic where the large majority of COVID researchers were using preprint submissions to submit and share their results directly without having to go through sort of this filter of peer review and comparing the sort of publications after they had been reviewed and accepting the journals with the original preprints. And again, a large majority of them are not really changing the figures at all. Generally the conclusions or other texts is being pretty minimally changed. So if one of the major jobs of peer review traditionally has been to be a sort of feedback mechanism for scientists to help correct potentially flawed experiments or methodology, it does not seem really in this age of sort of peer-to-peer feedback that perhaps this formalized gatekeeping journal mechanism is necessary. There's also a question of how effective the reviewers are for grants. And this was a study a few years ago looking at the sort of NIH peer review percentile scores. And essentially the takeaway from the paper is that they're pretty good at picking out the very worst proposals. Usually the bottom 20th percentile correlates well with sort of various measures of researcher productivity and output. But beyond that, it's essentially picking at random. And so that's led various folks to suggest either lottery-based mechanisms for picking proposals after a certain sort of quality threshold or other things that are less burdensome given the sort of amount of time that researchers are spending on grant writing as well as on grant review today. There's also a question of whether reviewers actually agree on grants, which I don't know if that's necessarily a good thing. People have talked a lot about whether high variance grant making might actually lead to superior outcomes, but it does mean that when we have a very small number of reviewers often on these standing sections, it may be highly contingent on who ends up being placed on your panel in terms of whether you get a grant accepted or not. And so in that case, sort of the low degree of interviewer agreement is something that should give rise to concern. And so the question is now that we have so many open feedback and curation mechanisms. How do we think about sort of informal peer review as perhaps a new venue for accomplishing some of the goals of the highly structured academic peer review system? This is a case study that I actually quite like where after Thomas Piketty's capital had come out, there was an interesting sort of blog post on Tyler Cowan's economics blog, Marginal Revolution, where this Grabson at the time found that Piketty had kind of made an error in terms of how he had calculated depreciation and that sort of comment on this random blog as sort of an open peer review blew up so much that the Grabson was later invited to submit a paper to the Brookings Institution and really got quite a bit of a claim from the review that he made. And so in some ways, open peer review can be a mechanism for building reputation, especially for younger scientists who have not established themselves yet. And so I think that points to a potential interesting case for one of the benefits of allowing open review that actually confers reputational benefits to the reviewer. There's also been sort of a lot of interesting peer review of biomedical images on a platform called PubPure, where there are various researchers, most notably Elizabeth Bick, who have spent a lot of time kind of going through, looking at images, picking out possible fraudulent data, which is again, something that reviewers tend not to spend a lot of time at, which shows that there is room for sort of not only just post-publication peer review, but also again, the power of kind of openly letting people comment. And actually in these cases, both of these figures were from papers that were kind of similar to these biotech stocks that later, after kind of having this revelation of fraudulent data, had massive sort of swings. And so earlier there was sort of a workshop of series around market mechanisms for science and there's a question around, what does shorting bad science look like? And in some ways we have at least some interesting case studies in the small cap biotech public markets today to show how we might be able to incentivize different reviewers to go out and look for fraudulent data. And whether these sort of skin in the game, financial mechanisms distort science is I think an interesting kind of experiment to run. But at the very least it creates an open feedback for people to kind of again comment on research that might not be of sufficient quality or caliber. So now I'm gonna just go quickly through some new review mechanisms. I think there's a lot of interesting experimentation, especially in sort of the AI community right now with open review. There's a fantastic researcher at Carnegie Mellon, Nihar Shah, who's done a good job sort of summarizing the different mechanism design experiments that these various AI research communities have done in terms of how papers get bitted on, in terms of how years get selected, how you ensure that there aren't sort of collusion and conflict of interest rings. So I won't dive too much into it, but it would suggest anyone who's interested in exploring that literature to take a look at Nihar's work. There is also a lot of interest in using sort of replication and prediction markets to kind of estimate the reproducibility of scientific research a priori. And there's data that shows that at least some of the work that does not replicate well can be pretty quickly picked up by these generalist kind of prediction market participants, especially in the social sciences. DARPA has also put out some competitions and actually funded some fairly large markets to steep. There's actually predictability in terms of identifying non-replicable studies. There's also a pretty interesting project early in the COVID-19 preprint sort of deluge of whether prediction markets would help adjudicate which preprints would actually end up being published. And so I think there is a kind of a wealth of possible sort of mechanism design space to explore around new ways to elicit more accurate reviews, whether that's on replicability or on sort of fraud, as I mentioned earlier. I think one of the other really interesting mechanisms that I'm pretty curious about is sort of peer-to-peer reviews. So this was actually a project that came out of, I believe, these astronomy communities where booking telescope time was something that was highly finite. And so how do you actually evaluate which proposals should be selected? And as they were getting more and more projects, they decided we should actually get the submitters of these projects, these researchers themselves, to come together and try to adjudicate which projects should get telescope time. And so kind of based on that pilot, the NSF actually ran this sort of distributed peer review project where each submitter of a grant would also take on the burden of reviewing things. And I think that's kind of an interesting mechanism of sort of peer-to-peer science that again might be an interesting mechanism to explore. There's also been a lot of interest recently in sort of using scouts and science angels, which David over here has also pioneered at Experiment Foundation, as well as some other organizations of trying to empower scientists themselves to rather be sort of inbound reviewers, could be outbound curators of projects. And that may reduce the burden both on the applicants for these different grants, but also reduce the burden on the reviewers themselves when they already have their sort of existing scientific networks. So I think there's a lot, again, a lot of interesting design space. And I kind of want to highlight one last one, which is kind of this notion of certificates of impact, which a lot of folks have been talking about. Actually, in the science space, there have been attempts to do sort of reviewer certificates. Elsevier has been working on it for a few years. Publons as another sort of review technology platform has also been working on sort of this notion of reviewer certificates. But in this case, because most of these reviews are closed and blinded, it's mostly a way of letting the reviewers know, or letting reviewers signal that they've been completing these reviews. It's not really a sort of a sense of alignment mechanism. And so I think it's worth thinking about, what are ways to kind of improve upon those very rudimentary certificates of just sort of, I've completed a review to something where the reviewers actually have sort of hopefully long-term incentives to leave prescient reviews. And I think there are a few sort of considerations. I think one is around this trade off between privacy and transparency. There's always been this argument that peer reviews should be this sort of blinded closed process to elicit the most accurate reviews such that there aren't sort of adversarial conflicts between the reviewer and the researcher. And I think there are crypto economic primitives that are starting to give us interesting tools to play around with both reputation and identity, where perhaps we can still accrue reputation in a way that is privacy preserving, whether they're sort of time bound pseudonyms or sort of pseudonyms that are locked away in some sort of zero knowledge proof methodology. I think there's also a question of how do you actually trust which reviews in an open review system are reliable? And in terms of sort of short-term reliability, I think there's a lot of interesting experiments going on in sort of web of trust systems in terms of building a social graph on top of a knowledge graph in terms of what sort of reviewers you actually have existing trust in for curating and reviewing the research that you're evaluating. And then there's also been a lot of talk around verified credentials and soulbound tokens as another sort of mechanism to quickly assess the short-term reliability of reviews. But I think really what we wanna move to is this system where we're doing long-term evaluations of depressions of reviews. And I think a lot of the sort of retroactive mechanisms that people have been discussing here, whether they're bounties or prizes can also be applied to look at reviews where in the sort of startup and venture capital ecosystem they have these sort of portfolio reviews where they have looked at anti portfolios of companies they have missed. And I think there should also be sort of retrospective analyses of false negative reviews that are happening in science. And we should also consider having sort of prizes and bounties for the best reviewers in science as a key stakeholder in terms of how attention and funding gets allocated in the overall research ecosystem. So I think to kind of piggyback on a lot of the discussion and ideation around impact certificates this weekend I think there's perhaps some new primitives to introduce to the peer review space. And I think I've run out of time and there's a lot of slides there but hopefully there's some interesting content to kind of grapple with and think about peer review as a public good there. Thanks.