 So with that, we turn over to our keynote for this morning. And it's my distinct pleasure to introduce our keynote speaker, Dan Whaley. I met Dan, I haven't, this is really the first time I've ever met Dan in person. And so when I met Dan, what I really mean is when I met his online presence and his online creation, that was in November last year when Hypothesis, his, I think, main current endeavor, was a Kickstarter for, what do you say, startup funding. And was immediately taken by the broad curvy that the project took and by the broad scope. You know, we as scientists, even most of us still pretend that peer review is actually working. And those who feel that perhaps there's something to be fixed about it is still a rather small minority. And here's somebody who thinks about peer review, i.e. others reviewing information on the internet, and that whether it's science or not science, much beyond science. And tries to apply the principle of peer reviewed information to the internet as a whole. And so I'm very fascinated by the project. I'm not going to make a secret out of that. I'm extremely pleased that we were able to get Dan here. Dan has had a very distinguished career. He set up an e-commerce system more than 15 years ago at a time when I was still trying to have my head around Linux to start with. And he's worked in the meantime with climate scientists. He's funded startups. And his latest endeavor is Hypothesis, about which I hope is going to tell us a little more this morning. So without further ado, Dan Wiley. Greetings. So if I understand correctly, does that work? Awesome. Anybody who's here in this room right now has survived almost a week of presentations, maybe somewhat like this one. And so I congratulate you on that substantial accomplishment and hope that I can keep the momentum going here towards the end. So as you can see, there's a URL up there for those of you who are online. You can download the presentation and follow along. There's a lot of detail in the notes and URLs and references and so forth. So that's bit.ly-ievo-hypo. And that will survive so you can do it later. Thank you for having me here today. Thank you actually very much. So I've come all the way here to Ottawa, not only because I think that you as evolutionary biologists are really at the forefront of both thinking about tools, building them and using them. But I also really resonate with the commitment that I read here and on the website and so forth and heard from Hilmar in terms of your commitment to open source and open principles. And it's in my opinion really the only way, the only direction of the arrow that's pretty clear and obvious right now. And it's a question of when the rest of the world is going to catch up. It's really the central thesis also behind the project that I'm involved in. And it really delights me to be around like-minded individuals. So my name is Dan Whaley. I got started in Internet Things in the mid-90s. I started the first travel company on the Internet. If you've ever booked a travel reservation, there's a good chance it's gone through some piece of software I've written. And we built that company up to about 600 people and took a public and sold it in 2002 Sabre. And it forms the backbone of their online reservation octopus now. So I'm going to talk today about peer review. It's for me a really fascinating subject. It's one of those terms that some of us use almost every day. And yet it means really different things to different people. And its meaning has really evolved quite dramatically over the last 100 years to mean a variety of things. And I imagine it's still very much in the course of its evolution over the near and far futures. Clearly I'm in front of an audience that is intimately familiar with peer review. So humor me. I do think it's useful to kind of reflect back a bit and try to remember how we got here. As well as try to tease apart the different elements that are at play right now. And I also want to show you with you a little bit about a project that I'm involved in called Hypothesis. So I think more talks ought to come with warning labels. Here's my cut at one. First I'm not suggesting that we replace all the systems you use today for review. Those have been created by your communities and are tuned to the ways that you guys like to do things. But rather that there might be something else, something more that we can create that has new characteristics and is perhaps useful. One of the most important things that recent investigations in open review have found is that it's probably an exercise in futility to create a one size fits all approach. But rather that what we need is a flexible framework that can be adapted to different systems, different thinking, different needs. Secondly, that we and I have only gotten as far as I have through the help of a tremendous number of people that have been very generous with their time and thinking this is very much an experiment. It's a collaboration. If you have thoughts, insights to share, I'd be very interested to hear them and really this is a dialogue more than anything. Hopefully that part will come towards the end here. So to begin with, it's probably really important to point out that nothing I'm going to say here is at all original or even very recent, far from it. Case in point, the Talmud, which has been around for the last several thousand years. It's a compilation of two things and I'm not familiar with this material, but they tell me it's called the Mishnah and the Gamara. The Mishnah, the original oral history, is in the middle and it's surrounded by interpretation, argument and commentary, which is often threaded. In addition, there are footnotes and off page references. This is post publication peer review in its earliest form. Despite there being some very interesting early models of review, scholarly publication for most of the last several thousand years has been done without really any formal peer review model at all. It was really the scarcity of people qualified and capable of doing research, the difficulty of publication that was the primary check on the spread of new ideas. Copernicus was a member of the Order of Jesuits, which was kind of an early organization, still around today, which served as a way to legitimize the ideas and the thinking and provide a scholarly environment for its membership. But even so, Copernicus waited until his death to publish, his most controversial work published on the day of his death, precisely in order to avoid what was likely to be an unfavorable view of his peers. Gutenberg, who didn't invent the press but who popularized it through his simple design, really was probably one of the other people in this beginning of the implementation of technology into the peer review process in terms of accelerating the technology of distribution, which simultaneously began to also affect the quality and the nature of review. In 1752, the Royal Society took over the editing functions of Phil Trans and instituted a process made popular by the Royal Society of Edinburgh where materials sent to it were made available to a select group of editors. This is also the same issue where Ben Franklin describes his experiment with a kite and a key. In the early 1900s, typewriter and then the carbon copy machine made it easy to circulate the drafts of papers to committees. And continuing, really changes in technology over the last 100 years have really defined that acceleration of distribution and the nature and character of the review process, which brings us of course to the internet. The impact and the latent potential for transformation of which really cannot be overstated. We've only begun to scratch the surface. This image was produced by mapping all the IP addresses accessed on a single day. This was some years ago, so probably much different now. It didn't take long for the internet to make its mark. In 1991, Paul Ginsberg created xxx.lanl.gov, now known as archive.org. And in a series of articles he calls blurbs, which ironically are not either peer reviewed or available on archive.org. He lays out his thinking behind why he created it and how it works and really what the potential of it is. This is not peer review. Really that they use these heuristic screening mechanisms to ensure that basically stuff meets a basic threshold. And that it wouldn't be preemptively rejected as nutty, offensive, or otherwise manifestly inappropriate, which I kind of love. And I'm going to read this. This is the only slide I'm going to do this on, so bear with me, because I think it's really fascinating. The vast majority of the articles can, would, or do eventually satisfy editorial requirements somewhere else. Virtually all are in that gray area of decidability and virtually none are entirely useless to active physicists. That is probably why expert archive readers are eager and willing to navigate the raw deposited material and greatly value the accelerated availability over the filtering and refinement provided by the journal editorial processes. Even as little as a few months later. So what do we have here when we look at that modern journal over the last 25 years or so? It's really benefited from this internet revolution through extreme reductions in the cost of distribution. While yet most of the value provided other than say basic editing or layout functions is provided for free through the labor of the scientists in the field. And which probably explains why very recently, right about here in the curve, we've begun to see some very interesting changes in the tension of this model. This is about where NIH created its policy to require that any taxpayer funded or NIH funded research be made available for open access about six, I think it's six months after publication. So how much does that peer review process cost? Apparently not very much. The Journal of Machine Learning might be setting the world record here in terms of the cheapest peer review in the world. It's actually free to both the authors of papers as well as the consumers of them. But if you aggregate their costs and their overhead, which are paid for to some degree out of pocket of the editors of the journal, it works out to about $10 per article over the last thousand or so. It has the eighth highest impact factor out of 108 journals in its web of science category. It's higher than the factor of the springer publication, actually, that its original founder left. And there's really some excellent articles on this and how it came about. So if we think about what's also happened around the same time is that the subscription cost for these journals has really gone through the roof. And about the point towards the end of that curve brings us to the present day where a Harvard, for instance, two months ago announced, put out a memorandum to scholars and academics at Harvard suggesting laying out, number one, that the cost of publications had skyrocketed, that it was no longer really sustainable, and suggesting nine points of action that professors and scholars at Harvard might consider, including number two, which was perhaps the most provocative one, in that they moved prestige to open access. This year has really been a fascinating year. Here we have one of my favorite Twitter accounts, fake Elsevier, talking about the RWA bill that Elsevier was funding through a pair of New York Senators. And many other things, obviously, happening this year, SOPA, PIPA, ACTA in Europe, there is really, it's really a very interesting time to be alive in terms of the debate. We now have had in the last month or two a petition in front of the White House requiring, asking the president to look into the matter of requiring all agencies, not just NIH, to make taxpayer-funded research open access. And so that really kind of brings us to the question of peer review, and as we're considering it, I think it's important to ask ourselves some basic questions. And to the degree to which peer review really helps with any of them. So does what we want to read, does what we want to consider, is what the article that we're thinking about doesn't meet a basic threshold of quality. Should we read it now? It just came out recently. And how should we think about it over the long term? So there are three different kinds of questions we might ask. And we might ask ourselves how peer review, what role peer review really plays and what role other kind of characteristics or factors play in terms of those three questions. We might kind of lay them out a little bit like this. Clearly the fact that an article is peer reviewed and in a notable journal can send a strong early signal. But that's pretty rapidly eclipsed by the social cues that we all get or maybe don't get from our community. If the article is an important one, our peers were definitely going to find it, regardless of where it is and whether it's in a journal or on the side of a bathroom stall. Eventually how we're influenced is really more over a question of to what extent the new thinking gets incorporated into the world around us. And that's really the true test of an article and whether it continues to be cited and thought of and used over time. So when you think about whether you might consider a particular article you're thinking, if somebody tells you an article or you see a citation or in reference, you probably go through a host of questions really within milliseconds. And I've tried to kind of arrange them here in time sequence beginning with where that value was created. So who was this article written by? When was it written? How old is it? Was it peer reviewed and by who and what were those comments? Who's talking about it? What journal was it published in? What was the real impact that it had on the field? Who cited it? How often? And of course what's happened since. And if you could kind of arrange them on our timeline like this, they might kind of sequence themselves into different chunks of value produced by different groups that are arranged in different time orders. The journal article and the journal model that we've had so far puts a paywall between itself and the community. It really keeps the community as an owned presence of the journal and makes it difficult to access that community from the people that are participating in producing knowledge and science to begin with. What open access does is really shifts that wall back potentially even behind the reviewers depending on the journal's model. And really allows us to think about the value that each of these different categories provide and where we should get those portion pieces of value from. We have a concept also that's emerged recently called open review, which you know many of you are familiar with, where the identity of reviewers may be to some degree more or less transparent depending on the journal as a way to try to encourage more of a critical review, provide that accountability and accessibility to review process. So if we were thinking about how peer review is changing here over time and what might be possible in the future, which I'll get to in just a little bit, we might kind of lay it out like this, the ghost of peer review past, present and future. So in the beginning, so the one thing that's been consistent is that we always go through a draft process where we create, we're writing something, we're creating new knowledge, new science and we pass it around amongst our friends, amongst our co-authors, get input on the paper, go through that drafting process. But in the past what we've had is kind of a private peer review process that's really locked away from us and where what happens post publication in the way that we think about those papers is a question of maybe the prestige of the journal, the citation rank that we've come through, that we've accumulated. The present, if you look at models like PLOS or Archive is where we still have that draft review process but maybe we have some assisted heuristics like Archive uses or a rapid review process like some of the other journals are using to get us to a publishable paper without necessarily that same kind of review process at the beginning. And the way that we think about those papers and that knowledge is really more of a question of the way that it becomes incorporated and much less a factor of which journal it's come from. And so if we think about how the disruption is continuing to happen what we're potentially looking at is where that review process is shrinking now and the potential for review, the potential for discussion and thinking and evaluation really is shifting in front of the or after rather the publication process. So peer review is kind of a convenient tar baby. It's been so frequently disparaged. I probably won't go into the history that this is only one of the most recent ones. This is Michael Eisen's blog. He's the founder of PLOS. But the summary seems to kind of go like this. It takes a long time. Kind of encourages a conventionality that echo chamber phenomenon that it doesn't actually work very well to do the thing that it purports to do. There's a lack of accountability. Screwed up sentence there. That the anonymity works across purposes to the process itself. That it takes an extraordinary amount of all of our time and it really fails to limit the volume of the research that most things eventually if they're even at the base level of quality of a scientific publication get published somewhere eventually. There's over 10,000 journals out there. And so I think it's really important to think about what we really need. And this is one of my favorite quotes. That maybe what we need doesn't have to be very complicated. In that the value of things, this is from an excerpt from the comments around the open review, one of the popular open review papers, is that just being able to have a conversation in an open way that extends in an open and public way, that extends beyond that publication process or through that publication process is good enough. And so if all we're trying to do is enable conversations but we're trying to do that on the internet, then we need to introduce a new concept. For those of you who are ham radio operators, you might be familiar with this thing called the squelch knob. So a radio set the squelch knob lets you set the noise floor. So if you turn that knob at some point, the background noise, the static drops out and you get nothing. And when you get a signal when somebody is actually talking, then you can hear it and you can hear everything they're saying in the full range of their voice and so forth. So in putting together our project, we have begun to think about how exactly we designed that squelch knob here. So if you think about the noise that you get on the internet, there's really a threshold. It's that threshold that Archive is going for when for instance they use heuristics to determine what meets the basic test of peer review. You want to keep out the spam, you want to keep out the obvious trolls, the people who really have no interest in participating at all in the conversation. These are not necessarily the peers that you disagree with but the people who actually are out there to damage the dialogue. Here in the system that I'll talk about, in the hypothesis that I'll talk to you a little bit about later, we approach those two, that noise floor, through two different mechanisms. One is through the identity model that we use and the second in terms of creating the headroom, we've established that noise floor is through a process called metamoderation. It's actually a term that was coined by the creators of the early internet site called Slash. If you guys are familiar with that. I'll talk a little bit more about how we implement that later. When we get, when we're able to establish quality conversation, pretty extraordinary things are possible. There's one example that's called the polymath project and this researcher had the idea to simply open up some of math's most difficult and interesting problems to an open forum. This first one happened about five years ago. It was a problem called the density. There's Jullet theorem which has to do with, one figures out the minimum possible configurations to make a tic-tac-toe board solvable. It took, very hard problem, took 37 days, 27 different teams, including a high school teacher and his class to solve over 800 comments. No super mathematician emerged, which was one of the surprising things. It was really kind of truly a pure collaborative effort. So this kind of brings us really to this, the kind of basic element behind why this stuff works. Is that, and this is in reference, I think, to a recent paper by Hugo Mercier and Dan Spurber called, Why do humans reason? It's an argument for an argumentative theory. Talking really about how, and kind of proposing the idea that reasoning was not really designed, quote-unquote, by evolution to pursue the truth, but really as a social tool to help us win arguments. And that, in fact, really falls quite short of reliably delivering rational beliefs and rational decisions. It may oddly be detrimental to rationality itself. That it can also counter-tuitively lead to poor outcomes, not because humans are bad at reasoning, but because they systematically strive for arguments to justify their beliefs or their actions. Which explains, obviously, confirmation bias and motivated reasoning and many other things. But that, on the other hand, that when we reason together, our confirmation biases tend to balance out. And this, of course, explains why peer review is potentially successful and the scientific process in general. So if we imagine ourselves 100 years in the future, looking backwards, what can we imagine knowing about the science of today and about what we'll learn in the years to come? What would we like to know, and what infrastructure do we need to provide it? One way to do that might be to stand today and look back 100 years, looking, for instance, at a paper of that time, this is Einstein's General Theory of Relatively, published in 1916. And imagine being able to see all the drafts of that paper in all both pre- and post-publication. Of course, we had one author, so there may not have been many of them. The commentary over the years, particularly what different notable people have said by year, by decade, how that thinking has evolved and what the links are between this paper and the others that have followed from it. What's the tree of knowledge that's sprung from this single root? What the impact has been on that subsequent science and how sustained it's been over the years? What was the pulse and how sustained was the pulse of this particular paper? So if we were going to try to design something that would solve for that, it would solve for speed, fairness, quality that would facilitate both draft reviews as well as post-publication review and commentary that would facilitate a kind of cross-disciplinary review that would blend the edges between domains that would work maybe not only on science and academics but on any other domain that would provide some accountability but also maybe occasional anonymity for reviewers, in particular maybe junior reviewers in a field. It works both on the formats of today as well as on what emerges in a beyond-the-PDF future and is open-source, standards-based, and so forth. What would that look like? So that's the question that we asked ourselves in creating and laying the foundation for the project that I've been involved in. It's called Hypothesis. And I'll take a little time here just to lay out the basic elements of what we're doing and perhaps if we have a little time at the end give a quick demo of very early prototype. So our mission at Hypothesis is to transform how we know what's credible in the world around us by enabling the crowdsourced peer-reviewed information everywhere. This is a draft statement as these things tend to be but I think it encapsulates what we're trying to do. So we're not the first. I have a spreadsheet of now accumulating previous projects that have been trying to do things like this, some of them closer, some of them not as much but similar. It's got about 55 lines on it now and growing. I've stopped being amazed when people tell me about new projects. And I really experienced seeing some of these come and go over the years. I kept telling myself, wow, somebody's finally got this. I don't have to worry about that problem anymore. And then years would go by and we still wouldn't have it. And so I finally have got frustrated and started calling some of these people up and spent about a year interviewing about 25 of the founders of previous efforts and people who were working close to them and asking them why we still don't have this. These people were remarkably very generous with their time. And out of that, those interviews and the kind of analysis of previous efforts, I really came up with a list of about 20 different reasons and sorting for importance came up with a short list. Nearly none of these projects had any kind of peer review mechanism at all, no way to separate even the most obvious spammer from participating in the conversation. They weren't annotation based, meaning they didn't allow inline markup, it was usually below the fold, kind of on the whole page. There was really no way to link into, there was no addressability for what was created. They focused on building technology but not really on building community, which is maybe one of the most important and yet under-served parts of this challenge. They weren't standards based, they weren't open source and they weren't nonprofit in structure. This last bit is probably, it's at the bottom but it may be one of the more important elements of this, that for-profit, having been a person that's created a for-profit organization that was quite successful and I assume maybe being potentially involved in similar efforts in the future, that creating a for-profit company and organization to serve this kind of objective really puts you at odds with the intentions and the motivations of your users and I think this is perhaps one of the most important things that we've hopefully done. So what we've done is try to flip those on over and really create some basic elements of the project. It's an open source, or if you're familiar with Richard Stallman, it's free, not in the sense of free beer but in the sense of free speech. It's standards-based, nonprofit. We're using a pseudonymous, occasionally anonymous identity model with an emergent organic reputation model that's domain sensitive and with a focus on the long-term archiving, the challenge of long-term archiving. We have a few grand challenges in putting this together. One is to create a beautiful product which is quite difficult. The second is really to design a reputation model that works at elevating a signal over that noise floor that works also at finding metamoderators, finding peers in a world of infinite domains of knowledge that is able to stick annotations to things in a robust way, which if you ever think about how quickly and how frequently text changes on the web is actually a profoundly interesting challenge. It needs to work well across the web and across the sea of different documents that we all encounter which are quite diverse and we really need to focus on how to get this thing started and boot it up. We're approximately here in the timeline. We have been working on, actually there should have been a kind of conceptual phase above that, but we're working really on the basic mechanism. We kind of divided the project into two technical components. Creating the ability to reach into documents and annotate them and anchor those annotations and the second part of the project is really beginning to design that reputation model for how peer review works and how peers, what the definition of a peer is and then after we've built technology or continuing to build technology, begin to focus on how to get it started. Hypothesis is really three things. Annotation, deep linking and peer review. Annotation is like an arrow with a payload. So if we look at the precursor, the kind of model for early annotation, this is Sir Isaac Newton here espousing something that noteworthy, which I can't read, in a treatise on optics. We have two parts of what we have in digital annotations today. The first is the target, it's the place in the page, it's the document itself, the book and then the location inside the book and then it's that payload, that thing that was said, the body of the annotation. So marginalia of old is kind of like half of an annotation. The third, that digital annotation really adds that third component that address that can be pointed to, like this. This concept of annotation is something that has pretty substantial momentum behind it. There is a very large working group, which is part of a WC3 community standard now called Open Annotation, Open Annotation Collaboration. There's a draft RDF standard that's been developed over the last couple of years by a collection of groups at NISO, the Internet Archive, a number of publishers like Amazon and O'Reilly and Burns & Noble and so forth, a bunch of academic institutions ourselves. And I'm not going to go into detail on this if you're interested, you can download it. But it basically lays out those parts of the arrow and how you identify things, how you target different types of media and how you create payloads, what the attributes of those are and how addresses are formed at the tail end. So if we have annotation, what's possible? Right now on the Internet, we point to the top of things. Occasionally we can use anchor tags if you're familiar with HTML to point inside things, but only when those tags have been predefined by the creators of the original HTML page, which makes them very difficult to use by other people in any kind of practical sense. What's possible in the future is instead of pointing to the top of things to start to point into things, to be very precise in how we locate the discussion that we're having and the links that we're creating. Once you can point into things, interesting things become possible. Like, for instance, pointing many times into things as you're having a discussion and making different points, you can point to different elements of the thing that you're referencing. Annotation lets us point into more than just documents. We can point into different media, pointing into, for instance, a timecode in a video or even to a specific gene or a molecule inside of some structure. There's early implementation of the open annotation model by a woman in Australia called Anacryst, which lets you annotate and share those annotations on, for instance, genes and so forth. Stuff you guys know a lot more about than I do. So if we're able to... So the other really important thing about annotation is that instead of the web now being a sea of documents that are really hosted and created by other people over which we as readers have no control, we can now use annotation to create links, to curate connections and references and so forth between things, even when we don't own them ourselves in a way that other people can then discover. Annotations are not purely comments. They can also be typed, categorized, and semantically. We can have both machine annotation, which some of you may be familiar with, is already being done fairly widely, as I understand within your field, as well as human-based human annotations, which can have various qualities, like, for instance, the nature of the commentary that they're providing whether they're asking a question or posing an answer, perhaps just supplying a hypertext link that the original author neglected to offer to supply and pointing to other similar articles that talk about a similar subject. And this typing allows us to do something very interesting, which is to look at the totality of annotations that might occur in a single place and to come to conclusions about them. In fact, to almost create kind of a heat map of sentiment and of interest and activity on top of a document that lets us see what's going on, see where the conversation is without necessarily having to dive in and read that conversation immediately, but only after we see something interesting. The third really important part of the project is the peer review model. It turns out that this is a very active body of research, not only in peer review, but in the computer science sense, what's called reputation modeling. So in the humble appreciation of the fact that a lot of people knew a lot more about this than we did, we, through Sloan's generous support, through a workshop in February, we had about 50 people from around the world who generously came to San Francisco for three days to help us design the model that we're going to use to lay these annotations on top of documents. And we basically learned a couple key things. One is that we need to balance direct participation with objective control, so people need to be able to come in and directly participate in conversation. But there needs to be some kind of a control on top of that that works on a separate scale, time scale, or interaction mechanism that provides a layer of objectivity to that process. And that ultimately that final objective layer should be semi-random and probably blind. The semi-randomness, the pseudo-randomness is vital. The blindness is helpful. It also needs to be fun. So in other words, if it's not delightful to use and particularly useful to use, people won't. It seems simple and obvious, but important to remember. So the design choices that we kind of came out of that three days with were fairly simple. They've gotten much more complex since, but they basically roll down to a pseudonymous identity model. So Twitter, for instance, is a pseudonymous system. You have a handle. You can choose to associate your real self with that identity if you want or not. The choice is up to you. Facebook has a real identity model. So you have to be your real self. If they find that you don't have a real identity, they'll terminate your account. Other models like 4chan and early internet community, if you're familiar, head in much more of an anonymous model. People could participate anonymously. So we've chosen a pseudonymous model for this system. A threaded direct reply approach to discussion. So instead of some systems are flat, there's no threads. Some have threads. We've chosen a threaded model. Where sentiment and score, a reputation score, propagate up through the threads, kind of from the lower right to the upper left. And a metamoderated loop controls reputation. I'll talk a little bit more about metamoderation. Metamoderation is blind and semi-random. And metamoderators are chosen by the reputation through domain proximity. I'll talk more about that in a second. So what is that identity model? So how do we basically create that noise, establish that noise floor? Pseudonymity definitely doesn't give us that. But pseudonymity gives us the basic interaction model that we want to have to let people be able to control their identity and to what degree people are aware of who they are. What allows us to kind of control identity is using, there's a variety of different approaches. We've chosen one called two-factor authentication via SMS. So that basically just means that when you sign up for a new account, we'll ask you to verify with an SMS message. What that does interestingly is it confines people to the namespace of telephone numbers, which are actually quite difficult to achieve. You've got to usually go out and buy a device. You might be able to accumulate 10 of them if you really wanted, but that would be a pretty herculean effort. And that it prevents things that happen where people create hundreds of thousands of accounts and use those new accounts to game conversations that are happening in online systems, sometimes called the Chinese water army effect, because they seem to do it frequently over there. Also potentially using an invite-only period using what's called invite trees where we invite people and allow them to invite friends of theirs. So we're still tuning and working on the system. We are considering allowing kind of a limited anonymity if you've already gone through that two-factor process. You'd be able to contribute or to post anonymously if you potentially on a privileged basis if you've kind of achieved a certain level already, maybe a modest level of interaction and reputation in the system. We're using these two moderation loops, moderation and metamoderation, and basically it's fairly simple. Moderation is basically the threaded discussion that happens. Metamoderation means that occasionally we ask somebody who's established a high reputation in the system to look over the shoulder of one of those conversations and just decide whether or not they concur with the point of view that others have taken and to give it an up-or-down vote. And that very simple statistical phenomenon serves to create a tremendous amount of headroom amongst people who tend to contribute on a high-quality basis, surprisingly. And the other kind of interesting attribute of the system is in the way that those metamodering opportunities are provided. So, whereas contributing directly to the conversation happens in those threads, the metamoderation opportunities that are presented to high-quality members of the community comes through an activity stream. And so how do we decide whether to put a particular opportunity in front of a particular high-quality person? So we want to make sure that people in economics are not asked to weigh in on evolutionary biology because they probably may not know much about it. And so the question is really how do we graph domains and individuals as nodes within those domains? And how do we find the edges and kind of understand the nearness and the proximity of an individual to a particular opportunity? This is really an opportunity for us to also be able to kind of set the dial in terms of how we blend domains together. So people who study climate modeling may actually know something a little bit about economic modeling, but maybe only some of the people and maybe only in certain circumstances. So this I think is one of the most more interesting opportunities that's possible in the system. This is obviously a fairly complex concept and one that we'll probably find a cruder approaches at the beginning to approximate and more sophisticated approaches to use later. There will be some social qualities to the system. You'll be able to follow people that are in your domain who are annotating papers and so forth, see what's kind of happening. Be able to form groups. So if, for instance, IEVO bio-conference could form a group of all the people at the conference, you could follow the people that were in that group or perhaps your professional society or a group of private reviewers that you've asked to pre-review a draft of a paper. You'd be able to invite them into a private group and share with them something that they could annotate and share those private annotations with you. Later, if you were satisfied, you were all satisfied with them, you would make those annotations that commentary public. So this is one of the ways that we are hoping to address some of these kinds of needs. Score and sentiment. We're not going to spend too much time on that. Basically, propagate up through the threads. This will be available through diversity of different interfaces, both plugins. You'll also be able to follow short URLs that will bring you to the source material and a little bit lay the conversation actively on top of it. You'll be able to follow embedded links from other places like blogs or other documents that contain links directly into an annotated discussion around another paper, for instance. So really allowing us to utilize the full power of that annotation as an arrow. It'll be a website, obviously, and see kind of a zeitgeist leaderboard of what's happening domain by domain as well as a full set of APIs. We partnered with the Internet Archive, which is the institution we think has the best track record of demonstrating the ability to store things permanently. They now have a copy of the web that's existed that they've been keeping since 1996 and updating every year. They're now looking to store, have made substantial progress on storing all the world's books and other ambitious tasks. They're going to store a copy of all annotations and also help us with versioning all the content that people are annotating. So every time somebody makes an annotation, we'll send them a pull request. They'll take a new fresh snapshot of that page so that we really understand not only what's annotated in as that content and those drafts evolve. They're also the co-lead on the open annotation project. So that's about it. I will talk a little bit about, we've said, okay, so how are we going to get warm this thing up from absolute zero? How will we choose if one of the mistakes that other projects have made is they've just kind of turned these systems on, that community hasn't been very big and that kind of small flame ember that they've been trying to nurture is going to get blown out by the internet. And so we're taking the opposite approach, really thinking about how we select narrowly for launch domains, actually initially even perhaps down to single pages or single documents, and to bring in high quality people in those domains to help us curate and kind of jumpstart those areas. So we've been to the question is how do we select? How do we select those domains? So we've kind of developed a set of criteria. The content needs to be relatively stable. Wikipedia would be highly unstable content. For instance, the Bible would be very stable. Longevity, so something with a pulse of a single day or two is probably not a good target, but something that might have the longevity of months or maybe years. Something where we can have single sources at canonical URLs are very important initially. Eventually we will, that's one of our grand challenges is attacking that canonical URL problem. Public interest has got to be fairly interesting, hopefully to the general public, but at least within the domain. Not highly subjective, everything is subjective, but some things are more so. We can't have an existing commenting system that we'll compete with. It's got to have high quality people that we can access and so forth. So when you turn the universe through that crank, you end up with some kind of very interesting, we think, target material. One obvious target is pending legislation, particularly in the internet and privacy realm, things like the SOPA bill. There's a very juicy one right now called the Trans-Pacific Partnership, the TPP. You guys follow those things. And also things like scientific papers, and particularly very interesting things like the recent paper a couple of days ago on archive around the findings of the Higgs boson, for instance. Terms of service, privacy policies, EULAs are also kind of generally in that wheelhouse. So most of you are probably familiar with a Long Now project. They've been working on this for nearly 20 years now to build this clock that can tell time accurately to within a second over 10,000 years. They're actually working, actually building this now in the desert. They got $40 million from Jeff Bezos, and they're going at it. And it's pretty extraordinary. I encourage you to go to the website, LongNow.org, and look at some of the videos of the machines that they're using to drill these 12-foot shafts in the center of this remote hillside in Texas, and crazy chainsaws that are building spiral staircases robotically. But the project that spawned this deep reflection in its proposers about time, about how as Moore's law accelerates, our perspective becomes increasingly foreshortened and about how counterproductive that is for just about everything that matters. And I think it's really time for us to start thinking about the long term. And that was really one of the goals behind this project, Hypothesis. How can we begin to build fundamental systems of information, of access and dialogue that can last not just for the now, but for the long now? Thank you. I did forget to give you a quick demo. It takes about 30 seconds, but I'll just show you the... This is a recent paper here. You may look familiar to you, hopefully. I tried to pick one that was present and current. We're at the Page of Nature's website here. As you can see, up in the URL location, and we have a sidebar that's available to us here. You can pop it out and close it back. You can see where annotations are with the tags here. This tag at the bottom shows you the count of annotations that are currently below you. You can pop it out by clicking there and kind of scroll through. I've tried to choose notable names here who seem to be having a discussion from the grave. This is what we call the Bucket View. This is the cluster of annotations that are grouped around this particular area in the article. We use a fuzziness algorithm to pick stuff that should go together, and then we place those annotations all together, clustered in this column. If there were many of them, you would be able to scroll down this list. Then under each annotation, you can basically select it and expose that threaded discussion chain if you want to create a new reply. You can do so here and so forth. This is kind of like the basic foundation. I can imagine a house with some studs for framing. There's a lot more to come in terms of being able to express sentiment and reputation and so forth. Guys were literally working on getting us even to where we were today here last night for the little demo here today. That's kind of the experience that you might have in interacting with something that's in your field. Interesting question. I don't know how many people heard that, but is there anything intrinsically, I assume you mean in my system, that might draw the elements of what would form the evidentiary basis for the review together kind of prepopulate it or put it within arm's reach, I suppose, of that review process. It's an interesting question. It gets into what people talk about as the holy grail of the semantic web and what we're headed towards. I think those things are possible, very possible. It depends on the domain and the nature of what that evidence is. Is it data? Is it previous annotations or works that are close in proximity? I think some of that stuff is very easy. Some of that stuff is considerably harder and which is what makes it a fun project for the long term, I think. So why shouldn't the review process be completely open? What is this about anonymity? Well, I completely agree with you. I think it should be open. But I'm also sensitive to what people have kind of framed as the Mark Zuckerberg view of the world and that just get on with it. Everything should be transparent and it's wonderful and I live in San Francisco too and it's a beautiful place. But not all communities in all parts of the world and so forth, certainly within science itself just as a narrow application and I read a lot of this literature about people challenging the question of peer review and of proposing these different open review systems. That there are some communities and certainly some individuals who do feel particularly constrained to speak when they don't have some ability to mask their identity when doing so. And so if you're going to try to design a basic framework, do you force that issue? Do you basically say everything has to be out in the open? Or do you provide some mechanism under some circumstance perhaps for somebody who's clearly established in the system to participate on a basis? Not that anonymity should preclude their comments being moderated and that those the influence of that moderation affecting their reputation. It's one of the benefits that we have in constructing a system like this. We can give you the ability to be anonymous but yet still provide repercussions for you being unfair in your comments in the way that you're contributing. So this is a question for us. This is by no means hard and fast but we think that it's useful to consider there being an anonymous way to preserve some degree of anonymity in certain cases. And certainly what we're trying to do is provide mechanisms not just for science but also for the world at large, for somebody in Libya, for instance, to reach in and annotate a local paper that might provide some unflattering criticism of their government so anonymity might be more important there.