 Hey there welcome to today's Protocol Labs research seminar. Today we are joined again by Joel Chan who is an assistant professor at the University of Maryland's College of Information Studies or iSchool and Human-Computer Interaction Lab. Previously he was a postdoctoral research fellow and project scientist at the Human Interaction Institute at Carnegie Mellon University and received his PhD at the University of Pittsburgh. You can find a link to Joel's previous talk with us below and today he will be talking about accelerating scientific discovery by lowering barriers to user-generated synthesis of scientific literature, which will include discussing how scholarly practices could be transformed. So Joel, I will let you take it from here. Good morning, good afternoon, good evening. I'm excited to share an update on what I talked about in the previous talk which was linked in the chat. And so let's jump right in. The focus of this talk is on synthesis. And so I thought we could start by talking about what I mean by that. My goal is to remove various defective synthesis so any scientist can ask better questions faster. So what I mean by synthesis is probably an intuitive concept by some examples include theories, models, design spaces and very good systematic or literature reviews. And the key intuition is that you create a new innovative conceptual whole that's created in some of the parts of things that you're integrating. You wouldn't just be surveying and saying here's a bunch of literature, but you'd be sort of like giving insight into here are some of the gaps here are where we should be going next, driving progress forward. So one concrete example of this led to a Nobel Prize. This is Astrid Dufflo, a recent Nobel Prize winner, who credited some of her key inspirations to a masterful survey of the literature in a handbook chapter of kind of developmental economics that really laid out some key problems in the field that she was able to sort of connect with her expertise in experimental methods. So synthesis is super important. If we ignore it, we risk wasting our time on questions that trivial, impossible, misframed. One of my favorite phrases is you can't play 20 questions with nature and win. In fact, there's a delightful paper that recently talked about how to play 20 questions with nature and news in the kind of area of brain training research where they lack this kind of synthesis theory of what's actually going on and just had a bunch of experiments. One of the key things that I'm interested in is this problem that synthesis is actually really hard. Not sure how much this resonates to your own experience, but this definitely resonates with mine and lots of people that I've talked to. One concrete example of systematic reviews, which are really important. And they're really, really difficult to do. They take a long time to pull off. And they're often not updated because it's difficult. And so often they're out of date pretty quickly after they're published. I think of this as a lower bound on how difficult it is to do synthesis, in part because systematic reviews are very singular focus on a single question, usually a single version. Whereas often we need to synthesize across multiple disciplines across different kinds of evidence at least and so cleanly focus on just a single dimension. However, right, so just to give another like metaphor here, studying an ethnography of medical system reviews. This bucket paper talked about this metaphor feeding its slave to the trap data. You have this sense that of fighting against the infrastructure. It's going to be easier. The growing burden of knowledge is making this harder. And especially as we face problems that perhaps will always into this pretty required as to converge and integrate ideas across multiple fields of knowledge. This is pretty pressing. So pretty, pretty neat. The core conjecture of this line of work is that it's not just about the tools. It's not just about our motivation. It's about the infrastructure. It's about the unit analysis. All right, why, why does Google scholar work that way. Why is it indexing papers. Why is it, what is the data structure that's operating on that's that's my focus. You know, when you go search for papers, you're looking for papers, but really what you care about is what's in them. The ideas that claims the arguments, the theories, findings, and the discourse relations between the right support opposition repetition lines of evidence lines of contradiction. Right. These kinds of things are what you actually care about. Instead, you get documents, metadata and article types. So a concrete example of this, if you're interested in whether bands are an effective intervention for hate speech, you key that into Google scholar. And what you get back is a bunch of papers, many of whom will, some will be relevant. They're all pretty much with me in the same topic area, but I'm going to answer your question about what's the evidence for this particular class of intervention in the setting that I care about. There is evidence problem solutions. They're not first class students. Same thing in other tools like semantic scholar. I should say it's not their fault. Okay, that these indexing systems will work with what they have. The data structure is what's at issue. Similarly, we have entire industry of system review tools and processes that are essentially dedicated to working against the underlying data structure. They're sort of like we've got these papers and we need to work our way around it and so we all these processes for screening for data extraction to get to the thing we actually care about. And then we don't share anybody else and we have to start from scratch next time around. This is a slightly cheeky mean about how this kind of system reviews are sort of a band aid on top of a, to me, fundamentally broken scholarly. That doesn't allow us to ask the questions that we want. So I think it's pretty important. I see connections between this terms of, I think it's an underexplored underappreciated potential mechanism for what people are perceiving as a slowdown in scientific productivity and progress. It's getting harder and harder to win at a well price. You got to sort of spend a longer time to kind of rock the field. We're doing more of our high impact science and teams, which brings its own challenges and overhead. And I'll give this somewhat controversial but to me pretty concrete observations of some slowdowns in the impact on research. So the core question is how to accelerate some of the things where we bite lowering barriers to synthesis. Today I want to talk to you about these cross graphs. What are they? Why do we think that they're interesting? Why is it hard? Why don't we have them? This is core idea about authorship bottleneck and I want to spend most of our time talking about this solution path of scholar powered contributions as a sustainable authorship model. Okay, so first this cross graphs, what are they? So comparing to what we had before where we're looking for bands and hate speech literature. Something like this is actually what we're trying to construct or look for, right? We have a question, are bands an effective way to mitigate anti-social behavior in online forums? We have claims, which are answers to that question, right? To be banning as a strategy is effective. Banning strategy cannot scale and then we have underneath that evidence, right? Particular results that are supporting or opposing these claims. Okay, this is a pretty intuitive model. We think about how we'll talk about it. Sometimes complain, this paper makes claims that are not supported by its data, right? Or we talk about theorizing from data. This is kind of core distinction between claims and evidence. This is not an invention of mine. This has been talked about for a long time. You know, decades, right? We've got a kind of decades of literature on standards for how to be actually formally represent a discourse graph. We have the Scolanto ontology was based on claims. We have micro publications. We have CPO. We have a bunch of these very mature, well thought out, clever standards for essentially how do we represent a discourse graph? Give you a little bit more intuition about why discourse graphs are a good idea beyond just the match on the surface with the kinds of questions we're trying to ask. I like to think about it in terms of three Cs, compression, contextualizability and composability. So compression is the first one that we were talking about where we actually want to find and manipulate compressed units like claims, not just whole papers, right? Papers contain lots of different things in them, but we actually care about manipulating thinking about the underlying ideas, right? So we've got these models of scholarly argumentation, but also in creativity with literature. If you break things down, that enables you to more creatively recombine them into new theories and models and concepts. Conductualizability is super important, right? This distinction between claims and evidence turns out to be pretty important, right? To really understand the scope of the claim, be able to question it, be able to repurpose it. It's really important to begin to, right? If you have a claim that most private annotations are useful to other people, you want to know how is most measured, what kind of annotations in what setting to really understand what this means for, say, some question you're interested in. Same thing on the band side, right? It really matters, for example, to distinguish sometimes between different subreddits that have different subcultures or different platforms or different kinds of behaviors and interventions and so on. The devil or diamond is in the details, right? We have lots of recent examples of the importance of context where we really care about unbundling, for example, children into different age bands. Turns out infants' risks are quite different from young kids who are different from teenagers and many studies that don't distinguish those. We lose resolution and ability to synthesize. There's a pretty fun Twitter account called JustSaysAndMice that kind of riffs on this where a lot of medical studies are reported on and talked about without the context that this was a mouse model, right? And I'll say, like, this implies X form for people, but actually the context is lost. This really impairs our ability to synthesize. So we need access to that easily. Okay. We see this actually in user behavior as well. Scholars will constantly reread during literature review. They'll return to papers. This is repeated also in studies of general sensemaking where you've got to keep the data information around. And in studies of computer-supported corporate work where we have this kind of knowledge to management infrastructures, it's really important to retain the context of how a thing was produced, how a piece of information was produced, by whom, and so on. One fun desire path that inspires this work is, you know, people repurposing tools for different functions. So qualitative data analysis software is meant for things like demanding analysis, interview analysis. But we actually see scholars a good amount of them actually repurposing it to do literature reviews because it supports the ability to do this dialogue between claims and evidence, between the theory and the data. Which is really interesting. This core distinction between claims and evidence seems really important. Lastly, composability. We get some of this from formality. If you have these units, but you also have connections between them, you can construct more interesting representations that help you reason about things like tables or causal graphs or arguments in time. Something like this. So as I mentioned, these are new ideas. I'm not the first by far to talk about this. There's a pretty large literature body of work building out these technical standards and infrastructures. We basically have most of the warehouses, they're built, but they're still mostly empty. In the bottom right, I'm not sure you get the reference to the field of dreams. We have some of these people talking about the, you know, we want to have an ocean of these micro publications. These describes crowds at the moment is no more than a puddle. And this is this is repeated across a lot of a lot of different platforms is a key bottleneck. I call this the authorship bottleneck. We have a different models for authorship that don't seem to be quite enough on their own. One of the most popular ones is a specialized curator model. We basically pay people or ask people to volunteer to do this extra work on top of literature. You can think of systematic use as one instance of that where they take in the dirty data, right, and then they kind of add structure to it. This turned out to be very difficult to sustain. This is an extra piece of work that people don't want to do. One very sad recent example is marked to cure effort that was curating literature for NGLY1 deficiency. And they had to shudder because they ran out of funding essentially, they couldn't sustain. Some examples of platforms like Research Objects Hub or Nanopublications, the number of active users is actually very, very small. So we need something more. Some people think that we can do this in a completely automated fashion with text mining. This happens to be partially in my field. It's very cheap, but it's significant accuracy and transparency challenges. Extractive summaries of research papers is very hard, still a very open problem. And, you know, some of the promising approaches that people are exploring today with language models. We have some pretty significant challenges in terms of accuracy and transparency. So I'm not optimistic that either of these are going to solve the problem of their own. So what I want to do is extend the space of possible contributions to look at this thing that we haven't explored yet, which is what I call scholar product contributions. Intuition. She had inspiration for the design patterns and some previous work that I've done where we should think about a collective brainstorming effort, right? Where we have lots of lots of different people producing ideas. Sometimes it's useful to really understand like how the ideas relate to each other, how they cluster. In the field of crowdsourcing, we have these judgments that they'll have people do. They'll look at these like three things to the same which ones are related. Again, similar to what we're seeing in scholarly domain, it's very tedious. People don't like to do it. It's annoying. However, one insight that we saw is that people naturally cluster ideas when you give them a digital whiteboard. And this means something. The ideas cluster together and there's some meaningful relationship between them. And we can exploit this. We can integrate the usually tedious semantic judgment work into this intrinsically morning activity. And then construct things like an idea map that helps coordinate efforts deliver more interesting inspirations with basically no extra work. Right here and integrating to the work that people are already doing. That's one of the things that we're not exploring enough yet. I want to explore more. Okay. So specifically, I'm interested in exploring how we can integrate scholar power contributions and discourse graphs into individual and collaborative synthesis practices. Right. If you think about it, people read lots of papers. Lots and lots of papers. Right. Just some, some rough estimates, but on the envelope, you know, multiply the number of faculty by the number of self reported number of papers read per year. We're in the ballpark of about, you know, 100 million papers read per year. Compare that to the size of literature is pretty big. If you look at it by itself, but it's around the same order of magnitude. Right. So this is, I think an interesting source of untapped creative exhaust. People are already doing all this work. We're wasting it. Whatever we could, you know, integrate into that work. It's also a little bit more feasible to me. There's lots of really strong efforts and very smart people working on this problem of changing the way that people publish work to begin with. Which I think is super valuable, but extremely hard. It's really entrenched in incentives. So I'm a little bit nervous about going into that area. I'll leave it to people smarter than me and exploring complementary paths. Okay. So that's all. You know, we talked about what this process are. You see an intuition for why it might be important why you don't have them yet. So now I want to talk to you about how we could have. Right. This, what does this mean the scholar power contributions. Right. The basic idea is we give people tools to build personal discourse wraps for themselves, or for their labs. We give them the means to share and federate this discourse graph with others. And then over time, we can layer protocols on top of this to start to aggregate these into these centralized comments of discourse wraps. That's the roadmap. So there's two questions here. The first is we started patch on this, right? Is this even a thing that people are doing that we could integrate into, right? Are there integration points for often this first class? And second question is so, so thankful one, right? Is it actually possible to build tools that can help people build discourse graphs that are shareable. I'm going to focus mostly on the second one. Well, move a little bit quickly through the first one. Just give a proof of concept, right? So the first question is about integration points for offering discourse wraps. Right. We've done a fair amount of user research participant observation need finding to really understand there's actually a lot of opportunity here. We've put go pros and people's heads while they do reviews. We've done interviews with people with participant observation in communities of hackers and user tools of thought. And where we've been kind of looking for where our scholars already creating artifacts that have properties of compression contextualize maybe and our composability, right? The things that discourse wraps provide. What we find is that there are integration points in the range of gears tools all the way from people using really specialized tools to just people using every day tools like pen and beaver, which is really interesting. So on the lower end of spectrum in terms of like technical sophistication or I guess not technical. New fangirless. We have traditional tools that are used really well by what we call virtual right. So you have things like color coding your annotations when you're reading, right, they mean something right you're trying to distinguish you're just trying to highlight things but you're trying to distinguish between things like in blue will be like a claim. And in green will be like a piece of context, for example, you know, writing things in structured ways in Word documents and Google Docs, right? They'll talk about the templates they'll have like here's the overview here the arguments here's the evidence right here's the key takeaway. There's these structured things that people already do. There are also other tools that are slightly more specialized that are mainstream like econ explorers using tools like liquid text that allows you to pull out these excerpts and then create structured links between items so we have this compression pulling out these pieces and then Composability having these edges between them and then contextualize ability can jump back to annotations. I told you about repurposing qualitatively and also software as an example, right. And also adoption of like tools like new tools for thought like room research or obsidian of oxygen so on. We also have a lot of hackers who create homes fun system enhancement systems to enable these features. So the Orgmo Emacs community has created a open source part of room research this idea of backlinks into Emacs, for example. We have, you know, it looks like this you can sort of integrate annotation features also into essentially a terminal. This is a simple building on top of tools like so terror to enable you to enhance contextualize ability of your notes, right, you can extract annotations in a way that allows you to jump back to the source of the PDF. Just very briefly, right. We see across all these settings and scholars in the everyday practice without anybody telling them do it already create artifacts with key properties of compression contextualize building composable. Okay, it's not enough though, right, this problem of private public alignment personal notes are contextual the idiosyncratic, they're informal they lack structure. What we want is something that's general or shareable that has some level of standardization and reliable capture. How do you bridge this is it possible. Is it even possible to have some of this creative exhaust. Right. Is it so should technically possible to integrate authoring of shareable discourse graph. So at this point, I'm going to pause while I pull up the demo is a box of a discourse graph tool in the software room research. So you're not familiar with it. What you basically need to know is it's kind of like a document. It's a giant Google Drive folder, but you have the ability to create outlines, and you can create links between documents. And what's useful for us is that you see these like bullet structures and links are actually in the data structure underneath the documents they became parts. So, but you don't need to know that right so just looking at this right now, you can imagine in the context of a research project you might have a single document with a bunch of sources right that you're interested in. So for instance, if you're interested in the susceptibility of children to COVID infection. You might have a bunch of studies that you want to read right so this looks very similar to an annotated bibliography or list of papers to read. And each of these things here right is a is a paper. So if you look here, for example, and see, we've got metadata. This is a paper. The key thing that the discourse graph extension allows us to do is to integrate into people are already doing in terms of say taking structured notes on papers. You can see here this is a paper. It's a meta analysis of previous household studies. And you can see here looks pretty similar to how you might take these structured notes as we saw in our document right you have thinking about the aims of the paper methods what they do you can drop in screenshots of figures to keep the context. Right. You have definition index cases. And then we have like, you know, the results and contributions right what are the key takeaways. So for instance, you can say here one main result is if you met an out doing my analysis of these 11 studies then you get an estimate of lower susceptibility for children versus adults. Right. And specifically you can see the relative risk ratios here. This is be this not too different from what you would see. What we have here is the ability to mark this result as a discourse node of type evidence, just by the manner that looks like annotation. Right, so you can highlight this, hit a key. And you say, this is now a piece of evidence. And then it shows up in the sidebar. This tells us that this is not a document itself that we can reference and search for elsewhere. Okay. So that's the that's the UX right so you don't have to do it from the start we can write informally. And then as we think, okay, this is the key takeaway I want to remember this but later I'm marking. This is not a piece of evidence. If I want to. I'll often do this in my own work as needed. If I want to keep the details handy. I could for instance, just drag this and put this in here so it's just right there. Right. I could also copy over in the Rome sense. The key contextual details of the study. It's all in one place right this compact micro publication has everything that we need to make sense of it using instances. It's got the summary. It's got the methods that produced it. If I want to dig into that later, which we'll talk about in a second. So that's no creation annotation. Let's talk about how do we create edges between a discuss right for this work out have notes and edges. The edges are part of the, it's one of the more annoying things to do without formality. So let's go and see inside here what it might look like. So inside this document resources, we also have a space to write in our life. So what I want you to notice here is right here. Let me zoom in a little bit. Let's talk about this first claim that you're not equally susceptible. This is a reference to a claim. Right. I've marked this as a thing that I want to remember. This is a type claim. And then I've indented underneath your some results. Right. So yeah, this one is the evidence. This is an evidence. This is an evidence. Yeah. And I have written down here following is the case for this thing. This is a support relationship with this. Right. If we go inside here. Right. We can see. Right. As you saw before, it's got all this data in here. But additionally, we also have discourse context. Right. We have this relationship now that it supports this claim. Why? How do we know that? Because we said so. We said this here. And so we have this information now. And if we jump into this claim, the reverse is also true. Right. We know all the papers that support all the pieces of evidence that support this claim. Right. This is what we just saw. That is part of what makes it formal in the sense that, you know, you might specify this informally in an outline, but you can't query it. Right. We need the reverse. Right. We need to know that evidence supports a claim and also the claim is supported by evidence inquiry from either direction. Right. And it's horrible. So this essentially makes the relationship real. Right. So what the plugin is doing is it's recognizing these relationships and essentially making them real. Right. We now have this evidence supports claim. Evidence supports claim. Same thing here. We have another claim here. We have these supports and we have actually surprising amount of evidence. Which I'm still digesting. Okay. That's edge creations outlining. Right. So creating the edges between the discourse graph. No, it is not an extra task, but it's integrated into my desire to structure my thinking here. So as I outlined it, I created that structure. What is this binding? Right. So now that the nodes are real, the edges are real. Let me give you a couple of intrinsic benefits. Right. So beyond, I just wanted to do this anyway to structure my thinking. Right. So if we have the nodes and edges and we know what the type of relationships are, we can do some computations. For example, we can see the number of discos relations that claim participates in. Right. So here we can see it's supported by these guys. Right. You can get a feel for roughly how much we thought through a particular idea. Right. So if I reference this in an outline, this claim has four discos relations. This claim has 15 discos relations. I might want to, you know, look more into this one. For example, if there's, you know, not, not evidence. I don't want to kind of really weigh those. Same thing about my hypothesis here. I can keep track of the fact that it's the hypothesis that I've got one piece of evidence, maybe that's the supporting this. Actually, I've not. Right. There's no evidence behind this. So I might want to sort of look into this more. That I can do with structured query. Right. So if we know what the nodes are, we know what the edges are, we can also query over the nodes and we can maybe create attributes for them if we want to. Right. So for instance, if you wanted to look over the evidence base and say, okay, well, you know, maybe we have this evidence for lowest susceptibility because there's something going on with the testing the under testing the kids. Right. We can now more focus and focus me. We can, you know, more focus we go over the pieces of evidence to say pull up, you know, which of these results were from what kind of testing. Right. Are they all, you know, symptomatic testing or they some of them, you know, exhaustive filter for the ones that are exhaustive and see what the result is, for example. That's just a quick overview. Right. Of the prototype. Right. This is what we can do. We can, we can incrementally formalize integrate the work of creating nodes and edges into our normal note taking workflow and writing workflows. And we get some immediate benefits back. We can also export to this cross graph. So let me stop. Let me now sit back. There's a few things that make it alright so we talked about this idea of incremental formalization where this targets a key long standing bottleneck and sort of integrating structure into this early stage creative knowledge work like literature reviewing and thinking. Right. Now this is a pretty old problem. Since it's early as the 80s, people have identified this right. The work of this system of issue based information systems is pretty, pretty famous back in the 80s for structure thinking over design. And even there we have this observation that, you know, the early phase of consideration of writing is needs to proceed big country forms and if you force people to only think in terms of structured nodes and edges, for example, it sort of kills the thinking process. Right. And so they talked about providing tools in the structuring of raw materials where we don't have to organize them in the beginning. We sort of graduated that and that's kind of what we're going for here to enable people to write informally at first and unstructured ways and then add structure as it's useful. So this is accomplished with like we said, no creation is annotation. Right. We create nodes as we need them as we're ready for them. And we create edges not by typing out properties necessarily but by integrating them to our writing and aligning work where we have this immediately useful notes with implicit discourse structure that the plugin parses into a usable shareable explicit discourse valve, which I didn't show you but you can export actually a CSV as Jason as Mark down. We have this structure that's preserved. Right. And then we also provide media and choosing benefits we have structure querying we have top back in terms of the level of development of particular ideas and exploring your notes in a more systematic way. This is a screenshot of what I wanted to show you which is your export. This is in the open standards compliant with your 4j version of the nodes and edges. Okay, so technically speaking, all you need is three ingredients need a convention for those writing which people are actually pretty open to adopting and actually most people already do something like this. We need something like a hypertext notebook that allows understanding of links between things of which there are many. This is Roman research. We have obsidian, you have notion, you have tinderbox, we have Emax, we have Athens, loxie, chrome, remnote, lots and lots of different tools can implement this kind of protocol. And then a simple plugin that parses notes into a discourse valve. I want to show you this because this turned out to be pretty important in our field study. The way that the system knows how to translate the writing patterns into edges is through a grammar that is user customizable that essentially says when I write something like this, which is on the left. I want you to save it and recognize that this is a particular relationship, right? If the evidence is indented under a claim and has a word supported by there, then there's a relationship supports between them. On the right here is essentially a data lock query pattern that creates the mapping. And users can actually extend this and create patterns of their own, create nodes in their own. This turned out to be really useful. So some brief snapshots from the field study that will close and have a conversation. So right now we're about 100-ish or so alpha testers. We are sort of deploying in these different contexts in the discourse community of academic Rome users, there are no taking courses, experimental journal club. Just to give you some snapshots, right? So obviously I'm actively using this pretty actively in my lab finding very helpful for my own thinking, my students are finding helpful. It's useful for our discussions, right? And this kind of gives you a flavor also be, you know, this is an outline for a paper that I'm writing that integrates these claims and evidence that helps you structure my thinking. And the right here you can see the context being easily available. There's another library information science research team is doing a systematic review that's using this to structure the thinking over evidence to understand impacts of, you know, a culture of stress on mental health or physical health, for example. We've done an experimental journal club working through a bunch of papers and structuring out the claims and questions and evidence that are in that knowledge base. A microbiology lab is using it to kind of structure a project in the review. I'm not going to go through the details here because I don't understand all of them, but essentially this is kind of cell biology for a particular team. And what we see here is that, you know, structuring a project in terms of questions, and also enabling you to then tie those questions to what are we thinking was the claims that we have and play what evidence we have so far. So what we can see at the bottom, we have what looks like results and hypotheses that are also added to the project. Right. And so this this user actually extended the discourse graph grammar to also enable thinking through conclusions and results that are primary, primarily produced by the lab as opposed to from the literature, and they can all be part of the same project so that whenever new new member joins the project they also have a full context of past results but also understand the chain of thinking. An international lawyer use the discourse graph to write two books is my understanding that really think carefully through a bunch of evidence. It was really helpful for structuring thinking. And he also extended the grammar to say distinguishing claims and conclusions, for example, and adding relationships like substantiates for him was useful to distinguish between those that flavor of support. And so we have a couple from is using this for road napping that are trying to tie discourse graph evidence to a discourse graph of outcomes of constraints and solutions to understand opportunity areas. And so here again you have to extend the grammar, and we can. Right, you have constraints relationships as opposed to support or oppose for example. Because it's out in the wild and open source, people are starting to part this of the other tools. So tinderbox is another tool that's quite one of the original hypertext books and you have a user forum post describing an implementation of discourse graph protocol into the box. Nothing to do with me. It's like sort of by chance. You know, you can actually tinderbox has pretty powerful affordances for exploring a discourse graph that relax and implementing this and that is pretty interesting. We have a grassroots funded bounty to part this protocol to lock seek open source hypertext notebook. And we have efforts to part this discourse graph to HTML annotation standard for broader web published news. So, excited about what's been going on. I want to talk about some really key initial insights from this field study right. The first is that I was mostly surprised that we didn't have to do any thing that's super flashy to convince people to use it. Fostering more careful and creative thinking patterns was most of the time enough for people to adopt this. They want to think in this way, and the tool helps them to think in this way and that's that's good. And using it to find an access to more ideas like later on they can use the more advanced features but just simply having the discourse graph as a way to structure. Right that model as a way to structure thinking what's really useful. Also extending a personalizing grammar is crucial right many extensions were finer distinctions right we have different flavors of claims, or different flavors of evidence. But you can also be collapsed together if you wanted to sort of translate to a different graph. And this connects to this kind of concept of boundary objects from information science and CCW that enables coordination between different social worlds. We have this like weak structure and common use very minimal ontology of questions came to evidence, and you can extend it in local use, but enables you to sort of translate between those. This is I think a promising design pattern. Okay, so summarize in versus question to I think we have a cool concept it's possible to write close to pros and create shareable discourse graph as a high product. So I'm going to, you know, now talk about some conclusions and we'll address some more questions right so where we want to go next is discuss graphs from or to everywhere. Right, I think we've established sort of the utility of the model. We've established that there are integration points. And I'm excited about the idea of this being a synthesis protocol. There are tools as clients on an open protocol, integrating this close graph extensions in the mural Twitter slack hypothesis and so on. Right. Why would the spectrum like, how do we enable portability transferability property in particular to avoid the one standard problem. We don't necessarily force everybody to use a single standard. And so this idea of translating between user extendable grammars, but also having a similar underlying idea seems like one promising way to enable this peer to peer. People will be concerned about formality machine readability. Right now we can see there's minimal formality in terms of like we don't have any ontologies in here, right there's no like connections to wiki data or all or anything like that. But it could be in principle if you wanted to, you could integrate that into the document itself, right you can sort of have an all representation of the particular relationships that are stacked out in a piece of evidence. For example, there's no technical reason you can't do that. So I'm excited about the sort of middle layer between the sort of more structured knowledge graph that's more granular and the more course documents this kind of middle layer of the discourse graph to enable sort of communication across the systems. I don't have much to say about this, maybe an LTO useful for stitching things together, because at some point we need some way to understand which pieces of evidence are the same or similar, which pieces of claims are similar or not. So ontologies probably won't be enough and so that's something down the line. I think it's doesn't seem I'm not working on it yet but that's like something something for the future. Okay, so revisiting a large vision you have this building block for a new infrastructure young items for papers. And yeah, we start by just facilitating collaboration system and you can scale up by prioritizing decentralization and federation. And then you can publish to say databases like ceramic or the graph or so on, and you can subscribe to graph queries you can start there as opposed to every publishing to a single streaming you can start to have people talk to each other. I like this metaphor, the way that we're working as opposed to starting from the top and saying everybody's going to do this, we start from the bottom and build tools for people to adopt and then you grow the infrastructure. This is the kind of general strategy that we're excited about. So close to the call to action for my friends, the tool builders HCI people. There's a lot of space for two building and science reform, it's not just about the institutions or incentives of funding. It's also about what we do in our day to day, the tools that we use. And so I'm hopeful that more of us work on this problem. And, you know, not just make it require, but also make it easy to be able to bridge from the bottom to the top. Okay, so we just light up and we'll have a chat. You know, so this is hard, our infrastructure pillars are wrong in your analysis, discuss grass can help, but we do lack sustainable means of offering. But I think we've got some promising directions with tools for a scholar power contributions. So, thanks for having me excited to chat. Thanks to all for as always, very interesting and thought provoking discussion. Join us again at our next research seminar I believe it is nine August, take a little bit of a break for the summer.