 A number of you have have experience with hypothesis already in the room. I tapped a few folks to share their stories explicitly as part of this round table. But if you have your own stories to tell the way this is going to work is that each person is going to take five minutes up here. Tell some stories and takeaways, some challenges, some hopes for the future in terms of annotation and teaching and learning all under five minutes after each person will talk as a group for five minutes you can ask them questions. But that's also a chance for you to share your own experiences with annotation. And then after, you know, five people go for, you know, those 10 minute chunks will have five 10 minutes at the end for broad discussion. So if you have your own stories to tell, please do, you know, do so at those at those opportunities but I think if folks are ready. We'll just jump right in and start with Ann. Hi, I'm Ann Oberman and I'm from Metropolitan State University of Denver, and I'm using hypothesis currently in a course and so I teach social work. And we're using it for an online groups and family course and so I'm teaching people how to do therapeutic in case management groups, and also do family therapy. And so we implemented hypothesis really it was part of a couple other folks that were doing it at Metro and I am the online coordinator in our program and so we wanted to see how it worked and to see how it engaged students in some really difficult texts that we were reading. So I said, sure, I'll try it. And I am so glad I did because comments that students have been giving me are it's like having sitting down and having coffee with you and that was one of the first comments that I got during a week. And usually I hear, you have to do another discussion board like this is such busy work. And the insight that even within the first annotation or within the first article that they were reading the the insight and the dialogue that they were giving each other was already rich. And I was thinking I'm going to have to model this for a while I'm going to have to share, walk them through it. Instantly it was almost innate the way they were interacting with the text and so I just like to open with that because it just felt really amazing with the, we're having coffee. And speaking or listening to Manuel and thinking about some of the more social justice pieces I really think that using hypothesis hypothesis has decentered me as the instructor, which is hard to do when you're online teaching because you're delivering, or it's, it's very centered with the instructor. And so oftentimes within the reading I will provide prompts and I'll say hey pay attention to this or do that, but most of the time I just leave it for the students. And what they come up with and the way they take readings and turn them, whether it's applying to their social work practicums or what they're doing in real real life or their family. And so they take those readings and put it in a complete different direction. And so I've really just appreciated the decentering of myself, the creation of community that's often a complaint of our social work faculty is I'm not connected with my students I don't know how to connect. And the intimacy that's created with the annotations is instant. And you see in the annotations people asking each other Oh this reminds me of your study your kid or how are they feeling I mean in the annotations. They're having these dialogues, which some of my faculty some of my other faculty were like yeah that's not the point of the reading I said to the exact point of the reading is that there when they're reading they're thinking of each other in their own lives and integrating that in. So those were just a couple of those takeaways I'm trying to think of anything else that I really wanted to share. So a lot of our bachelor students that are coming in junior year, they don't know how to read research articles. And so just really dissecting and being able to say okay. What is this and you have a voice in this and students are actually disagreeing with the reading which they don't do in class as much or they don't do in the readings they're reading outside when they come into class or disagree with me. They don't do that but in these annotations they're disagreeing or they'll say actually and I don't think that. Which is, again, different and maybe it's the way I'm presenting to my class but that I feel like hypothesis has allowed them to disagree to decenter me to center themselves. Some of the things I'm thinking about with future use is really within social work we work a lot with the community so other social practitioners could portraying as professionals. And so opening those annotations to their supervisors in their field placements to other professionals and then to other participants or clients they're working with. And so how cool to read an article about building resiliency and adult chronic mentally ill males and then to be able to have men that they're working with annotate an article right along with them. And so I was thinking how important that would be and even with mental thinking of participatory research so taking that D2L and then doing some amazing qualitative and that layering of annotation that you could do within participatory research and kind of blowing up in that access to that process after they figure out their tags and after they do that. So that's some of the directions that we see social work going in. And one of the last ones that I love is students say it doesn't feel like busy work. And I again I know I mean and Jeremy said this in the intro but you know where students are at and I know what they're thinking. They're like this isn't busy work and they are writing I don't grade my annotations as part of discussion group and I'm trying to change the way I grade and so they simply get graded for being in there doesn't matter when they do it during the meet it doesn't matter how much they do it. And I'd say that 50% of the class is I mean like I'll get with a class of 22 I'll get you know 200-300 annotations in one reading. And so they don't even it's not even the incentive of the grade that they're just they're participating and they're in there. So those are a couple of my adults to to the usage of it and just really excited to continue using it. Another colleague uses it in her in person classes and loves it because you're coming to class prepared and they've already had this dialogue to the intimacy level and their preparation levels. Thanks. Five minutes to talk. If you've done something similar to me or you have questions about. And project and aspects of it. This is the time people on the panel can participate as well as just open discussion will have these little open race in between. I'm just curious. These people are very interested in research. So is there research and is anybody doing that? Yeah, and I'll just add to that. I actually got an email this morning. I'm going to be doing a presentation in the classroom at Southwestern College. Texas has sent me the pre-print. So starting to come together. And I think actually I have the inspiration this morning that the next hypothesis webinar is going to be something about research. It brings together some books and working on that and sharing results. Yeah. And is this completely replaced discussions for you. Not in this. So this is the first time I've tried it, but I think it will. So the discussion, I feel like the discussions are having and because I can post annotations and ask questions throughout that they can choose to respond to. So if I wanted to guide that or had specific topics, I could do that also in the annotations. At this point, I was using annotation and Flipgrid. So that was because they're doing modeling skills. And so we had the videos, but those were, I was doing annotation and Flipgrid as my only discussion activities. So I mean, I also don't read my students' annotation contributions and their participation in the college studies. Maybe that'll also get me two questions. I have a tie over here. And I'm wondering if you tell your students that you're not creating that kind of content. One, how do you respond? And two, what strategies do you have that encourage more kind of parent-motivated, variety of term-authentic conversation when they know that there just isn't a percent, which is kind of just supposed to show up. So not flowing with grade. Yeah. So I feel like whenever I start going into that territory of not grading, there's a lot of discomfort and the students respond, what do you mean? I want to know how many posts, by what time, how long do they need to be? Do I need to use APA formatting? And so that initial response is always a little bit of that resistance of fear, I think. And then when I get in there and I'm posting pictures of my cat making, I mean, like all kind of model informality, then they start to engage. I would say it takes the first couple of weeks because students would post at 11.59 on Saturday night. And so no one gets to respond to them. And so then commenting later during the next week, oh, you know what, Jeremy made this great comment at the last minute. And I am a little cheeky about, you know, and people didn't get to see it. And so I wonder what that would look like in this context. So I'll bring things back to try to create that culture of responding. But I find that the, not majority, about half of those students have that self-motivation and are engaged. And the others, when I respond to comments and I say, oh, tell me more about that. Or can you respond to that next week? They do. And so I feel like the relational piece draws them out to create a little bit more of that self-learning. I also try to choose really meaty texts that are controversial. So I feel like that helps with that engagement piece. But I've been, I mean, faculty ask me all the time about the grading piece. And how, no, you get equal points with this Jeremy's comedy 11.59 or Remy's 25 annotations. And it balances out and students really like the flexibility. And when they say, yeah, my kid was sick all week. I didn't get to post this week. And so it made me feel great that I didn't have to do that. You're right. I'm so fazed by this, like webcam. I'm so glad to trim my nose hairs this morning. Hi, I'm Jonathan Lashley. I'm from Boise State University. I found out today through Lego that I also want to fight people, but also communicate equally effectively. So thank you, Janet, for that. But part of it is because I have multiple roles in my institution. I'm the senior instructional technologist for the university. So I think about instructional technology and its use a lot. I recently completed a PhD in learning sciences. So I think about the science of learning a lot. And I teach first year writing part-time. And so I think about writing and how we communicate effectively and exchange values with others on a regular basis. And so I have a few different stories to tell in my brief five minutes. The first of which is last year Boise State's first year writing program was undergoing a curriculum redesign. And as I'm prone to do, if you're going to make changes, you might as well make all the changes. And so in this particular case, the class I was teaching in the fall was based on genre. And specifically the major like capstone project was evaluating, writing a value of the report of some online genre of media. In which case hypothesis was one of those tools that I love because it just got out of the way. And that it was already simulating exactly what I want students to do. And what I've tried to do as a first year writing instructor for years, things like co-authoring my syllabus with them, things like having good structured peer review, having a place to keep and attract and select resources and file them away and take notes and annotate all the things we've heard a lot about today. And what I found was a number of things. And just to kind of brush up before today's presentation, since I've been working on dissertations to have teaching this semester, I was going through my course evaluations and I forgot how much I appreciated them. And then thinking about the lens of hypothesis, I appreciated even more today because I was going through and learning about the most valuable things the students took away from my class. And there were things like class discussions and practicing writing and interrogating writing and reading with other people. And all of this was stuff I did with hypothesis or was augmented through hypothesis. As you were talking about students would come to class and discussion who had been taking place on this platform with me with their colleagues. I started pulling further and further away as weeks went on because they were structuring conversations themselves. And also the space was designed around active learning. We've had little small group tables where they built bonds. I shake them up throughout the semester, but they didn't matter. It completely changed the way discussion happens in a very discussion heavy class like first year writing. And they were building knowledge. They were co-constructing knowledge, which is exactly what I've wanted to see. It also really problematized the expectations I've had in the past about what is working and what's not within class discussion, what participation looks like in a class, but also even better. And I guess this kind of goes to answer some of Randy's question as well about grading. I'm a contract grader. I do contract grading in class. I think the students need to come to me throughout the semester and especially at the end to validate their participation, what that looked like and justify the grade that they earned in my class based off the rubrics that we co-construct. And they had all of this knowledge stuck in one place, whether they posted 100 times, whether they posted five times and they were just five really thoughtful cases. They could go back to these materials and reference them and contextualize them and build a narrative. These are all the things we want to teach them in first year writing in terms of developing critical thinking skills. And I mean, even just the idea of going through some of these readers or articles and allowing students to ask questions that they would maybe feel uncomfortable or embarrassed to ask in class about what does this word mean or why would they write this way or why would they sort of sentence this way. It just changed the nature of the scope of the assignment and actually I like those kinds of questions so much in the last couple weeks in class because we just changed curriculum because my course is entirely based in OER. I actually had a small list of press books that I asked students to go through and said, listen, you all have read the concepts, you know what's worked for you this semester what hasn't in terms of learning the outcomes of this class. I want you to go through with my process and help me vet these materials. And so they were able to just pull up a press book, find a chapter that seemed interesting to them on a concept that they want to learn more about and talk to whether or not they appreciate it, whether they thought it was helpful, whether they think that I should use in the future of the classes and so the role of was modified because I was able to just continue to do what I already always do, but I wasn't relegated behind the wall garden of say just using Google Apps products or some other product mechanism. It was all just web based. So that's really cool. A couple other just short stories I have. One is as a co-author and co-editor of an open source book that came out, open access book that came out last year. Right after it was published by Pacific University Press, my co-authors and I got an email from one of our authors and she was concerned because some edits that she had put in a few months prior they weren't realized and so we contacted the press and everyone was really concerned because here it has been published and they're like oh we're on it, we're on it, we're getting on this and immediately I thought surely this is going to happen again. I can think of one other author in our book who just got married and so she changed her last name and there's an open access book. This is CC by, this should be edited and so a project I'm really excited about now that the dissertation is behind me is that I can facilitate what I've deemed the editor's cut that's currently hosted in Boise State Press books of OER field guide for academic librarians where hypothesis is already enabled in that platform. People can go in and comment and we can adjust that manuscript to be more relevant, to be more accurate, to provide better resources and access to our authors that we spent the last few years collecting. And so, you know, I'm a big OER nerd, open education nerd and so a tool like this is really helpful to see the sort of practicality and the transformation of those materials to be more relevant in our realm which is great. Last story is, I can't remember so it's fine. That's it. Oh, that's what it was. Yeah, I just wrote dissertation, small, you know, casual writing exercise. And it's great because back in the fall, if I was to try and remember or find the paper notes I had or even find the digital notes on my laptop when it's impossible. And so hypothesis, I've been using it throughout my revision process, my research process, and allows me to remember who I was back then. That's okay. Thanks guys. Can we talk a little more about contract reading and what that means and how the hypothesis plays. Also, just maybe give us a definition because it's a pretty radical approach to grading. Yeah, I find grading problematic, especially in general education courses. I see it in myriad ways, but my students are slaves for learning. They've become in that way. What I've seen materialized even more of the longer I've been teaching, some of the teachers put that decade down, is how often students want to strip the curriculum where I'm giving them a list of things to do and they go through that list and check out boxes. And I feel like a lot of the students are checking out the boxes to get them to be able to grade with that free sign. And so, at least in first year writing classes, folks like Peter Elmo and others who use contract reading, they see it as this is reflected in the past skills you want them to do as well. So, to think about what is the scope of the class to try and understand that upfront, to keep modifying the understanding that's worth writing and negotiate that, is ultimately that's far more reflective of the experience they're going to have in life than reaching something to a greater degree of reassess and expecting that all these students are going to have these types of skills. There's one thing I learned about discussions that are automated through that process is that learners need to be noticed, to learn this. Like, we're writing and research for years that learners have to make sure that they're going to engage with different contexts in their lives. And so, that's all contract reading is. I recommend you read it up. But in my class, I think it does, where students have an initial contract and then after every major assignment, we're able to go through and revise it. We do it some course, some class time and some final semester to reflecting and filing artifacts and evidence. And so, again, it's cool like hypothesis is really helpful. But we're going through and they're reflectively writing and reading and commenting. So, around that, but also for others kind of, so they have that record. I think that's why you mentioned Jonathan, that you noticed something different about, or you had to reassess your thinking about how discussion is working with that by using the presentation. I was curious, what is it? Yeah, so I haven't replaced my class discussions. Part of that is because I've found that the assessment I did throughout the semester is going to be like, I thought first you're writing a number of class records up the same class twice. So, I'm going through that in a couple. And it's true for my and I's in the last semester, folks were liking the discussion. I think they were liking most of the discussions in the last semester, so they came in more confident. And those who wanted to better explain themselves and writing into less than half of the students would do so. And those who felt more confident being kind of shrouded behind before class or even after class, because that was a cool thing to see. That's how we were done reading. People are reading weeks ago. And people are still confident. They're still having after-conversation. And because I have an OERB class, students have access to each other's breath. And so the only way to have access to their comments in that region is because they're also curious themselves. They keep going back and referencing. If they found that they really had a nice structured, interesting conversation, they could go back. And that's what we want in education and research. So those are the sorts of things I saw. And it just really called into question how much weight you want to get from whatever that looks like. Hi. I'm John Stewart. My role is more from the, I guess, administrative perspective. I'm the assistant director for the Office of Digital Learning. And a lot of what I do is try to help faculty understand new tools and integrate them into their classes and help them think through what can I do with the tool. And so with hypothesis, I spend a lot of time helping people think about, you know, don't just, you know, have the students highlight the thesis and make that the assignment. I actually get them starting conversations in the documents that they're annotating. And so a lot of it has been, how do I meet with faculty? How do I work with faculty? How do I let them know about something that exists like hypothesis? And so recruitment of faculty and integration of these technologies into their classes is a big challenge for me. We do a lot of workshops and bring people in. And the model that has been more successful is sort of a champion model for us. And so one of the early adopters of hypothesis on our campus started three or four years ago and actually did a presentation with Jeremy at a tech conference at OU. Nick Lourdo integrated hypothesis into his creative writing classes into his expository writing classes. And so he was having his students both annotate the readings that they were working on and then also annotate each other's papers as they were turning them in for peer feedback. And then from Nick's work, the rest of expository writing is now using hypothesis in their classes. And so we've got an adopter that sort of departmental role. And then one of our other early adopters was Dave Robell, who's a history professor, and he teaches Steinbeck a lot. His real passion, his research is on Steinbeck and his novels. And so he had his students again marking up these open editions of Steinbeck using hypothesis and then writing their papers based off their notes. And then luckily, I guess for us and hypothesis adoption on campus and luckily for us because Dave's great. So he became dean of the College of Arts and Sciences right after he adopted hypothesis and so then helped to sort of promote it to the rest of the faculty through his own work. And so we see a lot of adoption in history and English and in philosophy, poly science. And so a lot of my work is just how can faculty use these tools and I'm constantly interested in talking to faculty to see what they're excited about what their ideas are for using the tool things that I haven't thought of yet. Applications in STEM fields and engineering that I just don't come across as much with my history background and history interests. One, I guess project though to focus on it's been particularly interesting and successful, and especially the idea of it in the way that the faculty members integrated is a guy named Rafi Folsom, who again teaches history, has his students take annotations, but then dumps all of the annotations from all of their readings into a common pile he uses the group feature, and we extract all of the annotations and then share them and make sure that the students have easy access to all of the annotations. And then he encourages the students to write their research papers, drawing on not only their own research but the research of everybody else in the class. And some of the findings that these undergrad, you know, usually junior level students are making are outstanding because they're actually working with a research team of 30 people, all focused on a set of topics that are a little bit varied and their individual papers that they'll write are a little bit different. The student was focused on the classes a history of Spanish borderlands, so sort of colonialism, South America, all the Americas really. So one of the students was interested in gender dynamics in these Spanish borderlands and she was looking at how family structures are made and really how marriage, the marriage process works in these readings that they were doing. And so she found several of her own readings that were interested in several of her own notes, but when she started noticing a pattern, she then could go through all of the notes by all the other students and look at where they talked about gender, maybe they weren't talking about marriage in particular, but she could pull from their notes on what they were seeing, and then go back into the sources that she had in herself read students had all been reading different documents. And so she could go back into the primary sources that they were reading from, and then do her research and she ended up finding that women actually were the dominant force in marriage politics in these communities that the women chose their partners, which was a really interesting finding that hadn't been made anywhere in the history and literature for that field of history. And so it was a novel finding that would be, you know, sort of great work for a master's or a PhD in history, but she was doing it as just her junior paper, and it was because she was working with a research team of 30 other students and a research leader as the instructor. And so I really like that idea of not only using hypothesis to model how to read, but how to do research and then open up the idea of how we do research and stop acting as single actors in history and in poly science and in these other fields where traditionally the humanists have been, you know, a solo author on all of their papers. And so we're trying to model this new idea of open research, open scholarship using hypothesis. And I've been really excited about that project and we're trying to model what would a history lab look like in the same way that a bio lab looks like, you know, what would it look like what we had multiple postdocs and multiple, you know, graduate students and whole classes of undergrads all thinking about the same concepts and reading, you know, hundreds and thousands of documents in a way that no individual could. And again, annotation makes that possible. So a couple of our projects are particularly exciting. I think hypothesis is currently being used by maybe 10 classes that I know of, and then however many classes can figure it out on their own without asking me about it. And so we've got a pretty good user group on campus. And I'm constantly trying to use hypothesis to push OER to push open pedagogy to push open research and just sort of open this in general. So anyway, a couple of ideas. I'm happy to talk about any of those. Thanks. John, quick, quick sort of hypothetical thing you can, or anyone can react to me, but the interoperability is huge and it's the sort of underpinning I think the value in what hypothesis represents. If you have like endless time there you or others who have interesting data, you can just devote like all your time to making use of it. You have the examples of how that, you know, being able to dig into the actual data and how that creates these model experiences on the student side. If you have like all the time in the world, if you just do that, what do you see or what sort of like systemic changes can really be enacted with enough annotation happening again across the university. Any thoughts on how far you could go if you just had endless time to dig in and, you know, sort of create these supplementary tools. How far can the university go in really shifting broadly, culturally, like big, big things, how far do you think you can go? Yeah, it's a really good question. One of the ways I was thinking about it, as I was sitting here earlier, was not so much interoperability at scale but interoperability between the systems. And so I love that hypothesis has its API and it's very easy to deploy annotations from the hypothesis. And that's really nice in terms of interoperability because for as great as hypothesis is and as much as we can take everything, we can actually annotate some stuff that isn't, you know, digitized here. And so some of the other annotation systems that we have focus on archival materials. And so I've got a database where students are taking notes from the archive and putting them into a database and then again sharing those annotations with each other. Based off of primary things that you can't use hypothesis on. But because you can use hypothesis and because you have the API, you can then mash those two systems together and you can have annotation tools that work on video, I guess, for the next one. So interoperability with visual, non-textual media. So interoperability between systems is key. And then, yeah, scale, having multiple students working on projects. But you can imagine a sort of diachronic research project when you go through multiple classes. Maybe it's a smooth progress on multiple levels. You know, you get a freshman intro to colonial American history and a senior level capstone, both working on projects at the same time. Many different groups, hospitals right out of the multiple semesters, all sharing notes. And there's a really good example of that in history of science where a group of students in a class over the course of like six classes, those six classes of students all worked together on the history of chemistry. And so you can think about diachronic projects where students are sharing notes of course of two degrees and multiple classes and types of research. Or between classes where you look at history of polycystery. One of the cool things that came to mind as you were talking about sort of increasing institution level adoption when we see a department level adoption is, you know, what would it look like if history of the University of Oklahoma was, you know, almost every course was using it, or even more than composition courses as you mentioned. And then since our office of majors or history majors, following that learning record, you know, sort of portfolio of interactions that Jonathan brought to mind, but across courses, throughout ones, you know, tenure in a disciplinary training at a particular department would be really neat. I mean, just one way that I think that might happen for us, and again, just interesting hypothesis is that was our champion in the history department, both the dean, but then also right now a graduate student really wants all of the graduate students to implement hypothesis in the classes, whether the professors like it or not. This man? What's that? This man? Yeah. Because she likes it so much as a graduate student, as a group, as a GA, it's really making that change. And then you can see sort of how it works and how it's going at that time. So I have a question that's a little bit off topic, but because you're coming from the administrative side, I'm wondering what are the pros and cons in terms of like accessibility. Also, I was thinking just about, if you're really doing it, the more robust the conversation, that's a good thing, but it also means a lot of text to sit through. So I'm thinking about students that have cognitive disabilities, that have brain injuries, those kind of things. So can anybody speak to how, you know, this, ways they've seen that it's worked really well for those students? I don't know if that question is general to John Wayne. Yeah, I don't agree with John Wayne. So mine has just been based on experience. And actually, maybe I will have a conversation about this back in fall. One thing I appreciate is that I've had vendors a lot as part of my job. And I don't know what we're all at now with the progress of like we've had something else at the time. I was reviewing hypothesis and what I appreciate is how, except the ones that had a page that had this whole rationale about their philosophy on accessibility and being that compliance level at night. I have not seen that in the head tech realm. I really appreciate that more. But to, regardless, a tool like hypothesis, when my students are going through, but I have, I have taught students about brain injuries and disability. It's a writing intensive class. And so to presume that everyone can write the same amount at the same speed and the same pace is the title of the matter. And so furthermore, as I was kind of describing, the contract rating and the fact that students who compile and reference and reflect on their own contributions for class. That's not about quality. Volume, there is something that should be measured there. So the fact that you have a tool where maybe it's just jumping information is frequent. They're having a more engaged experience or maybe an experience of better accommodates their own needs. And then if I was to say, okay, I expect you all to read the same page article and what things to say, you should have like three questions and three comments. And so that level of flexibility, it's to me, as someone who works with faculty and talks with them often about pedagogy and thinking about accommodating their students. That's the conversation point that helps illustrate the need to be adapted. We need to think about not truly individualized extraction, but how can you see that that's important. So from a social responsibility standpoint, that's great. But I also know that we've been funds that are bulletproof. I feel like it really inspires me because I'm saying, I want to be following this vision for a time. And of course, I'll shoot the part of the practitioner in the DS-106 space. And that's a separate kind of story. But having taught and played around with that course for a number of years on myself, it's taught me that the idea of a robust discussion is not necessarily text-type. I would just kind of also kind of just feed that back in terms of the question, because I also share many of the queries and thoughts around what counts as a robust discussion. But as I've played around with hypothesis in my own teaching, I try and have my students think critically about what makes a discussion robust. Does it need to just be a lot of text? In addition to, I think, John, thank you so much for coming out. The hypothesis we've done collectively around aspects of accessibility. I've had students, for example, who have recorded responses and then put an audio file on song file and then just put it away into an initiation. I'm not saying that that's the best workaround in the world, but I am suggesting that there are ways that we might think about meeting students where they are trying to do the modality that they perhaps prefer. I'm actually not making any other choice. I'm just saying that it's not like that. But I am suggesting that I think that can say that there are visual ways, perhaps, in audio that we have to approach in place on that, if we do have a hypothesis discussion on time. And then what does that mean for a robust conversation? One of the things that I'm bumping into is the research thing, which I'm already looking at. Another thing with accessibility that I'm really interested in in the future in my own world is how I consume information and how my daughters consume information, and it's gone from a 17-inch monitor pretty much every day, all the time. I've got iPad, I've got my Air, I've got all kinds of devices, but my tool of choice is becoming slower and more and more. I don't know where we're going to be in 10 years, but I'm pretty sure it's going to be less than that, but bigger than that. And I've already, or I wonder, how hypothesis and other tools that lay on top of the fact will probably be doing in the future on the phone. I'm going to let that talk stand in the air. That is Remy and Francisco's talk to the stand. I will say that I am really unhappy now, because my computer is being used as the podium computer and I'm having to tweet through my phone, which is, I don't have the biggest phone possible, and I'm still stressed out about it. I want my keyboard. So, long live the keyboard. We're going to need to stand a little more. Come here, Matt, come here. So, that's right, because we're looking right here. So, very briefly again, I'm Remy Kaler, Francisco Perez, and we cheated and we actually put slides in the deck. So, I'm going to quickly show this, give one little bit of background, and then turn it over to Francisco, which is that, as I've mentioned now, I've been teaching with hypothesis of years, I've actually also been helping to co-facilitate a public series of conversations about educational equity. It's called marginal syllabus. Perhaps it's a separate conversation, but in these various contexts of annotation conversation, I would always have questions like, who's in the document and what are they adding and is it an annotation or is it a reply and what's the collaborative thread and kind of what's going on here. And that over a series of conversations and Francisco's expertise led to a tool that we call crowd layers. That's it, man, it's all you. So, this project started when we were looking at analyzing the data and we started thinking of ways to productionize the analytic process. So, getting the analysis off of my laptop and sharing that process with others. And I think one of the best ways to explore a data set is through visualizations. And so we've developed this tool that visualizes interactions at the document level. Right now we're working to expand it to the group level and then later on to other different contexts. And so if you want to explore interactions at the document level through annotations, we built this tool so you can get a cursory view of these annotations. So we provide you visualizations for that. Our next step I think is pretty exciting. We're going to start using more advanced analytics like machine learning to dive deeper into the data that's already there. And then we'll also start using artificial intelligence to augment, you know, through your understanding of some of the processes that are going on, whether it be through natural language processing or modeling of the conversations that are happening at this space. That's our next step. And I'm kind of excited about that. And this is an open tool. So if anybody wants to use it, you can just visit the site. So there's the website. There's crowdlayers.org. Again, it's a long acronym that you can read about when you go to the website. But this is our second slide. And again, we cheated the screenshot. Again, you could go there live right now. This is a URL that's been put into the dashboard. Again, it's an open dashboard. If you recognize the Atlantic, and then there's a magazine thing, backslash, and you might see in that URL there, as we may think. And for many people who are familiar with collaborative annotation, there's a very famous essay from the 1940s, I believe. Is that right, Jeremy? 40s, yeah. By Vannevar Bush about a kind of memory extension device that he calls the MemEx. It's often credited as being one of the antecedents to not only the web, but how we think about collaborative knowledge construction. And so this is a really seminal piece in the history of the internet and also annotation. And there happened to be 308 annotations on that particular document by 65 people. A lot of those 300 are actually by Jeremy, but that's a whole other story. You have 45 collaborative threads. There have been 44 days of active activity over many years. And if you were to take this URL right now and paste it into crowdlayers, it gives you those real-time analytics. And as Francisco mentioned, again, this is a kind of basic descriptive approach to the statistics. Again, a kind of social learning analytics. Our next steps are, again, to go deeper. So again, you've got this in your slides. Again, there's the URL, but we'd love to just answer some questions around what kinds of data-driven insights might be useful for you, whether you're an administrator working with many faculty across multiple course contexts or teaching single classes or teaching large classes or small classes or whatever they may be. And so perhaps that can guide some of our Q&A. I just want to make a quick point about verbal sticking up to sort of point out where we've come from in this presentation. The thing I love about what you're seeing is the intimacy of the conversation and sort of the presence of the community of the classroom. And you being there with students going back and forth, learning about their lives in the context of learning is sort of the sort of close reading, not in the classic English literature sense, but everybody's on a text together and very powerfully gives me a point. And that's one of the great things about hypothesis. But I think the work that Francisco and Raimi are doing, I would call it a sort of kind of distant reading of those conversations that has other insights. You learn about things from your students by literally replying to their annotations and reading their annotations, but stepping back and seeing what they've done in the course of the text also gives you insights into how they're learning and also how they're interacting with the text that you've chosen and their teaching and things like that. So I think this distant view is equally important and it's really an emergent one because this is pioneering work that they're doing. I thought this doesn't have its own tool set. It allows people to have this kind of perspective. These guys parked up with me and learning from it already. Thanks. Sorry, I didn't mean to... Yeah, thanks. What would you want from the data verification? Well, yeah. I just wanted to give a shout out to the hypothesis team and this is real credit to their commitment. I think John mentioned that they have this API where you can gather the annotations. That's what this is based on. The hypothesis team has really been... they've really been committed to collaboration. That's the only way we could have built this is if they did all the groundwork for this. Thank you. Absolutely. This is something that we should all revisit and then follow up with Ramian in Francisco but I will take that for you. That's a lot. Get us through that. Candlest whose name was belatedly added to the list here but now is officially there with... His university spoke correctly. It's been a great night, refreshing a few times but now Michael McGarry will be our last person to share one. Thank you. I'm Michael McGarry. I'm from Cal State Channel Islands. I come from more of the administrative side as well. I'm the instructional technology lead for our campus. So basically I oversee our LMS and all of our other learning technologies on campus to make sure that all the bolts and gears are still firing and when faculty have stuff that stops breaking and we have a lot of work to do. So we just actually actively are running a pilot at Channel Islands with Hypothesis. We had kind of sort of tried to integrate it into some curriculum stuff and into our LMS previously and there were some clunky bits that didn't work so well. So it wasn't met with a great reception but we also didn't really push for it all that hard with some of the newer kind of functionality and stuff that Jeremy and all the great people at Hypothesis have done with that integration, namely the main one that was kind of the savior for us to use is the fact that it doesn't require students to create another account anymore. That was the big hang up for a lot of our instructors. There's enough accounts, enough tools and enough everything already out there that that was enough of a speed bump to have them just now want to use it. So now that that's gone it can just load up directly as an assignment, the interface loads up clean, it's course aware, it's user aware. We were able to kind of get a group of faculty together and we're fortunate at Channel Islands that our team, I'm part of the teaching learning innovations team which there's gonna be a whole gaggle of us here so come find us throughout the event. We have a lot of faculty, what we call fellows that we bring on with our team because we don't want it just to be a group of designers and administrators, which I mean, the team that I have, they work, they do amazing work but without the insight from the faculty it's kind of, it could get misguided very easily. So we work really closely with a core group of faculty and through them we were able to get a solid group of faculty together for this pilot. So we have 15 or 16 people from a pretty good grab bag of disciplines. We have some from STEM, we have history represented at a large group of composition folks which was kind of the natural grab for us but we were able to put all of them together and so we started at the beginning of the semester with Justin onboarding where Jeremy kind of explained what hypothesis was, got faculty to be kind of excited about the tool and what it could do in terms of social annotation and people's eyes just kind of lit up at the idea that they're like, yeah, we already do stuff like this but we want to do it on an online space so that we don't just provide that kind of pathway. So we're, I'm very interested after hearing everybody speak like I'd love to hear the analytic piece and get that one. I'd love to see something like that in the hands of our faculty and see what kind of questions they bring up. I'm very much the kind of person that kind of just like to lay these things out and then see what kind of questions our faculty come up with. So right now we're coming towards the end of the semester so we're going to be doing some evaluation metrics from them to see what worked with the tool what didn't work with the tool. Ideally want to get some feedback from the students as well to see what kind of hangups there are from the student end but throughout this entire pilot so far we made it as low stakes as possible. We made it so basically the requirements for our faculty were use the tool once you're going to have to do this evaluation metric and then meet with Jeremy at some point just to give some kind of insight into what's working what's not what are some things that you'd like to see done differently. And we've had a few of those and Jeremy's been invaluable and being able to meet directly with our faculty but so is the entire support team anytime anything's come up I can just fire something off be like I don't know the answer to that question let me find out. And everybody knows this is a pilot space so it's been really been a really positive experience so far but like I said it's not over yet so I don't we don't have any of the data or any of the kind of metrics or anything to really explain but the way our team works and the way this pilot's going I think we're going to kind of use this model for future pilots in terms of gathering people and using that kind of group it's been great so far and I'm excited to see where it goes that's all I got. Michael you mentioned that you have this rat bag of different disciplines have they talked to each other yet or are they planning to talk to each other and share across the discipline what they're doing? Yeah so that's part of what we ideally do this semester we want to get the entire we're calling it cohort basically all the pilot faculty back together and kind of have that group conversation I'm definitely interested just from my position and our team's position to see how the different disciplines are using them because obviously there's different kinds of assignments that go with each one like I said like composition and reading and English and stuff like that it was a very logical step I didn't even think of having like how it could be used in STEM classes but the fact that we're interested to see how that's going so again since we're still in the pilot we haven't done it yet but that is the idea to get that at the end. So my my boss Jolte said who isn't here unfortunately she we went through we just did an LMS transition two years ago we went from Blackboard to Canvas and we set out with a pilot but we called it a pilot with intent to adopt very early on and we learned a lot through that process of basically being as transparent as possible when we brought something on board so it wasn't it wasn't really framed so much as like yeah we're going to test this out and if it sucks because one or two of you don't like it we're going to get rid of it it was kind of the opposite it was like if you guys don't have a full blown revolt this is going to be something that you get to you guys get to use I mean granted the LMS was a little different because that was a full replacement for an entire campus this is a tool that you can use but that kind of basically just providing a clear plan from the beginning so the expectations were set from the very beginning it made it very easy for the faculty to come on board be like okay I can do that I can do that I can do that sign me up so I made it pretty easy to answer your question about our faculty fellows that process has evolved over the years it was just kind of faculty that we were close in contact with and then now the way that it stands is actually an application process where we reach out to we basically put a call out and say you know to our faculty they do get time release for doing it so they have to come and present a project that they want to do with our team basically they run the whole project and we're just kind of there to support them as best we can and we go through all the applicants to see which ones work and kind of sort of line up with what we're working on already in which we feel would be a good fit let me go I need to get into one of those classes one of the first classes in the community so I can see it functioning functioning it doesn't have a super awesome you know part of how the system is spring to that class maybe interact with one or two of your people or you know whoever you mind that's a great way to sell the product successful stories from your place regardless they're actually moving conversations right and one of the things I didn't I just I need to see it kind of in the beta version I don't know I don't see it where you're already at yeah yeah certainly certainly glad to share all the findings and everything out I'm sure there's we'll put we're going to be putting a blog post together I'm sure after the end of this is put together kind of wrapping it up one of the other things that was really valuable through the LTI is the fact that you can actually grab files that are in the class so they don't have to just be you know publicly shared articles and stuff you can grab those PDFs and readings that the students are already having to read and go through in the course and then directly work with those and it also synced up with Google Drive so we're great thank you Michael thank you to the rest of our panelists that was really beautiful