 All right, hello and welcome to this project briefing on social annotation across content hypothesis and JSTOR a case study I'm Jeremy Dean for my hypothesis we build social annotation or social reading technology And I'm VP of education which means I oversee our education products and services This is my first time at C&I and I'm thrilled to be here. I'm here Thanks today to my longtime colleague and collaborator Alex Humphries from Ithaca JSTOR where he is VP of innovation They are of course C&I member organization and it was really his idea to share this project We've been working on with this community and like I said, I'm excited to do so and we're joined here Also by Laysha Palin who is really the rock star of the panel. Sorry Alex Distinguished professor of information science at CU Boulder Laysha's a rock star because she's a computer scientist and information scientist So I think she's got things very relevant to this community And she's also one of the most enthusiastic and innovative users of our tool hypothesis in the classroom And she was able to join us coming down from from Boulder today. So down is it down? Boulder's north I think So brief look at the agenda I'm gonna offer Introduction to what hypothesis is and some might not be some of you might not be familiar Laysha will give the pedagogical perspective and discuss how she's used hypothesis in her courses and Alex will discuss Why this matters to JSTOR and maybe why this matters to many in the audience who are in the in the business of libraries and working with libraries? And share details of the pilot project that we've been working on and then we'll provide you all then we'll provoke you all in a Discussion on how we might scale the project that we'll share today So hypothesis was founded by Dan Whaley who sit in the audience. Maybe you can wave his hand Dan saw a problem with information on the web He recognized that much of this kind of the content on the web did not provide space for discussion and Where comments were enabled? It wasn't particularly healthy discussion and on the other hand He saw the emergence of social networks that could be powerful places for people to engage in discussion But themselves could be unhealthy in part because they were separate from the context in which they were engaged But simply content and conversation were disconnected so he founded hypothesis and This is our mission To build new infrastructure connecting the connecting the world's people and ideas over every web page and document on every platform like Google Docs But everywhere and our solution is a standards-based open-source framework that enables diverse collaborative services Wherever you are all based on a new unit of speech the annotation Now I'm an English professor so I would quibble with it as a new unit of speech But I think it is being used powerfully in a new context and that's really what excited me about joining hypothesis originally When I taught I always emphasized annotation in my courses because I knew that reading closely and thinking and writing about the reading was critical to my student's success But I also believe that those skills and practices are critical to a healthy democratic society more broadly That the idea is that instead of sharing a web page or a document You can share a link to a piece of content within that resource along with your commentary as a former hypothesis colleague has Described it if the web is an information fabric web annotation increases the thread count of that fabric And of course we all want higher thread count sheets, right? One of the critical things that we recognized early on was that if this was going to be done right the infrastructure was key This functionality could not be proprietary so high hypothesis was open source from the start And it had to be interoperable so early on we advocated for and led it initiative to make annotation of web standard through the W3C And this is what it might look like a single place to access and organize your notes as you explore the web and other resources a Single space to engage in conversation and community building across that content There are wide ranges a wide range of use cases that hypothesis is already enabled through our technology There is of course the everyday internet citizens and how they engage with content and with each other online There's how scholars engage with the information that is on the web For example the climate feedback group is a collection of climatologists who use hypothesis to comment on popular journalistic coverage of climate change Sometimes provoking retractions through their work There is scholarly publishing itself from bio archive using hypothesis for feedback on pre prints to the journal elife using Hypothesis in peer review to a wide range of journals using hypothesis for community engagement post publication And then there is the classroom context, which will be our focus today the value proposition is Three way for students teachers in schools overall for students The tool makes learning fun and provides them note taking and discussion functionality everywhere. They go for teachers and engages the students So they can see that they've done the reading and know where they're struggling and For schools this can improve retention and we can provide rich data on this fundamental activity that is reading for classes a Key thing that we did when we started building out for education was to integrate in the learning management system for those in the libraries I don't know how much you're Interacting with the learning management systems, but the LMS is the everyday hub for students and teachers It's where rosters are added by the registrar teachers enter grades there students submit assignments and tools like hypothesis can integrate through The learning tool interoperability standard so that students and teachers can access tools like hypothesis without having to log in Hypothesis now works with over 300 institutions of higher education largely North America I know many of these schools are represented in the attendees today We largely work with digital learning offices and centers for teaching and learning But I think there's a real opportunity for us to work more closely with many of you here who work out of the libraries I Want to close by sharing the results of a study we conducted last spring at University, Minnesota in a freshman composition course If you can see it this show this shows engagement data with course materials over the duration of a term The red is from the sections not using hypothesis The green is from the courses using hypothesis And what we're seeing is that the students using hypothesis are more engaged with the content over time Accessing it more frequently as the data here shows. I Think every teacher and everyone involved in storing reading materials for courses like those in libraries We want to see this kind of sustained engagement with the content But for more on what hypothesis social orientation looks like in the classroom. I'm going to hand it to Laysha Thanks, Jeremy. Hey, everybody. I'm Laysha Palin. I'm really glad to be here from the University of Colorado This seems like a really exciting conference, and I'm glad to be able to talk about The way we've been using hypothesis in the classroom. It's been a real transformative Experience for me as an instructor 25 years of instruction, but in this last year and a half. I feel like I've done In improved leaps and bounds in my own pedagogical practice with students and that the students themselves are also telling me that they're having a Very different kind of experience in my classroom and to me I think it's because of what I'm able to do with readings assigned readings difficult readings With hypothesis as a support for me and for them. So I'd like to tell you a little bit about What we're doing and what I'm doing the first thing I might want to just make sure you're all aware of Or maybe recall your own college experience, which is that most students don't do the readings That they're assigned and if you don't want to admit that I'll at least admit that even as somebody who eventually became a professor I myself did not always do the readings that I was assigned in the classroom And what a loss that was because I ended up having to reread all of them anyway to move into my perfect my position as a professor and so What the only you know the ways we can reinforce that is to one not reinforce it and just Regurgitate the material in the classroom the next time therefore Disincentivizing the students who actually did do the work We can cold call a student just to embarrass them and hopefully next time They'll read the reading if they didn't do the reading We can give pop quizzes and all these are very kind of Confrontational ways of engaging with students in the material Which is not what you want in a classroom where you're trying to bring people along for 15 weeks Ideally in a learning community if you can at least if you can create that and you can't create a community when you're in this kind of This kind of relationship where you're testing They're they're they're reading and their commitment to the material and so when students don't read it really drives the The content of the course what can be done in the course in the classroom to the lowest common denominator And so the opportunity we have today with the state-of-the-art with these Hypertext support technologies with hypothesis in particular is to move readings from this peripheral thing that we assign and Can't quite depend on as instructors So something that becomes a centerpiece around which we can have elaborate discussions and springboard New that our projects and other related work that we do because we know everyone has read the reading And so yes So my point is that the readings as you'll see in an example is that they become sites for peer-to-peer interaction and for my Individual interaction with students and it really creates a much more customized experience for students even in a fairly large classroom when I can't Speak to everyone every single time with every single reading I get to every student a couple times over the course of the semester and so they feel that my involvement is Right alongside them in that reading All right, and so what I'm going to show you now is after An example from a sophomore class I taught last semester It was a sophomore level class 2000 level class, but it had mostly freshmen and sophomores in it And we had to engage with some really difficult material It's we're in a new department of information science Many of you might know what that is but many people others do not and there aren't textbooks really for information science And so we rely on the research literature to teach students important things around methods And the philosophies of information science And so this class is called information ecosystems and it teaches about the social life of information Which itself is sort of a heady kind of topic and so we're really asking students to deal with abstract concepts right away and And so the one example I want to show you today is a demo from A paper that they were assigned that had just been published or was about to be published I'm a co-author on it, but we were getting into some very difficult issues around disinformation and the rise of disinformation during the COVID pandemic as it related to Vaccine resistance and as it was tied to medical racism So I just said four very you know big ideas that are all tied together in this paper the looking at large-scale social media records to Show students how these how these things come together and how they might want to research them so let me So I think I have to Exit to get out of this you and get into my demo of you here Okay, great. Thank you. All right, let me just Launch this so I'm not going to give you a full-blown demo of hypothesis But this is the real live reading that we had in class and I want to show you first just the extent of this Research paper its length and I remember giving this to sophomores its length and its mix of methods Which are both quantitative and qualitative in this paper? They're encountering the Tuskegee Syphilis study, which is the first time many of them have encountered this So we have to establish what medical racism is and then we have to establish how it's being re or co-opted Once again in some complex argumentation around the COVID-19 vaccine distribution So I just want to show you the I'm just scanning through just to show you the range of this is qualitative Interpretivist work that they're reading about here, and then they actually get down to network Network graphs that show how different groups are communicating with each other So it's a large it's a big range of material and then what you see over here on the right are the students annotations and I'm going to specifically show you How they respond to my prompts and a little bit about how I've developed my pedagogy here Just to show you what how powerful this can be so here you see all my Annotations that I've made and I have two types one is a guidance annotation which Gives students I landmark things for them and give them background that if I could be with them one-on-one in a much smaller group I might say hey, you might not understand this phrase. This is what this means if you want to learn more about this go here This is a really difficult piece. This is the big point skim over that and don't struggle there I want you to focus on something else So those are that's the guidance so they feel supported and coached when they're tackling a big paper And then I give them prompts and the prompts they have to answer and the prompts are a range of pretty difficult Questions that ask them to reflect on what the reading means It asks them to summarize what different parts of the reading says It asks them to respond in their own personal experience And so I do a whole mix of things and for a paper of this size about 30 pages I have about 25 annotations throughout prompts throughout and the reason I say that is is that by having the annotations throughout They actually have to read the whole paper rather than just skip from one annotation to another Which they'll do if you only put in three and I learned that lesson So the trick is if you get them to read the whole paper They're at first a little bit resentful of professor Palin, but then after that very first reading They get through the whole paper and these are students who are Hearing I mean, you know, they're in such pain right now. They're experiencing all these things in the world around With what we're experiencing they they're looking for careers information science sounds like something that might help them solve All these problems, but even for them. It's a hard thing to define by the time they get to the end of this paper They have said to me professor Palin I finally understand information silence science and I see how all these things come together and it's Wonderful because they've read each piece of the paper. So it's not just getting them to read it's getting them to read slowly and Reflexively and start knitting things together and not being afraid to ask questions So just as an example in this class. I had 47 students by the time they got to this reading so here's a first prompt and 44 of all students are applying to these prompts throughout Okay, they're reading the paper you go down to some next prompts which I'll get to real quick and then close up They are so you can see they're annotating really extensively and Right and so the the amount of writing they put into this is similar to what they would do if they had to write a one or two page Summary of what the paper is about but it's just much more extensive so I'm going to close there on that and then go back to the presentation and close out with a few of The the lessons so I've experienced this wild success by create using By using hypothesis as a way to coach them outside the classroom so that when they come back to the classroom They they report feeling these things because they've read they feel this personal pride They're ready for a class like they've never been ready before a class before they feel cared for they talk about that Because they're getting what they crave. They just don't know how to read or why it's valuable And they're starting to even understand why it's important to be self-disciplined about things like reading at a at a young age In their college experience, which is just great and so The one thing I want to close on is this point It's not just that each individual person is reading and that I might be responding to them and saying oh, that was a really interesting Comment you made Joe It's that they all walk into the classroom on Tuesdays and Thursdays at 12 30 And every they know that everybody in the room read the paper Not just themselves, but that everybody read it and that I read all of their annotations And so what that means is our conversation starts after everyone has done the reading and I Pull out certain things that seemed especially potent And we talk about those things and then apply it to whatever project we're doing information science project We're doing in the class so I can offload lots of things to outside the classroom But the feeling that comes into the classroom is this kind of readiness and preparation and this kind of commitment to each other Around this learning experience, and that's what the breakthrough has been for me This is just one classroom experience and so in speaking to the themes that Jeremy and Alex and others are talking about in this room is that you know, we no longer need to think about sources The the use of the sources the uses of the sources being divorced from each other We can think about the network power of reuse of materials scholarly materials that can support Contemplation individual contemplation and Collaboration around scholarship and this seems to be an essential thing that we need to do right now. So thank you so much All right, if I hit slide show will it come back so that I don't have to crane my neck. Yeah, I'll crane my neck That's okay. Hi. I'm Alex I'm from Jaystore If you don't know who Ithaca is come see me afterward. You should probably know some of you work for it So first of all I feel like I need that experience in a lot of the meetings that I go to Because that experience of people having read the materials is important. I As you know, I work for Jaystore Jaystore is an enormous library has an incredible rich Content usually it's associated with the research enterprise But all of that material is largely driven is very important for the educational enterprise You can see our little usage annual usage chart at the bottom huge peaks During turn when term papers are due when students are doing research Undergraduates writing papers is a large part of our usage But some of our usage a lot of it is actually driven by more directed usage where a teacher is Sharing an individual link to a canonical article sharing including Jaystore material and syllabi and We would want people reading this material because it's just as complicated Some of it as the material that Laysha was just just showing to have an experience in an environment where they can Collaborate with their teacher and learn like that But there are some barriers and hurdles to that happening So right now teachers already do this they share Jaystore materials with their classes. I'm sure they do that for some of your platforms as well That happens in a few ways. So a teacher might share a link in the syllabus Problem with that is that students tend to drop off Especially because the whole authentication thing gets really complicated and it's very easy especially for when people are offline to Not be able to authenticate or get lost in a proxy server somewhere Teacher instead to avoid that could embed a PDF onto the J onto the into the LMS That allows the student to have immediate access But the nobody sees that usage and so actually it's against Jaystore's terms and conditions because that usage isn't Isn't visible and that's very important to our publishers to be able to see that teachers do it all the time though and then even when they do get access the Students can have to read the material and as we just saw reading that material is really Challenging and it's intimidating and especially if I'm you know at a community college or an early undergraduate Getting the scaffolding and supports for that environment for those challenges is really important So for the past year ish We've been working on a pilot project But between hypothesis and Jaystore to solve that problem We developed it about a year ago allowed into the summer and it's been running at 30 institutions in the fall semester and the spring semester We've been gradually Expanding it to more and more institutions as there's been interest and we're seeing some positive results We've worked with teaching and learning centers across at each of these institutions to build awareness Provide supports for teachers like Leisha to be able to use the integration And have conducted follow-on quality qualitative interviews with those who have used the integration to make sure that it's worth scaling and that there are Nobody gets dropped in the handoff and all of that so what that's allowed us to do is Have an environment where teachers can embed Jaystore material directly into the LMS using the hypothesis integration and do so in a way that Students immediately get authenticated at so they don't there's nothing lost on the way and it's on the Jaystore Platform so our publishers see the usage we see the usage usage and it's within our terms and conditions that then creates the environment that Leisha was talking about for collaborative learning and And for collaborative learning So let me show you what that looks like really quickly although And then we'll go from there. So you already saw with Leisha's quick demo canvas This is a test version that we have teacher assigns an individual in this case. It's an article One of the more canonical articles in Jaystore called. What is it like to be a bat? We're now as you make an assignment you select the teacher selects the tools that they want the the the Students to engage in whether it's a quiz or whatever in this case They select a hypothesis reading assignment and then choose Jaystore article when you're putting together a syllabus you have links of Materials that you're going to refer your students to that's what we ask for here They can get that from the Jaystore platform and then when they do that what we're doing is so the teacher has already Authenticated into the learning management system. They just sign in every learning management system as a has controls like that That's associated with an institution Jaystore And so an institution code and hypothesis gets accessed based on that We've worked with hypothesis to map the institution codes between Hypothesis and Jaystore so that we can then connect that we know we can look up for this teacher whether they have access to this to this article Jaystore is rapidly trying to expand access to all of its materials But still there are some things that not everybody has access to so we want to check that We verify that they do and if they do they can go ahead and create the experience This is a and so what this is essentially an iframe So this is a hosted environment that they the student gets to directly from the learning management system The left side of the pane is Jaystore's part. That's on Jaystore platform. That's hitting every reuse hits Jaystore And this is significant use so this is not just a single Download but you saw the number of comments that Laysha was engendering every one of those refreshes is a page view and a content access That publishers and librarians should care about One class of one assignment of one article at UC Santa Cruz led to to 30 people led to over 300 content accesses of that article That's great. We're thrilled about that And then the students don't have to reauthenticate get access and because it's at the site level not individual Privacy is protected. We never Jaystore doesn't know student accounts or your teacher accounts or anything like that or see the annotations Those annotations are then kept within the context. They can be kept private as a As a student I can write notes to myself I can write them to my class if I have a hypothesis account. I can save them for perpetuity and archive them I don't have to if I don't want to but the the context of this as a classroom is very important because annotations We don't want all of social Conversation in the wild that conversation around your paper would have been very different if it were in a public forum And it creates this collaborative learning environment that can be really excited Really powerful that's especially important in an era where there are all of these forces like chat GPT That may offer alternatives to the kind of close reading and engagement with materials that hypothesis And these tools can provide Chat GPT, I you know, I would argue that especially in the humanities a Summary of an article is not the same thing as the article itself and engaging with the actual text and language Deepins the understanding and Lisha's example absolutely testified to that What we learned during this pilot was that those who used it were over the moon at the At the ease we had very a lot of very strong and positive Experiences we did have some trouble some struggles driving awareness of this J store materials are not always thought of as textbook materials. So this was a little bit different kind of Activity, but as they grow as teachers had awareness We're seeing gradually increasing Interest and value and so over the next year. We're going to be extending the pilot Expand it to many more schools as we can it's still a pilot So there's some infrastructure that we need to continue to develop to scale that build awareness and then the other thing I'm really excited about is Sneaking in a plug to add image annotation J stores very interested in having we have a lot of images that are vital to what we have on the collection and Being able to annotate those in the same kind of environment is really exciting Can't be done now, but we're going to be doing some testing to be able to do that And that's how we're planning to do to scale it within J store environment And I'll hand it over to Jeremy about scaling beyond that one concern I have is that we've made it look too easy for you how all this works and so We're not going to turn to talk about scaling this operation or scaling this case study But I want I've heard that One of the things about CNI that's so great is that It's not the presentations that are great. It's the conversations that follow So now the pressure is on you guys because we're going to provoke you in a discussion of some of the topics that we've brought up today, so You all know that educational content is increasingly digital And that that content is delivered by many many different platforms some of them represented in the membership here or in attendance And so students every day are navigating They're going from the LMS as we saw and navigating multiple sources of content every day And they don't have a unified experience because sometimes those places have native tools that they can they can leverage Sometimes those content platforms do not and then when they're in the LMS They have other tools that they can access and so even though it looks really easy to do what lace is doing and What we're doing with the J store pilot. There's still a lot of infrastructure there's still a lot of work to be done to truly make a good experience for students in this environment and I Want to return to a sort of mirror image of the problem that hypothesis began with This is you have to be ready for this right I want to hypothesize that some version of the original Hypothesis hypothesis exists within the education space There's the LMS where a lot of content is being delivered There's some learning tools that students can access in there and then much of the content most of the content that students are Accessing is actually outside the LMS And so they're moving back and forth and they have certain tools in some places and no tools in the other and other places there are different tools in other places and So the problem that we're dealing with I think is trying to create a better experience for students and for scholars as they're navigating all these different platforms and That's where we want to employ you guys to start to think through Do you see this as a problem from your perspective wherever you sit in this? Marketplace what are your challenges? How can we address those challenges together? And so with that we're going to open it up for Q&A and discussion What happens to that doubt it So I'll answer a part of that but I'll turn it back over to Jeremy to answer it fully so the the so the demo I showed you was back was That reading is already nine months old So so the the content lives indefinitely in the local canvas instance But students at least at CU don't have eternal access to that canvas instance Certainly like So I'll be teaching that same class again in the fall. I already know where Based on things that students understood and what they didn't understand how I want to introduce things reintroduce things so it's a it's a it's so it's it's a lot of data about Student learning it's all so externalized I also use it to prep the next time the the actual readings themselves. And so yes having that content is helpful Just some small examples. I mean this is much smaller But you know, you know one of the big things I do is write a lot of letters of recommendation for students And it's really helpful to go back and say what did Louise how do we perform in that class? And what did you really think and so I can write a much more Much richer kind of much richer experience of each student that comes through And that's wonderful too I mean they come to me for other reasons because they know I know them through their Writings even if they didn't talk in in in in class I'll just add that one of the things I think you'll appreciate about Dan and hypothesis and how We've gone about building this infrastructure is that it's been deeply archival in its nature, right? And that's part of the problem is some note-taking platforms aren't really attending to the true source of a document and to locating You know commentary and discussion in those original sources and hypothesis has built its infrastructure So the things are grounded in you know unique identifiers for documents Whether that's a web URL a PDF fingerprint or stable URL in the case of the J store integration But then the trick is showing those annotations in the proper contexts, right? So Laysha has probably continued access to her courses at CU Boulder She should be able to take her prompt annotations and move those to a new context a new course Those students again depending on the university policy may not be able to go back and see those annotations in context In the in the course, but they're anchored to that document and that document has a unique Identifier in the world again, whether it's a URL or PDF fingerprint, right? So theoretically not all the infrastructure is built there, but theoretically students in Glacier's course if they Graduate undergrad and they go to information science, you know graduate school and they go to a different university That university also has access to the same, you know repositories They could we can reconstitute their annotations for them in those new contexts because all the infrastructure is in place for them Once they're in that, you know have access to the content they can call up their annotations again I hope that helps answer the question It does include book chapters. So not entire books book chapters. I mean theoretically you could add a hundred chapters Great That's right Absolutely, I mean, I think that's the crux of the problem right there and the crux of the opportunity And that's absolutely what we're working on the JSTOR Implementation is meant to be a model for that others might follow We are working with some textbook publishers as well as you know in conversations with other journal journals and journal repositories But it really only works if we that's the idea of the interoperable infrastructure is that it has to include all these different partners content platforms tool providers for really to be a universal, you know seamless up Experience for for students and for scholars and so we are in conversation We've launched a coalition called the social learning across content coalition, which includes a number of I think I actually have a slide for it. It's not hidden But yeah, I think Elsevier is part of that Dan and Hathi trust Evesco ProQuest all of these folks have committed to the idea of interoperability and starting to address the challenges of all working together so that tools like hypothesis could work more easily across all those different content platforms and then JSTOR is one of the first sort of concrete prototypes that has come out of that conversation But it only works if this is because you know students and and teachers are agnostic about the source of the content Right, I'm a big fan of JSTOR, but the article I might need is in you know Epsco or ProQuest or something like that and so unless I can access those other you know content providers It's limited For Alex you said that the 30 students in Lace's class generated 300 hits wasn't Lace's class was another it was another class But it generated 300 hits against JSTOR content accesses, but yes, so are aren't you afraid of hit inflation or You know having an artificially large amount So the way JSTOR and I'm not the best person to talk about this, but the way JSTOR accounts content accesses tries to Be careful with that with hit inflation Which can happen when you're just doing web data content accesses is usually a single PDF download or A set number of page views within a session and so 300 means they've come back multiple sessions I mean as far as is that what you were talking about with inflation or are you worried or do you think that JSTOR doesn't want that usage well It may affect your usage statistics in a way that is you're not prepared for It's possible. I I think I'd love like that problem right like I that sounds like a good problem to then have to work through and See what we'd have to change to address But I I'd love to see that additional usage so and you are tracking it on a granular level as to page views Yes, we we log. Yes. Yeah Counter it's all counterfeit. Thanks Save me time Bruce You know I have seen some interesting annotations done on Transcripts as a way that are time bound within the video And that can be a nice way to deepen the enrichment of A video and sort of force that close reading for it if you want to call it that of a video I thought that I bought this this is working on the image and video annotation this this calendar year So we're expecting to add that but I think that's part of it, right? There are video annotation platforms and their text annotation platforms and their ways to mark up images But if you're using a bunch of different places to take your notes and having your conversations and it's not unified So you really need to have a system that's taking into consideration all the not the content providers only but also formats of different Document types images video, etc This I guess I don't think about it like that. I I mean I have readings. I need students to cover in a course So I don't assign more or less To compel the reading I It's my my attachment of what they learn in the reading to Projects and things that are is meant to compel them and then you know if that doesn't work But it usually does because we're all we all then become pretty invested together Then some what's great about hypothesis is that it's integrated with the learning management system And you can assign I don't care if it's one point out of a thousand or ten points out of a thousand They will still read that paper, you know, we're just like, you know So so so there's just all those things together that keep them So I don't so I don't do more or less because of that. I just want them to read the ones I say and Please commit to that. Yeah The guardrails really help with that Yes I Love it. Yeah She's gonna earn an administrative role actually for all her advocacy for hypothesis So my learning curve Was three semesters which it wasn't that bad because the first semester was still a success Just not I wasn't as honed in what I came to by the third semester So so what I didn't show you was that in addition to the prompts that I asked students to write to They are also writing additional ones on their own. There's a lot. So I mean I must have I don't know Maybe maybe 800 annotations per paper like that for 50 students or something like that. I don't know So that's question What so so what I what I did learn was that? the biggest shift in my practice was Asking students grads as well asking them to annotate at least three times in a paper in a paper or an article They would do three times and They would do the beginning and the end and then somewhere in the middle they so they weren't reading it It was more than before because I at least got them to open up the file Right and so that I'm like, okay I have to be and so then I the slide we skipped because I was using too much time And that's my fault was I want I I realized I could use this as my Virtual me so if I could read so I'm teaching ethnography class right now for Computer science, which sounds crazy. I know but but it's with 50 students and when you're teaching ethnography class Ideally, you just want to teach five people at a time and you want to go out into the world and show them how to see things Which you can't do that. So like so the whole philosophy is how do you How can you coach students without being there and so what the annotations do? Yes, it gets them to read the paper and I have to make them do that But I prompt in a way that I'm acting like a coach Like I'm always whispering in their ear as though I'm reading right next to them And so some of that heavy stuff around the Tuskegee stuff a lot of these students had never heard of it before And we could get to the end of the paper and they've read some really disturbing things about Disinformation online right now and how black Americans are being attacked And at the end I asked them how are you feeling about this? How are you feeling about your classmates? What do you want to bring to the classroom and that's that kind of coaching slash Therapist maybe a little bit which gives them at least a place to kind of talk about that reading and and perhaps bring it back To the classroom or at least let others read it there even if they don't want to talk about it So that's how I evolve knowing that I could be that kind of person to them and not just read the paper I could be their coach and their person that supports them And I think with that we're at time, but we're gonna be around for the remainder of this Conference so please pull us aside and I hope you guys will you know consider that you know What laces being what laces done supporting students giving students an opportunity for peer learning in the kind of difficult content that? Some of you are providing or some of you are starting on your campus is really important to get them to engage with so thanks for everything