 And welcome to the Hypothesis Social Learning Summit. This is the session titled The Latest Research on Social Annotation and Social Reading. First of all, the session is being recorded. So folks are aware of that. And also, so people who cannot join us live will be able to access the presentations and the discussion following today's session. If you're using the AirMeet platform for the first time, please know that there are a few features similar to Zoom, which I think many people are using pretty frequently these days. There is a chat, and you're very welcome to use the chat to say hi, perhaps tell us where you're joining from, maybe what you do professionally, and also to use the chat for general banter throughout today's session. There is also a dedicated question and answer or Q&A feature that you'll see here in AirMeet. I'll mention this again. And as we hear from our featured guests today and as we move deeper in today's session, if you do have questions for either of our guests or for both or for me or for the panel in general, please do use the Q&A feature so we can bring your questions up. So let me introduce myself briefly. My name is Raimi Kahlir. I'm an associate professor of learning, design, and technology at the University of Colorado in Denver. And I'd like to begin my brief remarks this morning with a land and labor acknowledgement. Although you're of course gathered here in a digital space, we all occupy various places that have rich, if not quite complex histories. And I join you from the ancestral homelands of the Cheyenne, the Arapaho, and the UT people, otherwise known today as Denver, Colorado. And I also want to recognize the labor that is very much a part of what brings us together today in this moment. I want to first begin with folks at Hypothesis who have really helped to make today's event possible that includes Wendy Morgaine for all of her work or organizing speakers and logistics. Matt Dricker is here today behind the scenes, helping with some tech, as is Aaron Barker. And I also want to recognize the folks whose labor perhaps with our families like Maesnos makes my time and presence possible today as well. So many people make our participation possible and that labor must be recognized as well. Let me also mention that for some of you who are joining us for this session for the first time, you may know a little bit about me during the 2020-2021 academic year. I served as a scholar in residence at Hypothesis and much of my scholarship and public writing and advocacy concerns social annotation. And it's really an honor to be invited back to moderate today's session with really two very dear friends and colleagues. And I'm so excited to introduce them to you in just a moment. I'll just briefly mention the kind of outline for today's schedule or today's session. I'll again introduce our two guests. We'll then hear their presentations about 10 to 15 minutes each. And then following their presentations, I'll ask a few general questions to them both. We'll of course also be listing again Q&A from all of you. Our participants, I'll let the conversation evolve over the course of the next 60, 75, 90 minutes. We'll see how things go. So that's the general outline. Again, there is a chat and again, there's a Q&A feature here. So let me begin with a few introductions. And the first is to my dear friend and colleague, Dr. Cherise McBride. Dr. McBride is a post doctoral scholar at the University of California Berkeley School of Education where she works on the Writing Data Stories project. It's an effort to teach data literacy through storytelling. Her current work is supported by the George Lucas Educational Foundation with a focus on equity-centered learning environments. In addition to that role, she also serves as a scholar at large for YR Media, formerly Youth Radio, which is a long-standing Bay Area organization dedicated to empowering youth journalists. And there she researches youth cultural belonging and teachers development of humanizing digital media curriculum. Dr. McBride is also a former high school English teacher and as coach teachers and higher education faculty across a range of content and educational settings. I have known Dr. McBride for quite a few years now. I was actually looking for the first date of our presentation together at AERA and I actually couldn't find it. I have to admit it was two or three years ago and I just couldn't quite dig it out of ICV somewhere. I primarily have collaborated with Dr. McBride through our work together on the marginal syllabus project, which is an effort that uses social annotation and specifically hypothesis to invest educators in discussions about equity-oriented literacy education. And in that capacity, Dr. McBride and I have led the project. We have presented together. We have a research and written together. We've done quite a lot of work over the years and it's just an honor now to see her related research grow in a variety of ways and to present some of that work to all of you today. So Dr. Shreve-Bride, you're so very welcome. I'll introduce Dr. Hodgson-Brutely and then we'll jump back to your presentation, okay? So today we're also honored to be joined by Dr. Justin Hodgson. He is an associate professor of digital rhetoric and the coordinator of online first year composition at IU Bloomington. He's also the founding editor of the Journal for Undergraduate's multimedia projects and serves as a global digital literacy thought leader for Adobe. He's an award-winning teacher and digital rhetoric scholar whose research concerns the intersections of rhetoric, technology, play in games, digital pedagogy and culture. His 2019 open access monograph titled Post-Digital Rhetoric and the New Aesthetic was published with Ohio State University Press. Most recently, he's helped to create and lead the Digital Gardener Initiative at Indiana University, which is a digital literacy initiative focused on student engagement and faculty development across all of IU's seven campuses and two regional centers. I began to collaborate with Dr. Hodgson a few years ago now in my capacity as scholar in residence at Hypothesis. We quickly hit it off, quickly became dear friends and have now been deep in data, deep in writing, leading a team together that has just been quite a lot of fun. And so there really are no better folks to talk with us today about the state of social annotation research to ask important questions that are both critical and creative to look both broadly at scale but deeply at intimate practice and to think about how across both K-12 and higher education contexts, social annotation is really making a difference in learners' lives and in learners' literacies. And so with that long and very deserved introduction to both of our extremely distinguished guests, Dr. McBride, welcome. It's so good to see you. Thank you so much. The floor is yours. Great. It's always a pleasure to be in conversation with you, Ramey and your colleagues. You always bring such great people together, including those in the audience. I'm so happy to see so many of you joining us this morning or for me, it's morning. I'm joining from the beautiful Bay Area in San Francisco, Bay Area, specifically in Oakland, the land of the Ohlone people in particular. And yeah, I will go ahead and jump right into it. Let's get my screen shared. Okay, great. So as Ramey shared, my name is Cherise McBride and I am a postdoctoral researcher at the Berkeley School of Education. We recently changed our name from the Graduate School of Education. And today I'm excited to talk to you about social annotation research, particularly as rooted in teacher learning and what kinds of impact we're seeing in teacher learning. And this will all be connected to this frame of justice-oriented media literacies that's been an important part of my work in recent times. And this is in light of my work with teachers and youth. So I wanna start by saying that social annotation has become especially compelling to me as I'm interested in socioculturally situated learning. Meaning that I recognize learning, again, particularly teacher learning as something that often happens in community and in culture. So in my own former work as a high school teacher and as a continuing lecturer from community college across to graduate school levels, I recognize that teaching is indeed a public act. It's one that requires a great deal of collaboration, inquiry as teachers continue to refine their practice. So when I think about teacher learning, especially in a digitally mediated society, I think it's important for us to take up using technology in authentic ways that really support teacher learning both their own and that of each other. So as a side note, much of my work has been in participatory networks of teachers or social media use and how they learn together online. But another key part of my interest is in training teachers to use technology. And that's where I think social annotation is really useful in that it offers sort of a situated ed tech practice. It's an opportunity for teachers to use technology toward authentic ends, including their own learning. And then through that process, they're continuing to build their repertoire as a practice with technology, whether that's learning new affordances or developing specific connections to teaching and professional learning. And then all these together, of course, promote new literacies. As a literacy scholar, I'm always concerned with how folks are engaging different ways of speaking and listening and writing in ways that are particularly connected and social and multimodal. And multimodal in the sense that they're involving the creation and consumption of a variety of text types across words, highlights, colors, images, sounds, videos, all sorts of different modes of meeting making. So with that framing of the importance of the placement of social annotation in that cycle of teacher learning, what I wanna focus on today is work that I've done that comes from work with secondary and college educators. Specifically in this example that I'll be sharing today, I've worked with teachers who have been teaching a variety of subjects, including history, English, media literacy explicitly. And through these experiences, my focus is always on making thinking visible, right? Again, touching on that frame of learning in public, my approach to professional development, to curriculum development is one that really opens avenues for sharing and for really what becomes taking risks together in engaging multiple forms of media. So these values and commitments, of course, are central to my practice and specifically in my practice as a teacher educator. And I also seek to honor the lived experiences of teachers, as I said, how they're learning organically. So this construct of, oops, excuse me, justice-oriented media literacies is one effort to really encourage and push teachers' thinking and practice in ways that really center justice. And of course, I think that's important considering the history of dehumanization and oppression in schools, which really have served to subordinate certain experiences or cultures and perspectives to those of the dominant culture and perspectives. And so this concept, justice-oriented media literacy, intentionally makes room for those subordinated voices and contributions to be made visible. And that's, of course, an example as Rami was sharing, the marginal syllabus, this is an example of bringing the margins to the center. And the other thing about justice-oriented media literacy is kind of a mouthful, Jommal, is that they're rooted in digital meaning-making, digital technology. And this is specifically from the frame of critical reading and composition with digital tools. We're thinking about how media exists as artifacts of learning. And finally, I've intentionally grounded this as a curricular framework, rooting it in practice. I think it's really important, of course, for us to connect theory to practice, and this frame allows concrete building blocks, as we call them, for considering specifically authorship and commentary in a digital context. So I'm gonna turn now to the framework. And this concept, again, is really rooted in authorship. You'll see that at the forefront of the model. Authorship is, to me, a way to consider who gets to make meaning, who gets to construct knowledge. And we see how often in our historical understandings of the author, the author has become this codified, legitimated voice in meaning-making. But often, authors in traditional classrooms, especially in textbooks, are kind of this objective, in some ways, above questioning entity. And in this construct of justice-oriented media literacy, I intentionally push against that to think about the author as a function that's open to students, and it's a space where they're able to express their voices to create and compose. So, and then also with that is unpacking the authorship of texts and unpacking the authorship who wrote this, what might be some of those motivations. And that's where the media literacy is kind of takes front and center. So that's, as I was saying, authorship. Now, moving from authorship to thinking about commentary and annotation is really important for us to interrogate, like, what does an author do? And this is where I really appreciate Rami Kallir and Antero Garcia's reflections on commentary, and particularly, annotation as commentary. That's one of the functions of annotation. So, what this means is that given so much user-generated content that we see in our world today, commentary or the space to do these functions down at the bottom here really becomes accessible, and something that we can all do. I love how they call it any egalitarian affair, this idea that anyone can comment on anything, almost anywhere. And with that, I also appreciate that annotation is not restricted to written text or print text. Rather, almost anything can be annotation, whether we're talking about graffiti or perhaps posting comments on social media. So all of these serve as chances to categorize information, to react to information, correct inaccuracies, which I think is really important in the pursuit of justice. And these are some of my favorites, but there are tons on this list. So now that active, when we look at commentary, it's something that this framework invites, and again, inviting youth to be authors and opportunities to annotate texts themselves, their worlds, and with that, that's where text pairings become really important. As we consider what are our youth voices being put in conversation with? How are youth texts even being elevated to that level of being paired with canonical texts in conversation? So all of these are questions that I invite educators to consider through the work that we do together. I'll be running another series of workshops this summer through a partnership with YR Media and the Bay Area Writing Project, which is where I kind of come out of. And these are all rooted in how our students are becoming authors and how they're, all this together really, it addresses digital technology and content standards that we aim to not only meet but exceed. And I won't be going so much into that today, but I really want to emphasize how these three commentary and annotation, text pairings and authorship work together. So if you're interested to learn more, you can take a look at our article, which is coming out soon in the Education Leadership Quarterly. It'll be out this year, later this year. But what I want to do now for the rest of the little time that I have is to turn our attention specifically to the actual annotations and texts that teachers have worked through. Now what's interesting is this is directly from the YR Media site. If you go to yr.media, there's tons of youth-created media that has been shared, not only here, but even through other outlets, including NPR and local news radio stations, as well as other websites and podcasts. And I think this will give us an opportunity to see the inner workings of social annotation and action. So this particular artifact is an article, multimedia article that was created by a youth participant named Kiera Fraser. Again, this person works with YR Media. And this article is about a youth media organization, I'm sorry, not a media, youth mental health organization that was started by a young person named Diana Chao when she was a teenager that really responds to issues of mental illness among her peer group, as well as things that she had experienced herself. So in this example, you'll see the artifact itself, and this is a webpage. And then to the right, you'll see our, Anna, oops, I just clicked the webpage. I'll go ahead and let you take a look at it. So this is kind of what the YR site looks like. And you'll see these different news verticals. And this one I think was in the health section. So these yellow highlights are things that the teachers and the YR group have commented on. And my goal here was to give them a chance to learn from youth media, learn from, build from youth boys. And so here you'll see, I wanna zoom in on this conversation that's happening in the margins among teachers. Now, these are all put together also by students who are ages about 14 to 25 just to clarify what youth means in this context. So what I really appreciate is that it offers a potential mentor text for our students, but also content for teachers to learn and discuss. And so if you see here, I'm happy to answer any questions about the logistics of this space, but you'll see, oops, I keep looking at accidents. Sorry about that. Okay. You'll see here one of the teachers highlights how after being diagnosed with bipolar disorder at 13, Diana realizes the importance of advocating for mental health. And so this teacher's note here is I wonder how the author of this article heard about this. So as I look at it, he's already taking a step back and particularly focusing on authorship and how this youth author has put this piece together. And then here we see another annotation that I think highlights how as a first generation Chinese American, Diana encountered stigma related to mental illness and she wanted to fix it. And so one of the teacher's comments is around I wonder if this stigma is something she's feeling within the Chinese American community. Did we lose her? I think we may have just lost her for a moment. While she reconnects, I'm gonna just say that I've got a whole page full of notes here and for folks who perhaps are following along, I'm again, pretty familiar with Dr. McBride's work but the flipping of the script here is actually something that I had forgotten a little bit about and I think is really quite fascinating that we have here a case of educators learning through social annotation as they read texts written by youth. And I think that that's in and of itself a rather fascinating dynamic. And I know that we'll hear from Dr. Hachin in a moment about a more dare we say conventional although equally important learning environment, let's say undergraduates who are learning how to read and write in the university level, very important. And one of the reasons why I find Dr. McBride's work so compelling is because we are seeing here designs for in this case, professional learning that take advantage of the affordances of hypothesis in this case and social annotation more broadly and really invert expectations around authorship as she was mentioning and voice, whose expertise is worthy of study and then how educators themselves are the ones making meaning of these narratives which in the case that she was just showing as an example of youth mental health which is of course a pretty pressing topic these days. And so I find this to be again really quite fascinating work. And while she's again rejoining, I want to just again orient people back to the chat. It looks as though we have actually a link to the artifact that she was just sharing which of course is annotated. Although I'm not sure if those annotations are public or they may be private to a group that she was working with. And again, the Q and A function is, yeah, the Q and A function is also available if folks want to drop a question into the mix at this point in anticipation of her rejoining us or if we want to just kind of pivot but happy to hold the floor for a moment here and hopefully we can resolve the tech issues in just a moment. I can also elicit from no public annotate. Okay, thanks. Yeah. Oh, good to see you here. Glad to hear. Looks like we're going to have to have Dr. McBride rejoin us. And while she does so, happy to again address any questions, comments in the chat or if I could just put you on the spot, Dr. Hodgson, quick reactions or quick thoughts to kind of getting a glimpse of that approach to work particularly in light of your experiences, using social orientation in a very different learning environment in a very different learning context. Yeah, yeah. I mean, I think it's a fascinating way of thinking about on one level, just data collection, right? Like as a researcher, as a scholar, as somebody's interested in things and make an inquiry as an inquiry practice of different kinds of groups in the world. If you have students who are writing things up, you can then collectively and collectively analyzing that work through this tool. It's a new way of thinking about how we go about the kind of legwork of doing the scholarship, right? And so I'm fascinated by that. I never thought about using the tool for research itself, but rather researching the tool and its application. So I'm on the other side of that split. But it does, I mean, of course, there's all kinds of wonderful things that come out of that and other kinds of concerns too, like how do you get access to groups and student content, especially if they're under 18 and those kinds of things, but the general principle of making an inquiry and thinking through the inquiry on the text of young writers, especially someone who teaches writing, it's an interesting area that could be opened up. Madeleine from Queensborough Community College says, how do you communicate with ESL students? That's a really good question. In terms of like in class or in the antitions? Yeah, well, Madeleine, let me say that if you're learning about social orientation here for the first time or seeing examples like this, students can read and write in any language that they choose, including whether they're using hypothesis or any other various technologies. And so, Google's asked, can be written in lots of different languages. Chats can be written in a lot of different languages. So I guess I'm just like hypothesis is equally language inclusive in that we're reacting. We had a really lovely presentation, I guess maybe last year about specific world language use cases of hypothesis with educators from Mexico, talking about their students who are online in Peru and an Italian educated living in Korea, doing all kinds of different, you know, second language words have been some really interesting published scholarly pieces about English language learning in Japanese contexts recently as well. And so I think that, you know, a tool that affords as Dr. McBride was saying, socially mediated meeting making may not be specific to a given language. Yeah, and I was just gonna add that, like, you know, so I have colleagues who are like lexographers or they study like word etymology. And on the one hand, you know, this kind of thing offers a really fascinating way of like, in texts, arguing those things through. But a colleague of mine in linguistics who teaches sort of travel narratives and things, I'm trying to get him to use this to talk about translation activities, right? Here's the text, here's the translations and to embed some of those conversations. It's not a straight ESL kind of conversation like we have in composition, but I think the possibilities are really fascinating if you, you know, it's a specific construct as well. So just to put a session update just to jump meta for a moment here, I just got a text from Cherie saying, hey, I just lost my internet. So of course, this is of course, the challenges of living and learning and teaching in these, again, digital spaces where we take it, Fortin's is for granted and things pop up. Let me just say, Madeleine, a response to your question very again briefly that there is just again a really long history of the use of social annotation, again, generally speaking, used in language learning contexts. And so if you're looking for particular research articles, strategies, whatever it may be, you know, connect with the hypothesis folks, we have a public Zotera bibliography that includes some of those citations and you can definitely get access to some of those resources and that might help you as well. So I'm gonna keep my fingers crossed as Dr. McBride jumps back in and joins us for the sake of just keeping things moving along. We're gonna just mix it up. So Dr. Hodgson, let's jump in, let's run through what you're talking about and then we may jump back into Dr. McBride's work and we'll just kind of follow the associative trails as they may be given that we're talking about annotation. Yeah, yeah, we'll just annotate our way through today. A real quick acknowledgement on my end, you know, I'm at IU Bloomington campus and so I'm joining today from the lands of the Miami Delaware Potawatomi and Shawnee people who are the past, present and future caretakers of this wonderful space that I get to call home and then the campus that I get to work on. And so I appreciate, you know, those kind of acknowledgments as we get started. I'm gonna take a little bit of a different approach, you know, because it's not, I'm not gonna get too much into the weeds as far as like analysis of specific annotations. We're gonna talk much more broad-scale, but let me share my screen and kind of get us started a little bit so you can see something other than my face. Select tab. I like to narrate as I go, you know, so that's one of my, all right, am I sharing? Hey, there we go. Social annotation at scale. And to make this easier for you all, I'm going to actually, this is created in Adobe Express webpage, the Adobe Creative Cloud Express webpage and so it is live online. You can access it. There's not a lot of content here but the images I'm gonna show, especially as we get to some of the finer details will be a little hard to see on the screen. So just easier if you can access it when we get to that point. So I think it's important to understand a little bit the difference here. So while Dr. McBride is looking at specifically use cases in a sort of focused way, sort of my journey is more of what happens when you employ social annotation at a programmatic level of scale. And what this means is we decided for various reasons and I'll give you the backstory, but we decided to include a hypothesis into our first year accomplishing curriculum. And then with the pandemic, eventually it rolled out to being used in every single one of our first year accomplishing courses in addition to several of our 200 level literature courses. And so, but to understand that, you have to understand like the process. I think it's important to put in context. So I built our first ever online accomplishing class in 2014, let a team to do this. And then, and it worked great. It was like a learning commons model, but we used advanced instructors to teach. You want me to stop you? I'm gonna let it go. Rami, what do you want to do? Your call, man. You're the host. Oh, I don't know. Hey, I'm sure he's apologizing to tech difficulties. We actually just had a lovely little sidebar about all kinds of stuff. But given that you're back, hope everything is okay. I was just saying like how fickle everyone's internets are these days. So. That has never happened to me. I just cut off completely. It was like, oh, why don't you go? I think if it's okay, please don't apologize. Please apologize. Justin was literally just getting started. And I think for the sake of kind of narrative flow, let's just pick up where you were and then we'll kind of go back and hey folks, thanks for hanging with us. But you know what, such as life when we're bringing folks together across the country here in this space, making things happen. So, it's all good. Take it away. We're here. Almost at the end. Talking about the invitation. Sounds like we may be having a slight bit of delay on your end still. Just a slight. We're having just a little bit of a bandwidth issue perhaps. I guess you all can hear me. We can hear you. It's just broken up a lot, Trice. Maybe stop video for audio. Are you talking to me or her? I think she can hear us, man. I can see, you can just see that her wifi signal, her strength is really short. So, interesting. All right, all right. Well, unfortunately we're juggling some tech issues. So, Justin let's take along and then hopefully we'll just kind of pick up and move through as best we can. Yeah, hold on, it's letting me toggle my things back on. And there it goes, perfect. Okay, so let me see where I was at. How we get here, right? So, we had a successful online model that worked primarily for teaching with advanced instructors. Then right before the pandemic, the fall before the pandemic, the college asked us to try and scale it. We were only offering like six sections, eight sections a semester and they wanted to offer more closer like 15 to 20. But to do that, we had to redesign the course to make it work for people who were less advanced and particularly maybe had no experience teaching online or little experience teaching online. And so, as we started that process, there were two things we wanted to fix from the previous model. We wanted to improve. The previous model was fantastic and scored off the charts in our course evals, especially for online, as far as online learning goes. And our drop rate was only like 10%, which is way below the national average. But we had some things we wanted to fix. And one of those was, we were really good at creating meaningful feedback loops between instructors and students, but our peer-to-peer engagement wasn't quite as strong as it could be. And as a writing class, building community and having that kind of student-to-student connection and that shared modeling of inquiry is really kind of essential to what we want to do. So we knew we had to fix that. And two, we wanted to try and improve the way students engage with their course readings. And so, I reached out to Jeremy Dean at Hypothesis because I had seen some work on social annotation and knew their tool a little bit. Because I also, the folks who I knew, had said, you know, this is really good at doing these two things. So we tried to find a way to bring it into our new model to situate it in the context. And luckily we did this before the pandemic hit. And so then as the pandemic hit, our entire model shifted, where now we had to redesign our online course for every kind of instructor. Not only people new to teaching online, but people new to teaching. I never taught before. So we needed a lot more scaffolding, a lot more like, you know, sort of structural elements. And we built a model that would work initially. We intentionally built a model that would work both completely online, or in the classroom, or in hybrid modalities. Because we decided early in the pandemic, probably by the end of March, that we were just gonna go fully online with our writing program the next year. But if we had to flip it back and forth, we wanted that flexibility. So the long-winded version of this is, or the short version of this is our online model that my team was building, which was centralized around, including hypothesis, became the model for all 55 to 65 sections of 131 every semester. And as a result of that, we quickly realized that we were gonna have just an enormous amount of data available to us because of the role that I occupied as a program, sort of supervisor, and the fact that I'm the one pushing out the course models and we partnered with the folks in hypothesis to see if we could get access to the social annotation data. And that's when I met Raimi and we've kind of gone uphill and downhill ever since then, right? So as a fun weaving process. And so some of you may be familiar with this, but we've had this conversation a couple of times at different events. But it's important to know that context because our course design was intentional around improving student to student engagement and improving student engagement with text. And it just so happens that we have five core course readings a semester that students can now get to annotate and engage with one another. And so that's sort of the premise. We've been collecting data officially for three semesters running this as a program for four and next year will be our third year if you go through this cycle. So yeah, four semesters and data work so far. And it's taken us just about this long to get to a point where we can get into making inquiries of the data because it's just so much and to give you some perspectives here a little bit of how this works. And then I'll click on this. You can see a little bit better. Basically every semester we run 50 to 65 sections of W131. There are on average between 1100 and 1500 students per semester. And then in each one of these 131 courses these first year composition courses there are five core readings, five core core course readings that students have to annotate. And each of those students have to annotate and they have to have five annotations per reading of those five, three are additive and two are responsive or they make three original posts and then they reply or engage in conversation on two other posts. And to the question in the chat hypothesis is used through our Canvas LMS. And it's just part of our blueprint model that I have our course show and I push it out to all the courses and then they work with it from there. And so we started gathering data in spring 21. We have 21 data in place we've been working with. We have collected the fall 21 data that we haven't worked with it yet and we are in the process of collecting the spring 22 data but each iteration gets worse as it were in terms of the number of things we're trying to deal with. But to give you some sense we collect not only the social annotation data from the course but we pull out as much data from Canvas as we can that's relative to assessing student performance in the writing program and it's particularly in relationship to the social annotation activity. So we collect the course writing assignments. We collect small activities. We can collect the feedback loops from instructors and the comments they provide. We collect grades and individual assignments. We collect outputs on the course outcomes to try and figure out is there any correlation and what kind of correlations can we determine? And then hopefully one day move further into looking at what role does the annotation activity play in impacting student level of writing. That's sort of one goal. We also want to look closely at how social annotation the social part is being enacted. And so this is our kind of data set if you think about spring 21. We collected data from a series of literature courses as well that we call them the L20Xs and it's like introduction to poetry, introduction to fiction. But for purposes of what I'm sharing today we've separated them out because they have a different curriculum and they annotate extremely in higher volumes than we do and it's not consistently the same curricula from course to course. It is one instructor's L204 is a little bit different than the other instructors L204. So to give you some sense here we got 51 course sections in our spring 21 data that had 163 or 1,063 students enrolled in those courses of those 976 annotated at least one text. And that was across 79 unique texts across the different courses. Further within that, if you're curious at least 23 of the courses, there were more than three texts or three texts shared in at least 23 of the courses and four courses had the same set of readings in more than 10 sections. So it's a force readings at the same involvement in sections across 10 different classes. And so there's just an incredible layering of information that we're trying to process because these readings come in different times in different places. And so that's the scope, that's the scale of what we're talking about today. And I'm happy to go into a little more detail about the actual content itself but I wanted to give you the frame before we hop into maybe some of things what we're learning lately. And so we spent our first couple of semesters looking basically at descriptive data. And more recently I've gotten into now trying to make sense of that descriptive data so that we can start to make inquiries of the text and sub inquiries of the text. So real quick, I see Madeline's question here. Is there a word limit in the discussion boards? We do not have a word limit, a word limit. We do have a word minimum usually. No, we don't. In some cases we prompt for longer writing engagements but we try to eliminate, yes, that's great comment, Raimi. Or we try to eliminate those kind of empty ones but it fluctuates a little bit further for us depending on the course and the concepts that are being introduced. So what we come up with is to try to give how do the annotations work across all of these 131 sections and across this semester. And as you can see there are clearly five course readings during the semester and they fall around these particular dates on the calendar. And what you're looking at here is the blue lines are the number of original posts on a daily sort of count and the orange line is the number of reply posts on a daily count across our entire 51 sections of spring 21 data. And so you can see the windows and when the course is being, the reading is being discussed. I'm just gonna move through a little bit so you can see. And one of the things we thought would be fascinating to figure out is, is there a way to predict the kind or look at and understand the kind of timeline or time to completion that students are engaged in these courses with these activities? In theory, we assign the course readings on let's say a Friday. They have to add annotated readings by Monday and then they have to relate to reply to students by Tuesday and then they discuss it in class on Wednesday along with that kind of cycle. That's in theory, that's how it's supposed to work. But it doesn't always come out as cleanly as you think because you can't reply until somebody's annotated and it kind of goes back and forth. And so here what you're looking at here is basically the breakdown per day from total of posts with the zero day being the first time somebody annotates the text in the course. So the first time somebody annotates or the first time somebody replies is our zero day on this count. And you can see obviously between days zero and two the vast majority of the annotations take place. So students are quickly coming in, marking up the text and then leaving. As the sprawls out a little further, we can see the replies are spread a little bit more. And so this tells us a few things in terms of the general engagement. The majority of our classes are coming in doing their three annotations, waiting, coming back, doing their two replies and being done. And it's not the sustained kind of in and out of the text that maybe we had hoped for but at least it gives us some sense to start. And so the peaks and valleys here are the numbers are the total number of posts on a given set for the entire dataset not just for one class by any means. That'd be a lot of annotation, 6,000 on that one point. And so we start thinking about, well, this is clearly a product of our course design and the pedagogical implementation. We have checkpoints and due dates and things like that. And so we wonder now going forward, can we tweak the model? Maybe require the annotations to exist throughout the week and whatnot. But those are the kind of decisions we can make because of the data that we're gathering. Most recently we've been able to really start to get into some of the finer details about the annotations themselves and starting to identify the kind of threads that are at play. So again, this is just a broad scale overview. We had 17,608 total annotations from spring 21 in the W131 courses. 11,000 of those or 11,657 were annotations that had no replies to them whatsoever, right? They are the, they're a post and nobody touched them or engaged in them in any sort of content. But there were a total of 5,879 replies, what we call level one replies. So Raimi made a post and I replied to Raimi, perfect. And then the Shreese pops back in and she replied to me, that would be a level one repost but if she replied to Raimi's post, that'd be a level two, right? So you can see how these kind of move on down the line. So we had roughly 5,800 or 5,900 reply annotations across all of our data and then only had 72 reply annotations that were level two. So only out of all this data, only 72 times did a student reply to a reply to a post, which is to that point that's an interesting kind of discussion when you think about how this works across them. So we're not, we're not getting sustained discussion. We're getting you post, I respond, you post, I respond. And that's again, not what we're going for. But of these 5,879 plus the 72, right? Of those total annotations, they, those total annotations are comprised within 4,188 threads. So quite a few of these level one annotations were in fact, my original post, Raimi replying to me and then Shreese also replying to me and then somebody else also replying to me instead of having a conversation it's been three, three engagements with the top level post. And so this, we're really kind of excited now that we can see this, not only identify where the threads live but track their total volume and track them across courses and documents and things of that nature. And so this is sort of the most recent development on our end. And unfortunately it's just, it's taken forever to figure out a position of data to get it all into the same document, to get it scrubbed and cleaned so we can de-identify it in ways that can be used. And it's just way more data than what we can actually process. Hopefully one day we can make this more available to other researchers to make inquiries of our data as well. But we've been using it as a starting point to identify subsets to start making more targeted inquiries into. And so you can see here, and I'm not gonna, I'll just see if I can tap in so you can zoom in and see it. So this is a breakdown on each individual course and these are just dummy variables that don't actually correspond to the actual, I mean, to the actual course number, but it's, so in this one course, right, we had 535 threads for the semester. And however many of those are actually a discussion we don't know, it's mostly those level one sort of replies. And you can see that just as you go it just continues to, this is obviously scaling down. So 535 is our top number of threads in one class. Then we go down here and we have one class, another class in line 57. Part of this is due to enrollment fluctuations. Not every class has the full 23 students per section. And some of it is that students just don't actually participate in the assignment. There were lots of times where students were annotating and not replying and vice versa. And so we have that kind of stuff. And then last I'll show you this even though the numbers aren't there, a sort of breakdown of the post and replies per course, per document. And so we can look at like DeLoria here, this text. These are the total number reply posts and replies that the numbers didn't populate for some reason. I have to ask about that. And so this again, it's just another way for us to start to filter these things out. It's not, you know, not necessarily shattering in terms of like data findings, but it is absolutely essential for us in terms of where do we go next? A couple of graphs that I don't have here and then I'll come back to later that I can share with you is we've been able to identify, you know, which courses in the 131 class have read which texts and in which order. So there's five different course texts throughout the semester. The fourth one is what we call our Keystone essay and students really build off and around that really dense content. And then, you know, the question is can we identify a number of classes that have already the same document? And that fourth slot, and we have several. We have like, I think that is it the Anzaldur reading that's read like the McCrary reading, one of our course readings from McCrary is read in nine different classes at the exact same spot in the class. So they have different themes because the instructors choose across a bunch of readings in our archives. But now we can look at those nine sections and look at not only the, specifically the fourth reading and all the annotations and look at the different ways in which those students are actively engaged in that reading at that point in the semester and try to make comparisons in a way that might be helpful, might be useful. Again, it's like, I really wish I had like, oh, here's the brilliance of annotation here's how to impact students writing and that's where I want to go. Or here's the fascinating part about student threads and conversations and here's what it means. But as of right now, what we were able to do is identify how do we start to ask those questions and create those data subsets to then get into content analysis using computational modeling, topic modeling, those kinds of things. And so that's the basis. So Chris asked, Chris, what tools are you using to pull the data out of your course to analyze it? So we have a data wizard, as I like to say. So it's important to know that I partner with hypothesis but I also partner with the e-learning research lab at IU to help us provide services to help, you know, de-identify the data to merge the data sets together in ways that make sense. And de-identification is really problematic because we have to pull students' names out and some names appear in course text and course reading, so there's a whole bunch going on that. From that data though, we've moved into, now working with, I jokingly called him the data wizard, a Canon vendor, an undergrad who is volunteering his time to work with us just to learn how to do this stuff. He's been using Tableau and customizing sort of scripted elements to sort through and filter through the data and create a lot of these visualizations. The last two, the long flowing ones are out of Tableau. And that's a Tableau visualization itself where the earlier ones that I shared are ones that I created based on the data we have, we found in that. So that's sort of for that process. And we're trying to do other things like creating hotspots in the text, mapping out how to extract the data into subsets. Right now it's just a lot of data collection, data curation and building a system for us to be able to manage the inquiry. Like it's not, you know, sort of fascinating to think about it as a process because it's, you know, I hoped that it was like, well, look, we got two texts, right? I mean, let's read them and talk about them. But we do not have two texts. We have 79 different unique texts across 51 sections, all doing sort of thematic things that are related and sometimes unrelated, all around the same set of skills that are supposed to be taught in sequencing order. And so, and then of course, each semester we've had curriculum changes based on modality. So this data comes completely from an online dataset. Everything was online. The next set of data, the fall 21, has got online and in-person and hybrid modalities. And it's got a slight curriculum change with the way that we ask students to engage in the hypothesis activity as is the spring. And then in the fall, we'll have a different set of curriculum elements that have evolved around this kind of engagement. So that's like the 20,000 foot view. It's like 10 minutes. I don't want to go any longer. I want to do a few last questions. Yeah, thanks. And there's a lot of questions coming in about the specifics of the study that I want to get to. But I want to just give a quick update on again, dear colleague, Sharice McBride, who's entire area is experiencing an internet outage right now. So there's just no, everyone's offline. So we really... So it's just me today. You got me. Well, we really appreciate her presence. She's actually shared with me her slide deck as well as all of her typed up commentary that she was presenting. And I want to actually, if it's okay, go back to that because in organizing today's session, although again, now it is just me adjusting. It's like our research team meetings. Now we're just kind of hanging back out again for a moment. The reason why I wanted to bring us together is that the two approaches to a researching annotation that are represented by both Dr. McBride's work and Dr. Hodgson's work are really different and both really important. And I want to make sure that we don't lose sight of the advantages of those two different approaches. And so let me just kind of hit a few high level notes and then quickly jump back into her presentation. And then we can kind of get into some of the more specific questions about study design, et cetera. But as we heard from Dr. McBride, she's primarily studying K-12 educators. And I think it's really important to consider and remember that if we're asking students to use technologies, it's really important that we ourselves as educators also know what it feels like and what it looks like to use those technologies as well. And so supporting educators as learners and their familiarity as she was saying with these kind of new digital literacies is extremely important. And that of course contrasts with what Dr. Hodgson's presented to us, which is looking at how in this case, what we might call pretty traditional undergraduate students are using this class, or excuse me, are using social annotation in a required university class. And so an intentional pairing of two very different groups of learners who are engaging in annotation activities for very different reasons. And I think that's an important thing to keep in mind as we do the work. As we were just hearing from Dr. Hodgson's presentation, one of the things that makes our work at IU both very exciting and extremely hard to manage actually is the scale. We know the literature and the landscape pretty well. There isn't to date a ongoing research study about annotation as large as what we're trying to bite off here. We were just hearing about our first semester of data collection, 50 some odd sections of freshman comp, thousands and thousands and thousands of annotations. And we have data now from two additional semesters. Our data sets will be enormous. And to then briefly address one point of the chat that I see a question from your colleague Mo here, there is much, many other sources of data in this study as well. There is survey data based upon reliable survey measure. There are interviews and questionnaires with instructors. There is artifact collection. There's collection of original essays. There's just so much data in that project. And that's a very intentional in terms of this presentation contrast with the work that again, Dr. McBride is doing, which is extremely focused on a very really small scale voluntary group of participants who see the value of developing their professional practice through annotation activities. And so as we think again, and we step back and understand the value of researching social annotation, one of the key things that I wanna make sure that we don't lose sight of in this presentation, whatever various, for if you're a participant watching now or watching later, whomever you may be and whatever your interests are, there's great empirical value in looking at a handful of educators who are reading a text together, who are making meaning of it, who are in this case of Dr. McBride's research, reading texts written by youth, building empathy, considering perspective taking, thinking about the relevance to their pedagogy and using youth-generated media and collaborative annotation practices as a means to deepen their professional pedagogical practices. That is an incredibly important locus of empirical investigation, just as it's important to try and glean patterns from massively huge data sets, like again, what we're working with in the context of the IU study. And so the difference is in population, the difference is in scale, the difference is in the kinds of readings that folks are looking at, some of the reasons why I wanted to bring, you know, our future speakers into conversation with one another today. So I just wanna make sure that we kind of hit on those high-level comments before I turn back into some additional Q and A. So there is a question, and so Justin will turn back to you for just a moment here. There is a question in the Q and A from Courtney Gomez, Courtney's asking. I may have missed this. Yeah, oh, you did in the chat. Okay, group of students. I was just trying to, yeah. So we- Do you wanna talk about that? So our course, yeah, yeah, no. I think it's a really great question. Our first year comp courses are typically capped at 23 different versions have different caps, but 23 students, but not all the classes ran at a full 23 scale. So I think the lowest was 12 or 14 students in the spring, some data set. But yeah, they were actively annotating across the entirety of the class. We didn't, at that iteration, we didn't have groups as part of the campus install with hypothesis yet, but it was meant to be a full class discussion and as they come into those kind of conversations. So that was, I mean, the big one for us was how do we get them to talk to each other? And now the question, right, Raymond, we ask is, do the student X only reply to student Y throughout the semester or is there actually a kind of layered discussion which we can't, we have yet to be able to determine those kind of predictive outputs. So I know we have access to some of that content. So we do. And I think that raises another point. I guess I'm kind of like, and again, this is where it's nice to invite friends to talk about the research because I have some sense of what Dr. McBride is doing and researching and her work. This is another really important tension that I think is worth highlighting in this discussion, again, broadly about social annotation research, which is that if we're asking people to use social annotation as a way of having a conversation, and in this case, we're specifically using them, you know, asking them to use hypothesis, what's the motivation to have a conversation? Right? You know, what gets people talking to one another? And so, you know, just in the case of the work that you and I are doing together, we have a explicit instructional request that students author three individual annotations and then they reply to their peers twice. And you gave us some of that data where we're beginning to see those replies come in and we can then look qualitatively at the characteristics of those conversations. We can, for example, see if students are constructing knowledge together, if they're actually engaging with diverse perspectives, if they're listening through their annotations. But we can't ignore the fact that there is an explicit instructional requirement. And that varies. And I'm gonna see if I can share my screen here. And it looks like I've got to do a little bit of finagling on my end as well. And actually, it looks like Cherise has kind of come back in and join us here. So I'll let her do this. But it varies from the context that she was presenting to us here, which is to say, we have a small group of educators and they are choosing to pick up this particularly interactive and intuitive social annotation technology because they see it's valuable to their own learning. And so the characteristics of their conversation will be markedly different as they use hypothesis to advance their learning than, in some cases, freshman IU. And so I think we need to be appreciative of the differences in the kinds of social contexts, the kinds of explicit requests or requirements for annotation when we then go to make claims about how people use this technology and how they learn. Well, you know, in our own context, right? So the data we're looking at has a very explicit, like please do this many of this kind. And we usually have question prompts to figure out, to guide them into the kind of inquiry because we wanted to help them understand different ways of engaging the text. That's the goal, right? The primary goal. The secondary goal is this peer-to-peer interaction. But then in the fall, right? The next semester, that model, the first model was built around like your five points for completing the task. This model now has rolled into a different kind of incentivization, right? So it's still point-based, but it's not explicit. And so now it's a matter of, it's part of a larger ecology in which students are operating in for the newer design. So I'm fascinated to see if we get more interaction in the next set of data because the parameters were not as like explicitly instructional. It became like, you need to do these, they still need to do five annotations, but the process and the order and the one-to-one sort of point correlation is no longer evident. So I think that as you just change something simple like that, like the operative dynamics of the inquiry is gonna shift the way the behavior works. At least that's my thought. All right, that's my hypothesis, as they would say, right? So it's really fascinating in the context and how the behaviors are just radically different depending on where you're at, what you're doing, what your situation is. Yeah, so I wanna actually pick up on, I wanna bring, I think that Sharice may be rejoining us if she's literally, I think, driving or commuting somehow to a place that's not blacked out by internet right now. So I hope she rejoices in a moment here. But I actually wanna bring a quote of hers back into the conversation to share with you, Justin, because I think this is really fascinating in your context. And again, I think this is such, there's just such great value in saying, hey, there's this thing called social annotation. We know from research, quote unquote, whatever that looks like, however the methods we use, it's valuable, productive, but the social context varies so widely. So here's something that she said when she was presenting to us earlier. She said, the author is a function that is open. And my sense, again, from having done work with her is that educators, particularly literacies educators, both K-12 higher ed, certainly have a clear sense of that, that the author is not always the authority and that the author's perspectives and claims can be contested and that something like commentary and specifically hypothesis annotation is a way of making that function open. And I'm wondering, Justin, as you hear that and you think about that, what does that look like for an 18, 19, even 20 year old student at IU? Do we see in the work that you're doing? And I know that the focus of your presentation day was primarily on the 131 class, which is a freshman level class. Although we've also collected data and we've also been doing some smaller scale collection and analysis of work from higher level courses within the English department that are more specific like poetry and et cetera. Do we have a sense that students are also embracing the idea that the author as a function is open, that they can speak back to the author as a way of catalyzing their learning? Yeah, that's a good question. But the question I would say, do you mean pedagogically or are we, like for example, it's hard to teach a writing class and when you have students do readings, critical readings that they're gonna build from and not also include things like making inquiry of the author. The author is not the definitive source. The author is a perspective from which we operate and how do you position those thinking? How are they positioning their thinking? So the rhetorical elements of, and the rhetorical function of the author is sort of front and center and a lot of the kind of activities that our students are asked to do because the class is heavily built in analysis, right? That's the way that, again, it's not my curriculum, it's just I built it up for the online delivery, right? So for good or bad, the five texts all primarily revolve around critical analysis in some capacity. So I think that on the one hand is a fundamental drive of the class. But what I do think we've seen in some of the, and again, there's some data that we can share as part of the research we've put out, but then there's also things that I'm exposed to as a program, as a programmatic level of review. And so what I do know is true across a lot of our course evals when we analyze those for programmatic purposes, there's an incredible layer of enthusiasm for hypothesis as a tool, right? And students comment on things like how much easier it was to make an inquiry of the text or how much easier it was to try and find ways to challenge ideas in the text and their commentary on it. Again, it's not part of our survey analysis, it's part of that kind of feedback. And some instructors would add their own questions to the course evals, like, how did you like this tool? Because some instructors didn't like it initially. And so they were trying to find a way to get it out of the classroom. Since then though, we've adopted it even further at scale because people have been so excited. But so I don't know if that's like, it's not evident in the content we've analyzed yet but sort of the things that I've seen as a programmatic level of assessment and then just the principles we teach, it's kind of hard to avoid the author function as a part of that inquiry. And I think the tool itself, just the act of annotating in a collective allows you to discuss and question and or push ideas from an author in a way that doesn't feel as isolated as when you're just doing it at home by yourself in the margins of a book, right? Absolutely, absolutely. And teaching those key dispositions is so important. Absolutely. So I might guess I'm also really curious. When did they get there or not? It's another story. Well, no, this opens up the complexity, Justin, because you and I and our team more broadly have had a lot of conversations as we dig deeper from, again, our extensive data collection faces into our analysis because we understand that there are learning environments where the formality of the environment is such that we have these broad instructional goals, right? We want students to learn to engage with discipline-specific practices. We want them to take away kind of key insights, whatever that may be. And so we have those in mind as we design pedagogy curriculum and of course use learning technologies, maybe like hypothesis to pursue those goals. And at the same time, we know that a tool like hypothesis, and again, annotation more broadly, is kind of open-ended, right? We're kind of asking students to kind of grow out there and make meaning. And so there's inevitably a little bit of tension there between how they show up in the texts and how that aligns with our goals. And then of course we can choose to research how that fits. One of my favorite examples in there, I don't remember this early on in the data, there's a class where somebody was at, the students were responding to one of the early texts and it was about synthesis, synthesizing, or comparative analysis, that's what it was. And the text was, I think it was one of Gloria Anzalduo's piece on Spanglish and coming to language. And all of a sudden in the middle of the text, in the middle of the annotation is a link to, or is an embedded advertisement for like hot dogs, right? But it's not like a real company. It's like the student made a graphic in a program and then just put it into the comments. So like the open-endedness also invites kind of fascinating other issues. I like the multimedia, but like we were joking because it had no, there was no contextual reason for why this was there. And so like those kind of things are always fascinating to me too. It's like what are you doing? Always fascinating. Well, I think we may have Dr. McBride back with us. And again, thank you so much. I hope you're here for hanging with us. So sorry that there were so many kind of tech challenges. At a cafe that has internet. Oh my goodness, hey. I am so embarrassed. I'm really sorry. Not your fault. Not, it's where you're here. Nice, you're sorry. Let me tell you, I just actually took one of your quotes from your presentation earlier and actually asked Justin about this, about kind of how the author as a function opens up texts. And Justin was riffing a little bit about how students in his context, you know, do or don't kind of pick up on that critical aspect. And I recognize that again, because of the, you know, internet snafus there that you didn't have a full chance to see, you know, all of Dr. Hodgson's presentation. But I wanted to bring you back into it. And I want to kind of tee you up with a question that I know you care a lot about and that speaks, you know, greatly to your work, you know, which is returning to the idea that your work with annotation and the importance of annotation to educator learning is happening in a kind of less formal learning environment. And yet there's great rich insight that we can glean about how people learn outside of the kind of formal constraints of the classroom. And I would just love to hear you speak a little bit more about the fact that there's a lot to learn about annotation and learning when we embrace these types of everyday reading and writing practices. And when we see in the case of your research, educators engaging deeply with topics of interest, topics, you know, of interest that are written by youth. And can you just speak to like why it's important to really advance research about annotation and learning that is connected to the classroom but is not necessarily always happening inside the classroom? What does that do for people's engagement and people's annotation in their learning? Yeah, thanks for asking that. I mean, as I was saying, I think that teaching and teacher learning is often really social, collaborative, public in some ways. And the opportunity to dig into text with other people can be powerful because I think that's a practice that teachers often do on their own anyway. Teachers are excellent, you know, at seeking out resources and putting things together in sort of a brookalage of what they're doing. But it's so much more, I think it can be really generative when folks are able to do that together and to do it in a way where there's sort of a transcript or a repository of resources that they can then refer to and what they've created together. In terms of creating curriculum too, teachers are able, that's part of our goal is that as they do these annotations and make their thinking visible, they eventually build curriculum that can be shared with other teachers and then that curriculum can be remixed in ways that are always situated, you know, to the context that folks who do that work in ways that they desire. And so I think those elements of it being social and situated are really made, really there are affordances within social annotation that allow us to build on that rather than a sort of, you know, top-down mandating of what teachers do. And this is an out-of-school space. I mean, teachers sign up voluntarily. I'm always, I just feel honored that they're willing to listen to these ideas and then share their feedback and kind of engage in conversation with us because they don't have to, right? But the fact that they've opted in to these experiences, I think also builds, we're able to build from that, you know, and engage them in ways they want to be engaged. Yeah, yeah, that's brilliant, that's brilliant. Well, so now that you're back, Sharice, and again, oh my goodness, thank you for like schlepping and finding internet despite the outage, heroic. Thank you so much. I wanna, I had this question in my back pocket and now that we're all here, I really wanna ask this to the both of you because I think that you both know that I am increasingly interested in the relationship between annotation and critical literacies, broadly defined, critical perspectives on text, critical engagement with peers, maybe critique, again, of authority. And it strikes me that it's entirely feasible and we maybe see this in some formal course contexts, that social annotation becomes pretty similar to a discussion forum, right? That maybe the discussion isn't really deep or the kinds of ways that we ask students to pick up this technology are not particularly innovative even though we're using these really interesting technologies. And yet I have a hunch, and I'm increasingly, again, interested in exploring how because we're using an annotation technology and we're using it in a social way and we're using it in a way that allows us to speak to the text but also to our peers, and it allows us to move into these kinds of critical, discursive spaces in more substantive ways. And of course, Sharice and the work that you and I have been doing, we've given that a name, this justice-oriented media literacy. We've begun to look at that more explicitly but I also know, Justin, in the work that you and I are doing that we talk a lot about the fact that these are freshmen showing up at Indiana University. Maybe they don't know who Gloria Anzaldua is. Maybe they've never read a text by Roxanne Gaye before and now all of a sudden, they're not only engaging with this content, they're doing so with their peers using social annotation and we have data on that. We can look at their conversations engaging with one other for the first time. And I'd love to hear you both talk about how you perceive the ways in which social annotation opens up new more critical types of conversation and inquiry as relevant either to freshmen or to educators. Sharice has been gone a long time so she gets to go first. Sure, I was actually looking back at some of our annotations to see if I could pull up an example. We also use crowd layers. I heard you talking about that, Justin, just to kind of see, and I shared this with the teachers as a tool they might use in their practice. I'm not sure if folks or anyone has used it among the groups that I've worked with yet but I think what's interesting, and I don't know if you got to see the part of my presentation where I talk about the kinds of annotation that I am seeing teachers do which include commenting on youth as authors themselves, commenting on the content that youth have brought up and then commenting on their own teaching practices and how they might implement these ideas. Those are just kind of three big buckets that I'm noticing so far. And I think all of those spaces allow room for critique or engagement in a way that reads against the grain. For example, in the teaching area, some teachers will, we actually read an article from the marginal syllabus and I saw comments in the margins around how this is critiquing the practice that they saw happening there. That I see a note here from one saying this is not a best practice, it doesn't align with the sort of dynamic collaborative classroom that I've been taught we should be cultivating. And this is a relatively new teacher I remember and someone else, another teacher replied, agreed although sometimes it is necessary. So I think there's a conversation beginning there that's critical of practice and can be useful for teachers across spectrum, the spectrum of experience but also we see even the critique of authorship and who broke this. I mean, those things definitely come up in the margins. And yeah, I think there's a lot of opportunity across those areas to engage critical literacy, critical literacy, critical pedagogy. I'm thinking twice again, making thinking visible, thinking twice about our practice and opening that space to discuss. Yeah, sorry, let me turn my mic back on. I think you're absolutely right. There's a lot that, you know, this, I would say that the tool alone does not foster critical literacy, right? No tool on its own does the work. And as much as I would like to think that just adding hypothesis made for a more critical engaged sort of reading or awareness of these things, you know, nothing so far suggests that on its own is doing that. But the opportunity that comes with being able to engage with an author or an idea or your peers or in this case, teachers looking at something else as an assessment of somebody's work, right? And to anchor that into the text and to do that in real time, that's not just your, again, not just your own margin, marginalia, but sort of public facing thing where you think about your comments as crafted as real, even with the first draft thinking, there's still a real kind of thing. You know, that it invites the opportunity in a way far, far better than just saying, hey, go read this and come back to class and have a discussion, like at least for our first year program. Because it's just reading in isolation is not at all how one becomes aware of otherness of difference of margin and marginalization, right? Like, I mean, I think you can get exposed to a lot of ideas and become more aware, I guess I should say, you probably, you know, that's true, but the conversations that happen around something like Anzaldu is breaking down of the different levels of mestizo and how she talks about language and her coming to language. And then to see the students trying to engage in that, you know, whether they're like a, you know, poor rule kid from, you know, the hick parts of Indiana, or like I was a small town, small time kid, or whether they're from a, you know, different, different, different race, different ethnicity, whatever, that doesn't matter that the space is available for a kind of inquiry and opportunity. So what then we have to figure out, and this is where we struggled, I think in the first go round, is better ways of fostering that kind of engagement. And we've actually seen it done incredibly well in our literature courses because the instructors, they had more autonomy in how they scaffolded their activities, but also were actively invested in the role the annotation was playing. And so I think at the freshman level, because of our scale, what we accomplished was actually quite good for what we wanted, but it didn't hit this kind of next level engagement that requires that, you know, I always say teaching is it's people first, right? And it's pedagogy focused and purpose driven. So the tool doesn't matter, but if the tool creates a new opportunity for the pedagogical focus and the purpose, then we really open up the possibilities. And so that's, for me, I know Raymond, we've talked about this a little bit, hopefully in future iterations, we get much better at this. And we can look at like, you know, five or six different sections of students from Indiana, historically not, you know, historically white sort of oriented state, you know, it's got a sorted history for a lot of reasons, at least in the Bloomington campus in the southern part of the areas, and see how they are actively engaging with something like Anzalto or Roxane Gay, or these, what is commonly thrown under the critical race theory framework or umbrella, whatever that is. And to see what we can figure out from that, but it's still starts with, what is the pedagogy that's helping them get there? Or who is the teacher? And how are they prompting the conversation? So it's not the same kind of analysis of their work, but it's the opportunity to push them in that way. Does that make sense? I know it's sort of a broad spectrum answer, but it's sort of where we fall because the tool on its own didn't, it did not have the impact that we thought it was gonna have. It had an impact, absolutely. And we don't know how far it's gone in terms of their writing, but that kind of critical reading, we saw a bunch of students pick up with it in really wonderful ways to come to class and be upset that we already had this conversation. Why are we doing this again with the instructor? And then others who were like, well, I did my five, leave me alone. So it's just too big of a box to fit in right now. And I think that's probably probably some of my question here. And first of all, thank you both for those, really for me, I'm still taking notes for provocative answers. And I think that it's almost too easy to try and research whether or not a particular learning technology has some measurable gain, outcome, something. Maybe it works, maybe it doesn't. And I'm just gonna say, that kind of research starts to bore me pretty quickly. It tends to reify certain ways in which we hold to Dr. McBride, to your points earlier, about who's knowledge and authority and voice matters in these particular learning environments. And so what I find again, particularly compelling about how both of your works open up new avenues for research, we, for example, see an alternative approach to what it means to design professional development in the case of your work, Dr. McBride, right? And like, how do we get educators reading, discussing, and learning from youth experience? You know what can help us? Social annotation, it provides that initial entry point to have then those rich conversations that are relevant to their pedagogy. And then to the point just making it for moments ago, you know, into the context of the IU work, you know, Justin. It's this question of, do we see students having new kinds of conversations in new ways because there has been an intentional approach to the ways in which students are encountering perspectives that they've not encountered before. And if that's the case, maybe it doesn't matter that this, you know, technology does or does not help raise their grade by two or three percentage points. Maybe it's a opportunity for them to become more critically engaged thinkers, discussants, you know, and peers, you know, for their classmates. So in any case, I find both of your, you know, studies in this respect so helpful. So please, please, please. I would just like to say a brief thing which about the criticality and where it comes in for me, I think in the design of how I'm imagining teachers coming in with these questions and how we shape the annotation, we explicitly ask questions around what do the artifacts that we're looking at reveal about youth strengths, for example. We're starting from a place of asset, you know, an asset orientation in that in itself is part of the critical element for me in making sure that our teachers, I mean, you hear so many deficit framings of the youth that we're working with, youth of color, queer youth, youth who come from backgrounds of poverty, you hear what they don't have, what they're not bringing, what they're not able to do. I mean, I don't even, like, it's not, I do my best to start from a different place to say we're gonna share thoughts about what the richness is, the abundance that we see in this work. And we're gonna also work from the assumption that they are offering critical thought, powerful commentary. And so I ask those questions, like what issues, values and perspectives are they offering commentary on? Widening our apertures to see what learning is already happening. And rather than immediately trying to map what they're doing onto what, like standards that we may have, just asking open questions and then connecting it to our practice. So I think that's the critical element for me, given the history of deficit framings of education and they happen in education of particular groups of youth. Thank you. That's right. Thank you for bringing us back to this. I really appreciate this. So thank you. We did get a question from Chris Aldrich. Good to have you Chris. And I will admit that I'm not perhaps particularly familiar with the context of the question, but it is in the Q and A asking how about we better encourage tumbling. And again, I'm not as familiar with this particular references as I could be, but here's at least it sparks in my mind having just very quickly looked at that reference, which I think that both of our guests can speak to, which I'd be really curious about what you're seeing in your various projects and how in your research, you're seeing evidence that annotation allows learners, again, whether educators or students to write for different kinds of audiences and to perhaps write in more private spaces for their peers, but to also use that annotation writing to support other forms of discourse, whether it's again, their conversations with colleagues about their practice, or in the case of maybe their own writing for essays or for assignments and how we're seeing the ways in which annotation allows for multiple kinds of writing beyond just the immediate reader response to a text. Yeah, so I don't, I mean, again, I have to dive into the tumbling thing in much more depth to probably offer a more substantive answer. But I know that something we've increasingly had conversations around are things like digital empathy, digital ethics, digital engagement as a framework across IU. And I think, you know, there is a kind of an audience-centric nature to annotation, to social annotation, right? That is not present when it's just private annotation. And in that regard, in many ways, invites, it's a rhetorical space, right? You're posting something others are going to read, whether it's just for you or not, it doesn't matter, others are gonna read it. And so by design, and it typically leads to a more crafted thing, like it's, you know, I always think about the difference between like my photos on my phone versus my photos on Instagram. They've definitely been polished before they hit Instagram. And so there's a layer of that. But I think we've seen a little bit with students and I've seen students writing increasingly in attentiveness to this kind of willingness to put oneself out there, their ideas, their identities, their selfhood, like, you know, we always say we write from the personal. It doesn't matter what kind of writing it is, you can't get rid of yourself, you know? So there is some of that going on, I think is a part of the process, but I don't know to what extent we, you know, like as a intentional design thing, it's not in that conversation. But yeah, I just think it's, it has created an interesting conversation around like, what you always call first draft thinking, right? Maybe that's your, you know, thing. Are you willing to put your first draft thinking in front of your peers? You know, those are kind of challenges. Yeah, I think that's an especially risky move for teachers, right? And with whether it's issues of surveillance or even that peer kind of eye on us, what are we doing? Is it good enough? Like those kinds of questions and I can go back to, I mean, this is a question I want to keep thinking about what kind of writing does this afford or connect to, but I think just the conditions for writing are something that we need to pay attention to. We know that in social media spaces or in these spaces of public thinking, there's a lot of pressure and there's a need to, there's also this history of, you know, negativity and critique and unhelpful critique and hate speech and things like that that can happen. So another frame that I've thought about with colleagues Anna Smith and Chris Rogers is humanizing discourses of collegiality, particularly, and I think collegiality is not just in a teacher or professional sense, but also among students, like what are the moves that we can make with one another? For example, moves that encourage experimentation rather than a sense of correctness or having to do things the right way. I think emphasizing certain, emphasizing those as values and then also making mechanisms for another example, we had this spectra of like nurturing and welcoming rather than sort of a enclosed community. So there's these different continuum that we looked at to identify, you know, to categorize what we saw as these humanizing moves rather than what can sometimes happen in social media spaces that honestly turns a lot of people off from it. You know, in our pre-service teacher courses, I was just speaking with my colleague, Michelle Wilkerson in the curriculum and instruction, curriculum instruction technology course that we've taught in the past. Our pre-service teachers are actually really averse, you know, they don't want to use social media in their courses. And so it's a balancing act of like explaining, you know, and demonstrating some of the affordances and also demonstrating that we will take care to mitigate some of the risks, but it takes a lot of contextual like condition setting to make this work happen. And I imagine that's why some folks just kind of say, you know, I don't want to take the risk even as an educator, but I think it's important for issues of justice and having these hard conversations. Speaking of having amazing conversations and really struggling through some stuff, Sharice, Dr. McBride, thank you so much for hanging with us today and for sharing your wisdom and knowledge with us despite all the technical difficulties. Really appreciate it. Justin, Dr. Heisen, thank you as well. I really appreciate the two of you as friends and colleagues. I know that people who have technical skills, you know, are many and that there are again so many people who can easily implement learning technologies to do all kinds of things. But I learned from the both of you because you're constantly asking big, compelling, challenging questions about why we're doing this work and who we're doing this work with and how we design that work for consequential learning. Again, whether we're working with educators or with students in school, out of school, with a whole range of texts. And what makes my investment in the scholarship that we've done together and the projects that we do together so steadfast is the care that you bring to the work, your ethical commitments to really putting first our learners and their wellbeing. And I just so much appreciate your presence here today and your willingness to share your expertise with us. I know that this will be recorded for other folks to pick up in the future. We can circulate resources as well. I just want to thank everybody who showed up. We had pretty consistent participation and attendance throughout the entire session. So thanks to everyone who hung around. Again, my thanks to all of our hypothesis colleagues who behind the scenes have been making sure that all the tech has been working out as smoothly as it can. And you're all, of course, very welcome to follow up with all of our guests. It's easy to find Dr. McBride on social media. It's easy to find Dr. Hajj on social media. Their papers are out there. Their research is out there. Read their stuff, follow their stuff, follow these projects. They're really at the cutting edge of what's making critical and creative social annotation research possible. My thanks to you both. Really appreciate your contributions today. And thank you to, again, to everyone who was able to show up. Thanks everyone so much. Enjoy the rest of the social learning side.