 Hi, I'm Jeremy Dean. I'm the Director of Education at Hypothesis. And I'm just going to introduce our panel here a little bit and then a little bit. But first I just want to tell you about the Hypothesis tool, which is a web annotation tool. It lets you annotate web texts, HTML websites, PDFs, EPUBs, all kinds of digital texts. You can take private notes on these texts, make public comments, or form reading groups and have discussions in those reading groups with a group of folks that you've determined. We have Hypothesis users are everyday internet citizens, but we also have some verticals where we're seeing a lot of traction for this technology and journalism, for example, scholarly communication, but most of all in education. So the vast majority of our users are students in classrooms at the secondary and post-secondary level. Most of them are in the humanities, and most of our previous webinars about using Hypothesis in the classroom have been about humanities, about English, composition, history. And so I wanted to talk science today. And so I've got some science educators that have used or are using Hypothesis in the classroom. And I'll just say briefly before I turn it over to them to talk about their experiences and their projects, one of the cool things about Hypothesis as a tool, like I said, it's a general purpose tool. There are people using this all over the world in different industries, is that we have a broad range of scientific use. So we have the American Academy of Advancement of Science using Hypothesis-powered annotation tool to pre-populate articles from the journal Science for use in the classroom. So these are grad students breaking down professional publications in science to then get used in secondary and post-secondary classrooms to help students gain the literacy to understand professional scientific research publications. And then we also, at the other end of the spectrum, have climatologists in the Climate Feedback Group that are using Hypothesis in the public to annotate and grade the veracity of newspaper and magazine coverage of climate change. So professional scientists leveraging the tool to sort of become public intellectuals and intervene in the debates, I guess we have to say there are debates about climate change today. So really wide spectrum, professional scientists but also school kids at very young ages. And so, but today we have three science educators talking to us, or we hopefully we'll have three, maybe four, we're sort of waiting for one. But we have Brian Bennett, who is a biology teacher by train, has also taught chemistry and physics in public schools in Indiana and elsewhere. He's now an instructional coach, so he's been doing a lot of work around digital tools in the classroom and science education. And he's in the Elkhart Community Schools. And he'll go first, but I'll introduce everybody together. And then we have Craig Whippo, who's at Dickinson State North Dakota as a professor of biology, who's been using Hypothesis for about a year now on a web-based textbook with students up there and has some really great ideas for how this tool can be used in the science classroom. And then hopefully joining us will be Elba Serrano from New Mexico State University's professor, a regents professor of biology there. If she's not able to join us during the course of the webinar, Marion Martone, who's her good friend and collaborator, was very aware of the work she did with Hypothesis and will represent that. In any case, Marion is our scientist in residence, I guess, at Hypothesis, a hypothesis staffer, but also professor of neuroscience at University of California, San Diego, and she'll be sort of responding. Possibly impersonating Elba Serrano, definitely responding and leading discussion afterwards. So with that, I think I'm going to turn it over to Brian. Hey, thanks, Jeremy. Like Jeremy said, my name is Brian Bennett. I'm a biologist by training for a K-12 school system in Northern Indiana. And I came across Hypothesis maybe two years ago or 18 months ago, right around the time it was starting and started playing with it myself and annotating the web and having discussions around different articles, things like that. And then in my AP biology class last year, so I had 25, 10th through 12th graders and we were using the OpenStacks textbook out of Rice University. And for my AP class, I really wanted to push them into higher order thinking around the text itself, mainly because that content set by the College Board is just so vast and there's so much I had to cover is that reading was a heavy component of the course. And using an AP level text is written at a college level, especially the Rice University text was at a very high college level in some cases. And I was working with 10th graders who were 15, 16 years old. And so we really had to support the reading that they were doing. So I used Hypothesis a lot and I would go through their reading day to day or week to week and I would annotate the article myself and I would leave tips or I would leave explainers or links to other resources that would be helpful in the context for that particular content. And my students interacted with it that way as well. And in terms of just the interaction level, we were doing a lot of other things. We were moving one to one with Chromebooks and then they had iPads in their hands. And so we were playing around with some of the technical limitations, trying to work around those. We were also in the Canvas LMS. And so we were looking at the Hypothesis Alpha app. And so a lot of what I did still centered around the textbook reading itself and trying to push students to think out loud in any of their reading, obviously, we would annotate that by hand on paper if we were using paper articles. But really trying to drive Hypothesis is not only a reflection tool, but a conversational tool around scientific content at the high school level. Great. Thanks, Brian. Craig, you're up. Okay, can you hear me? Yeah, you're good. Okay, share my screen here. Okay, so my name is Craig Whippo. I'm an associate professor of biology at Dickinson State University. And it's a small four-year public institution in Western North Dakota. It serves a extremely rural population. And I have been using Hypothesis for the last eight months in my introductory biology courses. And let me just kind of go over why I'm using it. One of the things that I'm working on is trying to get my students to move from student mode to the way scientists think and understand biology. And this is a big adjustment. And along these lines, the National Science Foundation and AAAS came out with a document called Vision and Change in Undergraduate Biology Education. And in this document, they made some recommendations towards reforming the way introductory biology is taught at the undergraduate level. And so one of the recommendations is to focus in on a few key concepts and competencies. And then another one is for the students to participate more in the scientific process so that they understand how scientists are doing science and analyzing science. And then a third goal of these recommendations is to help the students step into that community of scientists and do so in a such a way that it's not normally taught in an introductory biology course. And then the fourth thing is this connection between math and biology. So making that more explicit. And if we look at the way traditional biology textbooks are written, as you can see in this slide, most of them are close to a thousand pages long. There's 50 to 60 chapters. They're just these encyclopedic volumes. And what happens in those textbooks is that the students are, if they read, they're reading for these bold terms and just this rudimentary understanding of the material. But one of the things that's really surprising about these textbook is how much the figures do not focus on actual research. So you can see that I've used this Campbell and only 7% of the figures in the textbooks are related to science. Everything else is sort of these graphics that are explaining what's going on in the text. And in response to these problems, a new textbook was published that's integrating concepts in biology. And I'm just gonna bring you over to the homepage here of the textbook. So it's written by Malcolm Campbell and Chris Paradise. And it's an online textbook. It cost about $35 for the semester of the students. So it's fairly low cost. And it is organized. Scroll back up here in these themes. So we can go over here. Oops. So if you go to look at the chapter, there are these themes. We start at the biological levels and then there are these big ideas in biology that it focuses on. And if we look at a reading, one of the things that stands out is that it's sort of structured like a real life scientific paper and that the figures or inscriptions are the center of the text. Everything is revolving around what's going on in those figures. So I looked at this textbook and I thought, wow, this is really cool stuff but I'm not quite sure that my students are gonna be able to do this. And I knew that I had to get them to read the textbook before they came to class. And I struggled with figuring out ways to do that. And what I ended up doing was I said, okay, we're gonna make this, I'm gonna encourage the students to do the reading by having a daily reading quiz. And in that daily reading quiz, the question is always the same. The question is provide convincing evidence that you did the reading for the day. And then using group annotation using hypothesis. So they do that the night before. And this is kind of the workflow that I use for those annotations. So usually I try to do a set of trailblazing annotations for the students ahead of time. So in that, I will post links to YouTube videos, YouTube lectures. Annotate things that I think are particularly important or things that I think that are confusing about the reading. Just kind of give the students that extra support they need to get through the reading. Then the students go through and then they annotate the reading assignment. In a particular fashion, I'll get to that in the next slide. Then I go back through before lecture and I reply to student annotations. Now I don't reply to everybody's annotations and I make it clear that if I don't reply to students annotations, it doesn't say anything about what I think of their annotations. I just, I reply to the ones that I think would be meaningful for the class to see how I'm replying to them. And I always structure those replies in a positive way. So I'm not criticizing how the students are annotating. I kind of give them nudges in certain directions but I don't say, hey, don't do this. This isn't right. And then after every exam, I go back through and I grade student annotations for that unit collectively. So I'm looking for things like, did they annotate every day? Were they annotating at a level that I would consider a college level annotation? Those types of things. It's extremely subjective. So that's the workflow that I use for the student annotations. And then I provide them with a handout about what are the types of possible annotations you can do. And I encourage them to tag. And so the students can annotate by looking up words and providing meaning. And I have a format that I'm specifically looking for in it definition. They can post links to relevant news articles, images, video or podcasts that might help the class understand the material. They can annotate their confusions. And when they annotate confusions, they really have to get into what's the nature of their confusion. An annotation that says, I don't understand this at all. Isn't gonna go very far. I want them to say, hey, I don't understand this because I'm stuck at this particular point in the reading. They can put their opinion down. They can reply to other students. And then they can answer the reading questions that come with the textbook. So if you scroll down, they read the textbook. There are these review questions and they can annotate those and answer those questions collectively as a group. And then the heart of the annotation are these inscription analysis. So this is where they annotate the figures. And the major focus of lecture time are these figures. And when you read a scientific text, the scientific text doesn't describe how to read a scientific figure. And the only place that a student can really learn how to interpret a scientific figure is in a communal group situation where everybody's struggling with what the figure means. And so this is the heart of what the students are doing and what I'm gonna do now is just show you one student's set of annotations. This student is one of the better annotators that I have and what she does for like a figure is she will highlight the figure like this, highlight it, and she will make a simplified title. So a good annotation will make a simplified title that of that figure, it will identify what the research question was that the scientists were trying to figure out. And then there's an analysis. And in that analysis, what I want the students to be doing is going in and explaining how the figure should be interpreted. What do the colors of the graphs mean? What do the symbols mean? And come up with some conclusion about what the figure is about. And so this student, and when students do this, they don't need to do every single figure collectively as a group. Every figure generally does get annotated in this way. And ideally when they study for the exam, they are going to these inscription annotations because the exams in the class are all about these figures. And so the way the exams are set up is that every reading, every exam question is exactly the same as the title of the chapter. So in this case, the exam question would be how will communities respond to climate change? And the nice thing about this textbook is that it has a bottom line. It gives the answer to the question. So what the students have to do is they are given the figures for this assignment and they need to make a valid scientific argument that the bottom line is correct based off of those figures. So the idea is they can use these annotations to study for those exams. So that's kind of how my students are annotating went over that. So that outcomes of using hypothesis in my classrooms are, I'm seeing a less distance between me and the students. And I can come into class and we've already kind of had a discussion outside of class and those barriers are much less. I'm seeing the students engage in the material in a more sophisticated way than they did with the traditional textbook. Before they were just looking for the bold words and a rough understanding of what's going on in the chapter. For me, one of the biggest benefits of using hypothesis is that I can get insights into how the students are thinking before we go into class. And I can say, hey, I know they're not annotating here, they're having trouble with this particular section. I'm gonna have to really think about how we go over that in class. And then the fourth outcome is my students are certainly reading much more than they did in the past, which was my primary goal for using hypothesis. So that's kind of my workflow and my thoughts about this tool. That's great. Thanks, Craig. I have all kinds of questions, but I want to turn it over to Marianne to perform the role of Elba Serrano. And then kind of take us into discussion. So Marianne, whatever you think you can cover from what you remember about Elba's projects that would be great. And I taught one more year after I joined hypothesis and so I mostly taught graduate level courses and neuroanatomy and then also medical students. And I did use hypothesis to annotate readings so that people could look at them, but not getting students to read ahead of classes, the bane of education all the way up through graduate education, medical students, you don't have to worry about because they're a whole different ball game. But I would have, I think, used it more for that purpose and I'm very impressive use of hypothesis. But Elba Serrano is a close colleague of mine. We've known each other for many, many, many years. And when I joined hypothesis, I immediately reached out to her because she teaches at New Mexico State. Again, it's a postgraduate institution, but it is mostly first generation immigrants and most of the ones are the first time that anyone has ever gone to college. And I know that one of the big things that Elba has been involved in and has been extremely successful at is really trying to encourage them to engage in science in STEM fields. But doing this by in many ways demystifying science and sort of taking it a little bit off of its pedestal and allowing students to engage with the literature in ways that doesn't just look at the content, doesn't just look at what it is that the scientists are saying, but how did the scientists do this work, right? Who are their colleagues, who are their networks? What other techniques do they reference? How do they assemble a set of tools and techniques? How do you get training in this? So I know that some of her early uses of it were really to help students look deeply at a scientific article and not just at the content of it, but the things that are sort of behind the article and being able to have people annotate, look for specific things, ask questions inside of the literature. This is mostly the primary scientific literature that she's looking at was extremely important. I also know that the last project that we discussed, I think she presented it at the Society for Neuroscience either last year or the year before, was again in this process of demystifying and looking critically at the scientific literature, the National Institutes of Health and other regulatory agencies have been issuing new guidelines in response to the fact that reproducibility in science is not what it could be or should be. And she used hypothesis and other rubrics to allow students to engage in the primary literature and specifically take NIH guidelines or other guidelines about what constitutes good science or rigorous science, I should say, not good science, but rigorous science, and apply those to scientific articles so that the students could go and look and say, are they making their data available? Are the statistics appropriate? Did they use proper controls? Did they use blinding? Did they use male and female mice? All of these types of things. So I think again, that even though she's dealing with postgraduate education mostly, the idea that this really fosters a more critical examination of not just content, but the process of science itself is something that's extremely important. I think also having lived through as Jeremy has the different phases of hypothesis, when Elba started, you could either put things in public or you could put them in private. There was no idea of a private group. But I think once the idea of the private group came out, it allowed them to sort of open up the sorts of things that they felt free to discuss, they felt free to reveal ignorance and other sorts of things in ways, I think that they were a little bit reluctant to do when they were annotating purely in the public sphere. But Elba and I in various conversations we've had said, there's some interesting possible uses of hypothesis in the sense that it does open a channel of communication with the authors of these papers and that it's some sort of future time. Scientists, this has been my primary role in hypothesis, is how scientists use hypothesis for research. How do they annotate things inside of the literature? How do they extract information and structure it so that it can be used in databases? And the ability to sort of reveal these public annotations, combine them, engage in conversations with authors themselves has been something that we've wanted to explore for a long time. Because again, I think it helps to humanize, it helps to connect students who may be far away from the big research centers with things that are going on at these universities. So I know that's only the tip of the iceberg of what Elba actually did, but I hope that gives you a flavor. And I hope I was accurate on disclaiming that Elba had anything to do with this presentation. That was great. Do you have some questions, Marianne? Or do you guys have some questions for each other? I think we can move into the discussion part at this point. So my first question is, I wanted to hear a little bit about any student feedback and what the students actually thought of using hypothesis, and wondering if you'd be willing to share those. So maybe just call on Brian first and then Craig. Yeah, I think for me at a high school level, this is not, you know, we're not def, I'm gonna say this, we're definitely not at the rigor of graduate or postgraduate science work. And so it was more of a training process for my students to when we take something like, I think ignorance in science is something that any scientist is willing to admit. Like I have no idea, you know, what is gonna happen in this context, but based on my background research, I can form a valid hypothesis and then I can test it from there. And so for a high school student, it's really hard for my students to admit, especially at an AP level where they're used to working at a very high level for a grade and trying to flip that around to this is how science works, right? And so, Marianne, what you said a minute ago was in using hypothesis for research to evaluate methods and systems, I used hypothesis in conjunction with unpaywall. If you're familiar with that, it has open access research articles because I don't have access to major databases. I don't have access to journal subscriptions, right? So I would grab these research articles and publications and we would do things like analyze the methods they had lined up with something we had done in class. For my students who were college bound, my juniors and seniors, they saw the connections and I've actually got an email back from one in particular at Purdue University and he said that was so helpful. Like going through those and seeing the annotations and having you interact with it and having an opportunity to ask a question of why did they do it this way and bringing those discussions out of the digital into the classroom space was also helpful for them. So it wasn't just this academic exercise or something that we just did because it's available, right? And for me, being very intentional about making those connections helped my students see the process of science and is a long-term process from what research has been done, what's been published about it, how do I take these publications and how do I evaluate them? How do I tie that into what my question is so that I can move forward? That kind of primary exposure was really, really powerful for them and I could see their growth as we went through the year and they practiced these skills over a long period of time. And so I was able to walk out of that class knowing that I had done something to prepare those who are interested in sciences as they get into the college level, but it was definitely a big transition for a lot of them. Craig, do you wanna give us some idea of what the students thought? That the feedback that I've received from my students have been mostly positive. They certainly like the feedback that I give and they get an email when I reply to their text and I think it has almost of a social media dimension to that and so they get that little buzz when it's like, oh, the professor replied and it makes them, it gives them a little reward. So they seem to like that and I've heard students say, where they're out with their friends and I've replied to an annotation and they get it and they're excited and they tell all their friends, hey, so they really like that. There's a difference between this semester and last semester, so I started last January and I had already had the students a semester prior and incoming freshmen right now are just beginning to figure out how to do it and there's still some grumbling going on, but at the end of the spring semester last year, everybody was on board and I had very high success rate in the class and I think hypothesis was part of that and the students did seem to respond to it well. I think Jeremy, there are some questions from the audience, do you want to go to those? Yeah, there's a couple of questions here. One is, I'll just throw them out together actually, one is what recommendations do you have for using hypothesis in a completely online environment, which I don't think either of you are in, but maybe we could just sort of hypothesize and then Brian, I can't remember, I know that you, there's a question about the use of hypothesis in Canvas. I know that you played with it or got it started, I don't think it got off the ground, but I know you took a pitch because we were at InstructureCon together, so online environments, but also maybe specifically Canvas, even if you haven't done it, what are your thoughts? Yeah, so for the Canvas piece, we had the Alpha version, so it was very, very early in the development process and we were running into issues where, just on a functional side, it used an iframe in bed and on a Chromebook, it was a stinky little space, and so the experience wasn't great for my students in that sense. I also wasn't grading my annotations because I really wanted them to just get into the scientific habit of doing it. Now, the wisdom behind that, I think if I had graded them, it would have been done more, but I didn't want to put, like, you should do this because it's graded, I wanted them, you should do this because this is, if I had to do it again, I might grade it, I don't know, so they didn't really get into the Canvas aspect, I am though, starting a course with some teachers in my new coaching role, and hypothesis, one of our initiatives in the district is to improve reading and writing fluency at all levels, and this is something that we would use at the higher grades, nine through 12, mostly, as part of the annotation process within that, and so playing with it more through Canvas is something on my list of things to do, to see how stability has changed, to look at different devices, so I don't have a whole lot of experience in what you can do and can't specifically do from that Canvas aspect, and I've never taught fully online, but I think along with anything, it's part of the toolkit, and Craig, you're much more rural than we were in urban district, right, so we see day-to-day, but in conjunction with other online tools, it can be very helpful because it can expose that the thinking processes that my students are going through, right, if we teach them to be honest about ignorance, or honest about what problems they're having in the context, and that was the biggest piece for me is I could see those problems in context of the reading or in context of what we were doing, and that helped me address those issues in context. Craig, do you have any thoughts, Thad? I don't teach online either. One of the things that I struggle with in hypothesis is how to get the students to respond to each other's annotations, and if I were teaching strictly online, I would want to figure out a way to encourage that behavior. In terms of what I'm doing in the classroom and the focus on interpreting these figures, I think hypothesis would be a good tool for that, but again, you'd have to figure out exactly how to create that space, and I haven't quite figured that out yet. I think that's dead on, though, just from my experience with hypothesis getting used in fully online environments that you really have to lean into that discursive piece. I mean, I do believe that hypothesis is an incredibly social tool and about as close to a genuine class conversation as you can have in a kind of electronic space, so I think it's a great tool for even an entirely online class, but I think Craig is pointing to the beginnings of how you'd want to really structure that around just pushing that social element, because you're not going to have the class to follow up on the annotations, so Craig is doing this with his face-to-face class, but the fact that he's following up so much with those students, and they were getting that thrill of a response, clearly that piece would also be super important in a fully online environment, because then, again, you're emphasizing the social piece, both for peer-to-peer, but also student-to-teacher, and I think a lot of the things that Craig mentioned, one of his lines was breaking down the distance between student and teacher, that's obviously gonna happen more in an online space because the distance is even, I would say, perhaps even greater, I don't know if that's actually true and how they think about it, but yeah, I would say it's probably greater online, you have, you know, feel farther away from the teacher, you don't have as much of a human interface, and this can provide that. I wanted to say one thing about grading annotations, it is true that there's a grading piece in the Canvas app that will probably be in other LMSs, I forget what Dickinson is using, Craig, but probably in other apps for other LMSs, not a requisite piece of it, but it is there, the real sort of heart of the Canvas apps and the other LMS apps is just to get hypothesis up and running natively so that you don't have to use the Chrome extension or something like that. But while we're there on the issue of grading annotations, I'll, sorry, Marianne, I'll just ask a quick question. Craig, can you talk a little bit more about the grading piece of this actual grade and annotations? So my grading is extremely subjective. You know this is being recorded, right? Yeah, yeah. That's my students, everything that I grade in the class is subjective. And it's basically a five point scale. And so a zero would be a student is doing middle school or a high school level work. And so they're just coming in and they highlight and annotate one word and they put a definition down or they have some question that tells me that they didn't bother to do the reading or just going through the motions. A one would be a poor quality annotation or a collective body of annotations where the student is coming in one time a week. It's an adequate annotation, it's not spectacular. It doesn't really contribute much to the community. A two would be a C level work. It's average level work for that grade level. The students are trying, they're coming in but they're still playing teacher games where this is an exercise that I've got to get through. And if I do one or two annotations and I pop in and I pop out and I'm done, that's passing. A three would be a B level work. This is good stuff. The students are answering those questions. They go along with the textbook. So the textbook has those questions in it. They're trying to answer those questions. They may be annotating a figure every now and then but collectively they're doing good work. It may not be consistent all the time but it's good work. And then a four would be A level work. And an A level student is annotating those figures. They're analyzing those figures. They're truly operating as a college student. And that's how I grade. It's subjective but there are criteria that the students are aware of. And kind of to tie into that, right? When I was introducing annotation, a lot of the early ones, and I wish Craig that I had had mine do figures much more than we had done the text. I love that. And that's something I'm gonna share with the science teachers I work with now because that's in particular at the AP level, right? Every long response question requires a figure of some kind. And so we did a lot with graph interpretation and analysis and construction. But anyways, I think I saw improvement, significant improvement if they paid attention to what I was doing as a model, right? And so anytime I made some kind of annotation, I tried to make sure I linked out to something else or included some outsider, tried to pull in seemingly disparate ideas and connecting them in that context. And so I think Craig, you called your trailblazing annotations, right? When I would do my pre-work or their pre-reading, essentially marking up that text for them, paying attention to what I know are gonna be confusing points or points of confusion for the students, just consistently year to year, that's what they are. Making sure to draw those out and have quality annotations there that either prompts them to respond or prompts them to think differently about the content. So I don't think there were more questions were there. One question I have, because I was looking at your students' magnificent annotations and the effort that she put into, this is for Craig, put into those annotations. And of course, hypothesis mission is to enable a conversation over the world's knowledge. And one of the reasons we keep promoting annotation, of course, for scientists is that they can share their knowledge over these papers with other scientists. Do you regularly share work between classes or between years? Is that something that is desirable or how does that work? I've been thinking about this issue. I think for my freshmen, what I'm doing is I'm just trying to get them used to a way of being and not sharing and being private allows them a playground to do that. And so I'm thinking as these students move through the biology program of incrementally making their annotations more public. And so then what they're doing is they're making a product for the world. And I see that as more in line with what I'm trying to do in the overall biology program here. And so in upper level courses, we would be moving away from this textbook to the honest to goodness peer review literature and making those sorts of figure analyses more public there. So the workflow that I'm thinking of is, as a sophomore, junior level work, they would be maybe going to like triple AS is science in the classroom and annotating using hypothesis, those articles that have already been annotated before or kind of using that as the next step towards learning how to read a peer review literature or going through the science in the classroom articles and looking for the rhetorical elements that are used in those. So annotating that way that it's more of a rhetorical analysis because the science in the classroom, they're already annotating the figures. The students will have already done that in introductory biology. So they kind of have, okay, this is what the annotator did. Now I can go in and I can look for those rhetorical elements that make that paper persuasive. And then as a junior and a senior, they would go into the peer review articles and annotate and look for those rhetorical elements and analyze figures and put them out in the world for the world to see. Yeah, I kind of see both sides to it. I think when I started with my students, it was prior to the groups. So everything we did was in public. I think it makes it easier for me too then because if it's a public annotation, I can reach out to someone I have a contact with and say, what are your thoughts? Can you help my student with this? And it's already there. But I also, and Jeremy, this came up in the Instructure Con discussion as well, right? If I want to use the same article or the same teaching piece between group to group, I would like to keep one set away from another set. So they're not biased to one direction or the others. We're building up the capacity. I don't know, and this is a struggle for me because I value open knowledge repositories, right? And so seeing the history of annotations, I think can also provide a launching point. And if I were to go back into a classroom, being critical about what text I'm choosing and what context is important, right? So if we're going through the practice stages, if you want to call them that, building analytical capacity, right? I can see the value in a private group setting. So we have a clean slate and we can work on that as a group together. But then, like Craig said, scaling it up, if we're looking at a graduate or a college level where we're contributing to the scientific community at a high school level, right? Scaling up is just getting used to being a good digital contributor in any sense. And we definitely have students who are interested in internships and things like that, that they are building up a good footprint for themselves in the annotation realm. Because now that's an open web standard, right? This is something that could go into that digital profile of the body of work that they have present on the web. Yeah. I just want to follow up there because I think the, and we've talked about this extensively, Craig and I, or we've talked about it before, it's not just the private to public thing, right? It's not just scaling up to public, although clearly culminating in public intervention in the scientific community is sort of the climax and the end point of it, right? When graduate students are suddenly, you know, publicly annotating on journals that have embedded hypothesis and talking back to the article or other scientists are doing it, right? In a public, discursive way. But it's also the different skill sets. I mean, I could see us talking in a year, maybe with an NSF grant or something like that and having mapped out and scaffolded from ninth grade through graduate school, the different specific skills that we're looking for in annotation and how we're prompting students to annotate and what they are demonstrating in their annotations from ninth grade biology all the way to like a senior biology class. And you guys have touched upon this, but you know, there's a different stages of that literacy. And I think you could really evaluate it or, you know, see it, witness it, however you want to put it, it doesn't have to be evaluated or surveilled, but you could, you could witness it from, you know, a portfolio. And this is what we talked about with Craig, was that if Craig continues to work with hypothesis year to year to year and some of those freshmen, I guess that the students you're working with now from the beginning of the year, right? If in four years from now, they will have demonstrated, they will have a profile portfolio, whatever footprint you want to call it, that you'll be able to look back on and that they'll be able to draw from themselves. One of the coolest things I'll just add from seeing those, they really was a pretty remarkable annotations that the one student that you showed was doing is that she has access to that stuff. You know, that's part of her archive of thinking and writing with hypothesis moving forward from that class to the next class and next class and maybe to grad school at New Mexico State University too, with Alba. Shout out to Alba. I think she's trying to join, but her computer. Oh no, oh no, maybe, maybe we can figure out a way to record her and splice her in somewhere. That's it, that would be very good. Yeah, so yes, I was also thinking again that it's not just public to private, right? Oh, there's Alba. That it's being ability to share from student to student or as you say, track your sophistication as you go through on even courses to say, look at my annotations now versus what they were when I was a teenager. I think there's all kinds of interesting sharing mechanisms. So Alba, I gave my view of what it is that you've done and it's probably completely different than what you actually did. And then I did talk about rigor and reproducibility. Yes. And learning the process of science. So maybe we'll just turn it over for the last few minutes so you can give the real view. I'm so sorry, digital technology. I'm very sorry. So just to, I did prepare a little presentation and I'm gonna call out a few of the slides. I'm sure Marianne said pretty much what I'm going to say. And so can you see my screen? Hello? You're not sharing yet. There should be a little green icon in the middle of the bottom row there. That's fine. Let me find a meeting zoom. Where'd the meeting go? Oh here, down here, sorry. I am new to all of this. There's an outcome share, share screen. Yep. Great, okay. So probably that one's mine. Do I click on mine then? Yes, share screen. Okay, great. Do you see it now? Yes. Yes, yes, there it is. So my hypothesis. Okay, I'm not going to do the projection. I'm just gonna go ahead and take you through it. So essentially if you or ever have taken graduate science seminar, we read papers and they tend to be dreary discussions. One person reads the whole thing and then somebody's laughing and they know exactly what I'm talking about and everybody checks in for 50 minutes of not much happening. So when I saw hypothesis, this seemed to me the perfect solution to engage everybody. So essentially I use Bloom's taxonomy because I'm a professor, so I have to set up some pedagogy. I'll come back to that if there's time. We can just share the slides after. So what we do in the class then, I tried this in two classes, disorders of the nervous system and neurogenetics. And these were classes that had students from multiple majors. So there was the opportunity for interdisciplinary collaboration and conversation. And so they set up the account and then I will tell you now that the students preferred working in a group rather than public blogging. They were very uncomfortable by the public blogging, although there is a public page that I can show you and I'll come back to that in a second. So what they did then is I opted to tell them just paste the PDF link as a URL and then it was a simple assignment. Two things they had to do. They had to post seven items, five were comments and the comments were specifically evaluate rigor and reproducibility in the paper. Things like statistical power. Did they give you enough information to replicate the experiment, et cetera? And then questions. So they could ask each other, hey, I'm not an engineer. What is this AFM they're talking about? And then they confirm the completion of the task. The outcomes were beyond expectations. It completely engaged the students. They filled in knowledge gaps for each other. They had lots of impromptu conversation and I experimented doing the annotation in class as well as outside of class and both of them work. So this changed the science paper from something that you endure to a dynamic conversation. And I'll see if this link will work. This is the public link. I can, maybe I'll do that later if there's time. And so then, so that's what I did and there is an example I can show you. But before I sign off, because I know we're knowing the end of time, for those of you who are teaching, there is- Take your time, Alba. Can you walk a little bit? We started a little late. Yeah, take it. I want to make sure this story gets told. It's a huge piece of the overall story. I mean, I'm very excited and I'll tell you. Okay, so let me see if I can just show you the page. The page was amazing. What they did, we actually picked a paper before it became super hot. There were multiple papers, but this is the one they did publicly on the connection between psychological state and the microbiome. That was the paper they annotated, which is something that has a lot of interest now. So hopefully the page will open in a second. While this is opening, and I'm not sure why it's not opening more swiftly, let me just copy the link sometimes that work. I had so many technical problems today. At a certain point, I felt like the force is just not with me on this one. Okay, here it is. Yeah, you made it. Yeah, I did. The computer decided to do a system update. I had done this. Anyway, I will bore you with computer details. So I think you guys are over here on this. So this is a page. I'm moving you over. So this should open up. Psychobiotics and the pursuit of happiness. We should all be interested in this topic. We truly are what we eat in more ways than we like to think. Oh, but I think we're just seeing your PowerPoint right now. So I have to go back to your application. So if you share your desktop, you can move things around. I'm going to do Zoom. Okay, yeah. Okay, meeting controls. That's not open. You're probably gonna have to on-share screen and share like a different sort of window, but you could also just drop that into the chat. I'm trying to open up Zoom again, and it's not giving me that window. Zoom cloud meetings. Now it's asking me to join the meeting again. I can't. Meeting channels. That's pretty meta. Okay, hold on. I think when you're sharing your screen, Zoom kind of, Let me stop share. Thank you. There you go. That was an experienced person. Let me put the screen where I want it, and then go back. And thank you for that tip. This is my first webinar. So I'm learning around. You're doing great. Oh, thank you. Thank you. Okay, share the screen. I'm assuming it's this one, the desktop. Is that it? Yes, you're seeing my desktop. Can you see it now? Okay. I do not want to leave the meeting. Okay. So here is, let me go back. This is a public one. So we can go here to the annotations. These are people who came in. I don't know when they weren't in the class. Here's the annotations. Here are my guys. Biomedical dude with water hose. And so what's not showing here, let me see if I can see the highlights. The color you bought for some reason. Anyway. You turned off the highlights, the little eyeball, just to the right of that black strip, yeah. Yeah, eyeball here. Yeah, okay. There, yeah, there we go. So they would highlight and make comments. And so this was the kind of discussion. They were, some of the, some of the, we're just like statements, symbiosis, and then they would really get into it at a certain point. And so she's asking, she's a nurse practitioner who take a master's with me. Talking about causes for diseases, et cetera. Mansi talked about concepts, keywords. So they basically go in and they say different kinds of things. They blog and comment in different ways. This student really got into it. And it was just fun. And then in class, and then I read about altered osmotic pressure. So they start having a conversation online that they follow up afterwards. And this link is publicly available if you want to browse it. And this was an example. They shared, shared videos, which is kind of fun because you can drop in other kinds of information as well. And they can, and it was fun for me to participate in this. It enriched the paper for me as well, which of course is important for us to feel that we are a part of the learning process. So can you see my PowerPoint again? Yes, okay, good. So that's what I wanted to say about that. I'm kind of out of sequence. What I do want to say, have you talked about assessment at all? Probably not enough. Tell us your thoughts on it. Just a little bit, yes. Just a little bit. So for those of us, and I'm sorry I'm coming in late, but I assume that many of the participants are professors. We have to do assessment for our portfolios. And we do it for good pedagogy. So when I designed it, I use Bloom's taxonomy. It's straightforward. We use it. And I keyed in on understand, analyze and evaluate as three of the parameters that this would facilitate learning in. And since then I have found on the web a lot of discussion of digital Bloom's taxonomy. And so I have some links here for you. I gather this is probably very prevalent in the schools, high schools, not so much in universities. I'm one of the scientists who uses this. And so this is really cool. It took the original Bloom's, the revised Bloom's and then the digital Bloom. And a lot of hypothesis maps nicely onto this, playing, operating, editing, validating. And this is exactly what my students were doing. I just showed you, they linked. They validated. They, the media clipping. So it really is very much in alignment with digital literacy. And then the final link I'm gonna give you in terms of assessment. I found this last night in an individual named Andrew Churches and he has a beautiful website called Educational Origami. He also founded Global Digital Citizen Foundation. And this is his website and you can navigate it. Just look up Educational Origami. And so the second to the last slide that I'm gonna show you was this, which I'm excited to try. He actually has rubrics. That we could all adapt because we really, it helps to work with somebody else's rubrics. So he has rubrics for digital taxonomy and for all the different Bloom's domain. So if anybody uses hypothesis or if we decided to do a collective project, it might be fun to do this as a group of classrooms develop a rubric. And then, and so it's interesting that he has this global website because my student, one of my students in the evaluation and the students evaluated this really highly, one of the students, John, who gave me permission to share his name said this. And I know Mary Ann and I were very excited about this. A website like hypothesis could serve to enhance the process of peer review, allowing for people from different regions of the world different areas of expertise to communicate and give feedback in real time. So I'm using this now in a case study, debate team activity that I've constructed. I think we could collaborate on papers with this. And that was my last slide that I feel the real future opportunity is to build global classrooms. And especially in our next-gen STEM scholars, global perspective. And again, my apologies for having it late. No, I was just so glad you did. More than made up to it with all that, that's just amazing stuff. I think we should, maybe I'll just give two minutes to see if there's any kind of back and forth responsive ones going on. Elbow, that was amazing. Thank you so much. I have a lot to say about it. I'm really excited about it. But Mary Ann, do you have any questions? Oh no, it's great. So Brian had one question that perhaps since we have the two college professors asking, is Brian still on? Yeah, I see him. You can ask your question yourself. But basically, what can he do at the K through 12 level to really start prepare his students for scientific literacy? I mean, it sounds like he's doing an awful lot already. What are you looking for? Our students get to do this. So I would say you could, I did this with a primary literature paper, but you could do exactly the same thing with something like Discover Magazine, something that is directed towards the general public, more so. You could do it in background material. I think you could use it. I could imagine doing it in many different components. So I teach a large non-majors human biology class that I love. And I thought about doing it with that, but I think I couldn't proctor it really well. 130, there might be things going on that would be unfortunate. But I think if you have a classroom of 30 or under, you could even take a digital textbook and have them highlight, I don't get this term. What is this? And these are informative for us. I do this at the end of class with postcards, little cards. What didn't you understand? Or give me some keywords of the concepts. So we would get instant feedback about that. And so, I mean, this translates. I think if in a classroom I might take Discover Magazine and then I set it up for rigor and reproducibility, this is huge in our area. But I would say for you, what I do with my non-majors who would be comparable to high school and elementary is, how is this relevant for your community? Really make that connection at first. So we read about a global explosion. How is this relevant? And have them make those connections. Yeah, would you suggest the same for majors? Because, I mean, working with higher level juniors and seniors in science classes, what science, and this is something that we discussed before you were able to hop in a little bit, but are there literacy, scientific literacy skills that you see freshman major or sophomore majors deficient in that we can begin to prepare? And it's not just them, it's all students. We're having a huge conversation. And this is something I've forgotten the slides, but within there. So this can be expanded. And here's actually the opportunity. So thank you for saying this. It's the annotation of images and graphs. Yes, you got it. Yeah, that's exactly what Craig talked about. So I would say, skip the text. Put a graph up and say, describe the graph to me or as a group, what is this graph telling you? What is it not telling you? Have them do it in a graphic format or an image. Have them annotate a brain and give them fun. Make it fun. Say, okay, you're now in teams. How many can you annotate in the next minute? Something like that. Competitions are fun, friendly competition. And now you've given me an idea for something to do with my non-mages God. Because I think this is what this, we need to have them think like scientists. Now, at the level of grad school, we hope they can do a graph, but now maybe I should check. But certainly, yes, graph and image annotations. Which is what Craig was presenting on. I was very impressed with the introductory textbook that he was using that really is based on actual scientific data as opposed to interpretive figures of what osmosis is. Yeah, we'll have to connect you guys so you can see the work that Craig was doing with. I'll let you go ahead, Craig. Go ahead. I think at the high school level, the most important thing to develop is the ability to read recursively. My students coming in don't know how to read something once and then read it again. They think that if they've read it, they've read it and that's it. And the students that are struggling in my class are not reading and then going back in and really doing the sort of metacognition to go back in and say, this is where I need to look at again. Do you think annotation kind of organically helps that? Just by the virtue of the fact that you're reading a text, you're commenting on it, that's gonna make you dig deeper. But going back and replying to or looking at other comments is gonna just make you cycle through that text so many times or should. That's always what I thought was great about teaching collaborative annotation was that they were constantly going back, especially if they start to get messages that say, Dr. Whippo commented on my annotation or my classmate just annotated something. We don't quite have all those notifications set up, but once you get that notification system, my hope is that people will be going back to their biology textbook as much as they go to Facebook or Twitter or Insta or whatever they're doing now. I think the biggest problem that I have is just students' attitudes towards assignments are that they're teacher games that you need to get through and they just need to slow down and think about why they're doing what they're doing. Yeah, I saw that as well. And one of my favorite things, especially at the high school level because I was working on a master's degree at the same time as I would bring in my actual books that are just filled with writing and showing them like, it's a skill, something you're gonna use outside of even biology, but it's something, the more you do it, the more you'll understand it. That's kind of getting back to the public versus private group things, is that you can link annotation to annotation. So the recursiveness can definitely be scaffolded back into later concepts. If we link back to old ideas and kind of force that discussion back to, if you don't remember what this is, maybe you should go read it again type of thing. So I don't think I've heard the words recursive reading put together, but I really like the image it conjures for me to help get it back to my students. Well, I think we should probably call it there. This has been an amazing conversation, Elba. I thank you so much for just jumping in when you could and making it happen because I'm just really glad your voice was part of this and your project got the full articulation. I really do think, I said this before you got on Elba that maybe some point down the road with some NSF funding, we could talk about how to scale this from secondary to post-secondary to graduate levels and talk about how people should be annotating at those different levels. And indeed, I think Elba, you mentioned just this wonderful idea of maybe, I don't know, maybe this is how I read you, but if some point Brian's students are annotating a text and your students were also on that text and possibly acting as the experts and answering some of those questions that were coming up, that would also be really cool to see some annotation across different levels. So thank you guys so much for your work. I hope that we'll continue to share resources and projects and maybe move formally for a collaboration to bring annotation across science and enhance scientific literacy. Thanks, everybody. Thank you, Jerry. Thank you, everybody. Nice meeting you all. Thank you, participants. Thank you. Thank you so much, everyone.