 One of the things is that as I was rewatching your the little talk you gave at at OLC You you mentioned that you that you and or CSU CI Kind of think about piloting and maybe a little differently than some folks Or you you try to do different things and pilots some folks do like some folks are are really seeing it as a kind of like Purely evaluation kind of exercise Where they you know they'll they'll make a choice based on the outcomes of the pilot and I'm sure that's part of a component of What you're doing, but you talked about pilots a little bit differently in that talk. I don't know if you remember But how do you think about piloting? Yeah, so I mean it kind of depends on the tool and I mean this is actually a process that we're just now kind of Really trying to formalize on our campus So it's it's something that we're really trying to kind of develop exactly what are the steps that we go through I think what you're referring to is when we were doing we called it a pilot phase for canvas, which was our LMS shift And when we did that one we framed it as a pilot with intent to with intent to adopt And that was that was really but there was a lot of other Kind of stuff that went in that was a CSU level. We went through an LMS evaluation With different vendors and stuff like that. So there was a little bit of kind of background work already done in that instance The stuff that we do with hypothesis Well, I mean the benefit we had with hypothesis is technically we already had the tool on But we weren't really getting a bunch of use out of it and part of that was we weren't publicizing it that it was there And then some of the faculty that we had Use it when it was there It had some stuff that was a little buggy at the time And then we heard from you guys. Hey, you know, we fixed some stuff We had looked at it and we had used it within our team seeing like yeah, this does work a lot more stable And then when the idea of a pilot came up It sounded like a great opportunity to kind of partner and also get the tool some more widespread use Because it's definitely something where the functionality was there We knew that the faculty would be able to use it We saw merit in it and it was really just this was kind of an opportunity to one get the word out for it But also to have some kind of top-level support from you guys, which was great And that was a big incentive to get our faculty on board too. It's like hey, you're gonna be in this But it's not just gonna be like a pilot in the sense that here's a new tool go play with it There was actually some structure So that was part of what we're formalizing at CI is kind of that plan So having the administrative support from you guys, but also having we did multiple check-ins throughout the pilot And they were all scheduled out So we we got our group together had an initial on board and then from there, you know We said these are the onset requirements of you as a pilot member So we required them to do certain things and parameters within it. So they were held accountable For actually doing it. So it's not just like here's a new tool you get to use it and if you don't like it I mean, whatever. No one cares like we needed some actual Metrics out of it to some extent, but we also wanted the group engaged Both with the tool and with each other and that actually kind of turned into one of my favorite parts of the pilot process and Like as a whole was actually seeing the faculty share what they were doing with it with one another It was just like sitting in a think tank with this new New tool that they hadn't used before it was actually really fun to watch That actually hit on a couple of points that I was that I wanted to draw out. So You know, it's interesting that you use the pilot as an opportunity for this kind of what's called a you know communications campaign to to reach out to faculty about the existence of this tool and You said you were saying that you gathered people in these focus groups, you know kickoff and then You know kind of touch points along the way. How did you how did you first? Gather the faculty. How did you identify you would participate? So we're fortunate in the sense that our teaching learning innovations team We have a group of what we call faculty ambassadors or faculty fellows And this is a group of faculty that we've brought on We bring them on like this as a semester basis. So we've we've built up a group over the years That works pretty closely with us. So the nice thing is once we get them It's really supposed to be like technically a semester long fellowship But once we work with them and they work with us, we we don't really let them go So our team kind of indirectly grows. So we have a good core network faculty fellows that we kind of reached out to initially but then we also took a look at what the tool was and You know focused on collaborative annotation and how it really could apply in different kind of disciplines English and composition came to mind really easily So we tried to grab some specific folks out of those kinds of programs and then we also put it out there just to anybody that we had talked to Hey, you know, we have we have space in this anybody that you think is interested Spread the word and have them get in contact with us and we actually had Once I once we threw that kind of invitation out I actually went to the composition teams program meeting at the beginning of the semester and just did like a Really quick fire snapshot of like hey, there's this tool. We have this is kind of sort of what it does If you want to learn more and you're interested and we ended up getting like four or five folks out of that group So we did it was kind of a combination of just organic word-of-mouth and then also, you know, it's kind of intentional programmatic You know demonstrations and invitations And then so it was a grab bag of different things and we ended up with I think 16 people that signed on to do it It was it was a good group and we got we got people out of you know, some disciplines We don't work with all that closely like history. We got a great great professor out of there And we don't do a ton with history faculty historically But they've yeah, it was it was great. We had a pretty diverse group and it was awesome That sounds interesting. I'm really I'm Interested to hear about this faculty fellows group that you guys steward So that's kind of like folks that you build a relationship and then that relationship sort of maintains over time It sounds like yeah, so we we end up I forget exactly how it works I think there's a there's a bit of course release time that we build into the fellowship So they come to us. It is actually an application process So they they come to us and they propose a project that they want to do And something that we could in some capacity support them in doing and then From that we bring on It's like two people And we'll bring on those two people and we work with them to get that project So there's a little bit of the course buyout so they have the time to work on this project And then we just get to build and foster that relationship with them the whole time So some of our first ones that we had we had Jamie Hannins and Jacob Jenkins and now those two are Basically leading and running our open CI initiative, which is our Affordable solutions initiative on campus. So they're doing all this work around OER stuff and our z-degree programs Which are degree programs you can get through entirely without having additional costs for textbooks And I mean they were some of our early fellows and then they just they that was kind of their project that they started working on together And then it's taken off and they're running this whole thing So that's kind of like the stuff that we try to foster and I mean our It turns into where we have like our small core team meetings like every week or so But every like like once a month we have what we call our big team meeting And that's where we have all the fellows and everybody it's just this big long table of not just us But we have the faculty voices around And it doesn't it does a lot in the sense that it helps keep us grounded in the group that we help support And it also connects us closer to our students. So it it serves a really big purpose With our team and our campus. So it's it's very intentional and it works out great That sounds really really great. I wish every institution we're doing that That may be more hard than I realized but I haven't always seen that intentionality. It's I mean Yeah, I mean we and we talk about it with a bunch of people and there's a lot of campuses that are doing stuff similar So it's I mean that kind of networking happens to but We definitely have a special group of faculty at our campus and are there it sounds like it and partially because you've been Kind of engaging with them in this way. Did are there students also kind of engaged at that level of the process? Not a ton However, we do have a couple student assistants that we try to bring into the conversations as much as we can Especially when it is stuff that kind of focuses around You know things that we're doing faculty and faculty are doing with their students We try to incorporate them in on like any kind of panel that we do or anything like that So that we have that student voice represented because I mean we don't the last thing We want us to just speak in a silo and think that we think we know we're talking about where we're actually way off base You know, so having that kind of representation is important I'd say that's something we could probably build up a little bit more but Student assistants that we have right now are great and they they did one of the best things about them as they are Honest they will tell you exactly, you know, what's working? What's not and why? So it's it's great to keep them incorporated too Yeah, that makes sense. It's it is I mean students have other needs and are really busy and so it's It's I know from my experience. It's really hard to get them to you know, large numbers of students to Participate in things that don't count for their degree really, but um when you're also paying them to you know to be Involved that's a seems like cheating, but it's actually probably the best way to do it I mean it definitely works and at the end of the day like the students are the ones that we're serving So if we don't have that voice represented, you know, you could end up doing stuff It's just not doing justice to the students and that's ultimately what we're after right so it's definitely needed No, this is stepping back a little bit But how did hypothesis first get on your guys's radar? Did it come from some faculty member or was it something that you guys? Identified or how did it first do you remember how it first reared its beautiful head? I Can't remember exactly who it was, but it was definitely a word of mouth thing And we just saw the web version first and it was just kind of like hey There's this there's this tool where you can go ahead and you can just mark up anything on the internet We're like what no we figured that out and then once it was You know the step further was you know We can plug it into your LMS and people can use it with you know the readings and stuff They're already doing in their class and it's like Okay, there's there's something here like this is something that would be really useful I mean it makes it so that You know you can take your own notes privately in a digital format in a way that actually makes sense But you can also develop this really rich conversation around These readings and stuff that you're supposed to be doing for class because It's really hard to tell especially in an online class if you know the readings if students are actually engaging with them At a level that you're necessarily wanting right? I mean you can have those conversations and discussion boards and stuff like that but You know for what I've seen a lot of times that that leaves a little bit more to be desired Um, and it's kind of just like yeah, you could have skimmed it and got that information But when there's a conversation happening directly on the content It's kind of it's irrefutable right so um I don't remember. Yeah, I don't remember exactly where we first heard about it But I remember it it took hold pretty quickly once we saw the potential with it and then You know move forward to the pilot stuff. He showed it to some faculty and they instantly saw the exact connections too They're like, oh man, I can actually like Take this thing that I'm teaching them how to do and they can apply it directly to a text In class with their classmates and not only apply what they're learning But also communicate with each other why or why it's not working or you know actually Take this thing apart Which I mean My ap English class back in high school. I wish we had stuff like that because it would have been great I mean, that's pretty much what we did in a classroom But it makes it so that you can transfer this to an online space and actually maintain that rich conversation Yeah, that that totally makes sense and resonates with with what other people have Have been saying that idea that the conversation can take place on the text Like you said, it's irrefutable then that you know How and where the conversation is going on as opposed to the discussion forum where it's sort of to the side and could be Could be less intense So um What uh, you know, you talked about how for your team, you know making connection to a discipline that you hadn't made before was um Was kind of a great outcome of the pilot with that connection to history Was from what you've heard from faculty and or students Were there what was were there other kind of like, uh, you know kind of Amazing possibly unexpected or expected outcomes that came out of that piloting work Um, I I don't know about necessarily outcome But it was really like I was saying earlier like one of my favorite things a hearing is just the different ways Instructors were using it. Um We had one of our instructors. I'm gonna mess this up exactly what it was But they had they were doing something where they were I think it was they were doing a unit on paraphrasing Um, and instead of just, you know, she actually used hypothesis to do it So they threw a document it up and it was like, okay, I need you to take a chunk of this And then paraphrase a part of it and then They were able to like work all the students were able to do their own But they were also to look at other people's and be like, oh, yeah, that's okay That's interesting and then the instructor was also leaving comments on it and be like, okay This worked because of this or not And it tied directly into their curriculum, but it also served as an active practice Um, which was one of the coolest that one one of the coolest outcomes out of it that I saw and it was just like Because it took something she was already doing in her course But it gave it this whole new kind of life in terms of an activity with the class And she said the students were really it it drove the point home. The students really she said Grasp it really well with that sort of type of exercise So getting to see that kind of thing and how these different instructors were hitting it from different angles Um, some of my favorite things about any kind of tool that we adopt, right? I mean the end of the day a tool is a tool it does, you know, it it serves a purpose to Do a certain amount of functionality, right? But it's when you put it in the hands of people that you really get to see what it can do and how it can be used Because at that point They can come back and be like never would have thought of doing it that way Like I literally pull up a website and add notes to it like a google doc Like not innovative groundbreaking stuff, right? And then you have an instructor come with it be like, yeah, I had this active engagement Like activity that I used to do in my class and now I just threw it up here and everybody got to you know Actually practice what we were teaching. It's like, whoa, I I mean, I'm not on and that's why I love working in the faculty. I'm not at that level in a classroom every day Um, so getting to see these tools used in unique and innovative ways is the best part about it That makes that makes sense. Yeah, and it's we see such a variety in the way people use it in ways that we can't even anticipate You know, it's like, oh my god. We never thought of that. Exactly. And that's like the coolest part of any of it. It's like, oh cool um You may not know this but that made me think um, were you uh, do you have any insight into the different ways that faculty incorporated annotation into their assessment like were Um, was it being used as a part of the great, you know for assignments that were part of the grade or You know, was it similar to maybe like discussion forum assessment practices or do you have any insight into how that was being used? Yeah, I mean it was um, I definitely know they were using it for um, you know graded work just because One of the biggest pieces of feedback we got out of it was I need this to integrate with speed grader Which I know you guys are working on so Getting that that's gonna make all the pilot faculty and I think everybody else that wants to use this thing They're gonna be doing like cartwheels once they see that thing working because that was the number one Like feedback point that we got was everything was great. I just wish I could push it into a speed grader and was like That makes total sense. Um So I mean in terms of what kind of it. I mean they were I know they were doing some sorts of uh, you know, kind of Summative assessment like that, but I know that they were also like the uh Going back to that paraphrase activity They were doing formative assessment with it to just to kind of check in and be like it's active practice to feed And I think those things were even graded to um, but to a lesser extent it was more just You know, I want to see where you're at with this to make sure you grasp this concept Um, and I think there was a lot of value in that for sure Yeah, well that uh, I mean what you're talking about, um, really just just toward how this is super valuable for hypothesis too Because it's through engagements like this that you know We hear this direct feedback from folks who are really using the tool like on your team So that we know what to do next on the roadmap to make sure that it's happening and um, yeah from uh From my understanding we'll be showcasing the speed grader integration tomorrow. So uh Your dreams have come through your input has been heard And I want to give you guys kudos for that I mean you guys have been receptive to all the stuff that we've gotten It's you know, it actually impacted your roadmap and being able to see that is That's the kind of stuff we look for and um, you know partnerships with vendors is like we want to be able to Last thing you want is where you have a tool and you throw feedback into a black hole and you're like, yeah That'll amount to nothing. Yeah two years from now. It's still in the in the forum or whatever with no remarks Yeah, getting getting thumbs ups and whatever it's like, okay Yeah, I'm still on a couple of those forums for something where it's like every time somebody gives it a thumbs up in github I hear about it and it's like six years later Still still hasn't happened though Yeah, so um, actually I remembering what it is now it was a lot so that github issues could have uh deadlines But anyway, that's another that's a whole nother story. Um Let's github itself So I know that you guys because you shared it with us. I know that you guys did some Surveying of the participants and so is that um, is that the main way that you Got feedback from folks? It's only probably also did face-to-face in those sessions that you had Um, but could you talk a little bit about how you guys structure getting feedback from People and how you incorporated into that about thinking about what to do with tool next Yeah, so I mean the check-in meetings. So we did along our pilot timeline. We had two check-in meeting The check-in meetings that were scheduled. Um, and that was with you guys and me and the faculty that could come So in those meetings, we got uh, because it was kind of just an open floor Like let's see what's going on how you're using it show some stuff But also let us know what's working. What's not so that was kind of the the informal feedback But those were definitely times where we were able to gather feedback about Just about the tool in general and see what the general experience was um, and then the formal Uh The formal data that we gathered through the evaluation form that we sent out at the end of the semester That was kind of like the more formalized and documented data Um that we got so we did It was a combination of the two but we did require that our faculty Or our pilot faculty did go through at the end of the semester and complete The evaluation survey so that we had some stuff to actually report on after the fact And it was it was good stuff. I mean that way we got to get you know Stuff to refer back to and be like well, this is what they liked about it This is what they not and then we got to get a general sense of Is this a tool that's worth sustaining because that's ultimately what we need to look for in a pilot to see Is this something that is worth the adoption like if we get generally speaking positive note on it likely that will trickle to Majority of faculty that use it. I mean you're never going to get a tool that makes everyone happy But it's not a mandatory tool either. It's not something you have to be using um So those are the that that kind of metric is what we use to kind of guide and steer the next steps um in terms of adoption And if I remember correctly the outcomes of that survey were showed pretty strong You know, uh, kind of uh enthusiasm for for the people who participated in pilot We had no negative feedback at all. It was all either neutral or positive And even it waited more on the positive side There were a few that were kind of just you know indifferent on some of the some of the questions we asked like You know, some of them were like, you know, would you refer this to someone else and they were like, I don't know maybe Um, but you know, most most of it was was on the positive end of things. Yeah That makes sense and so, you know looking back on it now that you're on the other side of this pilot Was there anything that you would do differently in the pilot than what you did or would ask hypothesis to do differently? Um, you know from your standpoint, I don't think so Like you guys were very available, which is really the best thing that we could ask for um, like Jeremy allowed us to have a Calenly link put up on our website that went directly to a place where our faculty were able to book an appointment with them. I mean To to an extent like that's kind of unheard of like you guys are literally like, yeah schedule meetings with me um, so that was great and I mean just having that in there as Something that was offered even if it wasn't utilized a ton. It was there. So that kind of support was great um, like I said our pilot process, we're still kind of Figuring it out. I think part of it is really just figuring out Exactly what evaluation data we want and need Moving forward just to make sure that we actually have basis for adoption of new stuff when we get to that point Um, that being said, I think we did a pretty good job for the first one that was formalized like this But it didn't form something. So we're refining a little bit. Um, but overall and generally speaking I think that the overall process worked well and the faculty seemed pretty receptive to it That makes sense. Hey before we move on could you um, just give me a quick You said about 16 faculty participants were in the pilot. Can you just to the best of your memory? Can you give me a quick rundown on like what disciplines they represented? Oh, yeah, yeah, that's a morning test. I think I know we had history. We had English composition ESRM I believe Who else did we have political science? What's the ESRM stand for? I'm really testing you environmental science and resource management. I hope I'm 95 percent sure that's right You know, we can this is one of those things that we could follow up on later and get right in pros. So Yeah, for sure. I don't want to put you on the spot. Um, I got it. It's environmental science and resource management We have a winner. Oh, man. I was just terrified it would be so mad at me if I got that wrong Yeah, well, we wouldn't put anything like that up on video. So don't worry about it. Um That's whoosh. Yeah, so you had uh, you had humanities folks. You had some stem folks Yeah, so we had we had arts and sciences group. We had stem folks We had stuff out of you know, political science history. So it was it was a pretty wide net Which I was stoked for I'm that's one thing I would say I'd probably like to improve on is just get an even wider group of um A faculty moving forward in our future ones, but Like I said first one out of the gates. I was really excited with the the diversity of the group It was it was cool to see And was that um, was there a mix of both online and face-to-face or blended courses or yeah, yeah, we had uh I think a lot of them were blended or online, but we definitely don't actually know we had we had it was a Mixture of all three modalities for sure. Um, so yeah, it was it was all over the board, which was awesome Yeah, that's great. Because then you really have uh Everybody testing in different environments exactly Well, so I don't want to take up all your day But how about at least one more question here? Like what what are you guys thinking to do next with annotation hypothesis? I mean, obviously the speed grader thing is coming out And so at the very least he can uh can implement that Although I don't think the implementation will take that much actually, but um, what are you guys thinking for next? I'm looking forward to that not having to do much part No, I mean the biggest thing I think moving forward is just kind of getting more uh more people at the table and using the tool So that part of that is just going to be ongoing support, but also, um You know these kinds of anytime there's like a feature a major feature update like the speed grader integration It gives us an opportunity to kind of communicate it and bring it back to the forefront be like Hey, there is this cool thing that we have That folks should use And you know, it's just it's just putting more word out. Um, and we really really like having That kind of organic Growth where it's like, you know get the this group of faculty that we have right now excited enough about it to the point where they're recommending it Um, because some of our most successful stuff has really come from that Um, so it's right now. We're really just trying to get I think get people hyped on it Um and just kind of explore and get them sharing we want to get people sharing what they're doing with one another Because that's really what ultimately will drive the adoption. Um, you know We can stand up there and give demos all day and talk about it and say this is great, but When you're an instructor and you hear from other instructors that are actually using it They're like, well, I was able to do this really cool thing with my students That's what's going to get people excited and want to use it and that's the kind of growth that we really want Um, so, you know, we we're good at finding those excitable groups that are just like, yeah new stuff Give me because that's what we are But really just emphasizing that collaboration amongst their peers and that that kind of organic growth is our next kind of focus really It makes total sense that reminds me of this thing. I've been wanting to do for a while, which is um put up a little Sort of easy way for a faculty member to leave a kind of little video been yet About um, you know, one of the cool ways that they've used annotation So they could just like kind of pop in and do like a, you know, one or two minute quick video I saw somebody else do this and you know, there's folks all over Nation and the world really that are doing this. So it'd be great to just have that kind of compendium of Of ways different people using it could become a kind of recipe book for other folks Yeah, that'd be awesome. It turns into like an inspiration wall of sorts You know, it's like, well, there's this thing I heard about but I don't know what to do with it And it just gets some ideas. Yeah Love that kind of stuff Especially if people could tag themselves by discipline and modality and stuff and then Yeah, that could be really cool. Oh, thanks for reminding me of that idea because I love to do that And if we get that up and running, we'll let we'll let you know so you can yeah, absolutely Because we we'd leverage that for sure Put your faculty on it for sure. Yeah. Hey, um, one thing I forgot to ask before is um, did you So you guys were obviously offering your own hands on support for this and you talked about how hypothesis was Was also there for support. Did you feel like the support burden during the pilot for hypothesis was large or how would you characterize the support side of it? Really minimal actually, um, that that was kind of the nice thing. It wasn't like every time we were doing stuff You hit roadblocks. I mean there were questions that came up. Sure But I mean that that comes with the territory of anything new. I mean even stuff that we've had for Ever we get questions on stuff. So, um, it was I would say by no means was it like we saw You know this flood of stuff like well, I tried using and this didn't happen. I don't know what to do from here You know, we didn't hit any of that and I think a lot of that has to do with one the tool worked great But also that we had that initial kickoff meeting where we kind of showed that, you know, general workflow of how everything worked That helped a lot. I mean that that and we recorded that webinar So we were able to blast it out to everybody so they could refer back to it. Um, I think that that did a lot of it and even during that kickoff people were kind of building stuff As we were going through it so that way they were able to kind of at least see Things and if there were parts they were confused about we got a lot of it out of the way in the beginning Um, and then from there we had, you know, your guys' support We were able to you know field some questions then we had documentation that Between all those things it was fine. It it was in no way shape or form Did it get in the way of, you know, everyday work at all? So it was it was great in that regard And is that did you have much student support needs like students having trouble with annotation? No, not really. Um, we didn't hear we didn't really hear any I think we had maybe one faculty said they had a couple students that were having, you know Some difficulty getting it loaded up, but I mean Even those things they were like little browser glitches which happened with anything You know, it's kind of the daily grind of just, you know working in technology for You know an education it wasn't there was nothing that was like, oh my god We can't run this thing anymore Nothing close at all. So Yeah, well as you and I both demonstrated this morning there can always be a little technology glitch, right? I mean all the stuff you use every single day I think What it happens Well, that's uh, that's all really great info. And I think there's a lot of stuff in there that We can use and I really I want to um Make something that um, you know, someone just like you at another institution can look at and be like Oh, here's a human being who did this thing with their team and it actually worked out So I think there's a lot of good stuff in there Is there anything else that you would give as advice to someone like yourself? Who's just maybe thinking about embarking on a pilot like this? I'd say take the plunge and go for it. I mean Realistically like that's the best thing you can possibly do. I mean obviously research, but for speaking specifically to hypothesis It was easy. I mean it was it was Painless to get up and running and get going. Um You know, it's it's kind of setting that initial bar for as we get this process going with You know, potentially bigger vendors or whoever else comes through the door that might be more challenging You know, we know that it can be this smooth and easy Um, so I would just say, you know gather a group of faculty get them excited about it and try it out I like the idea that you would hold other vendors to our standards. That would be great I mean Anybody that we get to work with it's simple and easy. That's that's my standard of working with anyone Like I don't want headaches and jumping through hoops. No one's gonna be like come back to us when your hypothesis level Start throwing that out there You