 Okay, so we're gonna get started today. Thanks everybody for joining us. Really excited to have this on the agenda today. Basically go through some introductions, give you some overview of what we do at CCCOER. Then we're gonna talk about some specific research, talk about some strategies that you can employ at your campus and then open it up for Q&A. I put a note in the chat if you can find the chat window. And if at any time you wanna leave a note there, that would be great. We like to kind of keep folks on mute when they're not asking a question just because of background noise. So the chat works well for that. I'll be monitoring that. I know Una and Liz will be monitoring that as well. And then we've got some final announcements about upcoming events. So our speakers today, do you all wanna just introduce yourselves? Philip Grimaldi. Sure, my name's Philip Grimaldi. I'm the director of research at OpenStacks. Hi, well, I've been super happy to be here. Virginia. Hi, I'm Virginia Clinton. I'm an assistant professor in the Department of Education, Health and Behavior at the University of North Dakota. And one of my main lines of research is open educational resources and their efficacy. Awesome. So at CCCOER, you may know this already, but we are a community of practice dedicated to promoting the adoption and development of open educational resources to enhance teaching and learning. So CCCOER was founded to support the community college mission of open access by promoting OER as a low cost alternative to make instructional materials more affordable and accessible for students. So the main way that that happens is by, the community of practice that you see through these webinars, through the listserv, through the website, and various events that we host. We also coordinate regional leadership and try to focus on promoting student success. So members of CCCOER across the country, you can see the membership there. There's 15 statewide members as well as these are 90 members from 34 states. So those are individual institutions. And then this year we have two new member institutions, Trinity Valley Community College and Butler Community College. So welcome Trinity Valley and Butler. So that's cool. Do, if you haven't, if you don't already go there regularly, please check out our website. Some really great resources here. All of the webinars are recorded and posted on the website. So this webinar will be recorded and posted there. We have case studies, blogs in particular. We have a diversity and inclusion blog, equity diversity and inclusion blog series that we're doing right now. There's also a calendar of upcoming events. So make sure to check out the CCCOER.org website. Okay, so coming back to the topic for today, I wanted to kind of go back to our guests and if you could just talk a little bit more about your research background, kind of your education, but also like what you've been studying and particularly how it intersects with OER research. So. Sure, so I'm a cognitive psychologist by training, which means I essentially study the mind and mental processes. I specialize in learning processes in particular. So I wanna say I've been studying learning since I was an undergraduate at Kent State University working in various labs there. I did my graduate work at Purdue University where I was doing more traditional theoretical members and switching more applied learning research. Along the way, I made a lot of interesting friends, one of which was Rich Baranek, who's the founder of OpenStacks. And Rich invited me to come to Rice University and do a postdoc at Rice where we were doing research, kind of integrating cognitive sciences into the OpenStacks technology and resources and sort of also looking at ways to use cognitive psychology and cognitive sciences to inform his area of research, which was machine learning is still as machine learning and kind of educational data mining. After, yeah, so being at OpenStacks for a little bit, we start, we're doing a lot of research and that's more kind of traditional applied learning research and somewhere along the way, my boss and managing director of OpenStacks, Daniel Williamson, we're gonna meeting one day. And if you're familiar with OpenStacks, one of our biggest kind of achievements really has been the number of dollars we've actually saved students over the years that we've been in existence. And that's kind of been the number that we've used to sort of tout our value for a long time. And then Daniel kind of noticed in the last few years, that number, although it's very important, a lot of funders and people that are interested in this kind of area started asking the question, well, like, okay, so you saved all this money, so what other kind of impacts can you see from giving students a free book? And so Daniel and I were in a meeting one day and he was like, what can we do with the research wise? And kind of my knee jerk reaction was, oh, look, I mean, so our objective, right? When we go out to create an OpenStacks book is to create something that's essentially the same in terms of quality as something that's on the market and they're charging $400 for it. So in many ways, if you're swapping out this $400 textbook for a textbook of equivalent quality that's free from my perspective as a cognitive psychologist, I say, well, these two things should be equivalent. I don't know what I'm even researching here because I don't have a plausible reason to even think that there would be a difference. But nevertheless, I went and kind of looked at with some of the existing research that was out there and started thinking a little bit more about ways in which OER might be unique or distinct from traditional materials. And that's kind of where my world is met with OER. And that's kind of like my introductory field. We'll get into more of some of the stuff that I've done and we have been doing and I'll turn it over to Virginia to introduce yourself. That's awesome. Yeah, Virginia. Yes, so my background actually has some overlap with Phil's. My PhD was in educational psychology with a focus or a minor in cognitive science. And my training was very much related to how cognitive science could be applied to educational research. So I got trained not only in cognitive psychology, research techniques, but also for education research more broadly. And in my career development, as I started teaching classes, like many instructors, because I taught large enrollment, intro to psych courses, I was getting constantly bombarded by textbook publishers and was a little taken aback by it. So I very much get motivated from a cost perspective. Also talking to my students about how they flat out told me they could afford their books or they could afford food. They couldn't afford both. And when I heard about open textbooks as an alternative and I looked into it, I figured as somebody turned in education research and I've quite a bit of experience doing scholarship teaching and learning my classes, I should test. So I did, I found that the learning was relatively equivalent between the two semesters. The withdrawal rate was lower with the open textbook and student perceptions were comparable. So I didn't really intend to continue with that, but I happened to connect with John Hilton, who's a very big game in OER. Encouraged me to present my findings at open education and to apply to the OER research fellows. And I just became aware that it was such an important field and that it involved a lot more than just free books, even though the free portion is in and of itself pretty remarkable. There are quite a few people who are concerned that if it's free or if there aren't the access fees that you have with commercial materials, it can't be any good. And yeah, I remember having commercial publishers tell me like, well, if you actually cared about your students, you wouldn't use those open textbooks. So I said, well, let's see what the data say. And that prompted my focus in my research. A lot of my other main line is reading comprehension, specifically text comprehension, as far as how you make connections and how college students learn from reading. So studying textbooks has been an obvious area of focus for me given my reading background. So that's how I got here. That's so cool. Great. I love hearing the stories. And I think it gives folks a picture of kind of where you all are coming at this and kind of how someone might approach OER research. I'm noticing in the chat that we just have a ton of folks from all over the US and Canada. Una says about 80 people have joined in. So thanks so much for joining folks. If you're just joining us, we are working with Philip Grimaldi and Virginia Clinton and we are talking about OER Impact Research. I'm Nathan Smith. So I want to turn to Phil. You mentioned how you got into this area and then this key question that you were asking yourself. I mean, what's the difference between an open textbook and a commercial textbook, given that they're covered the same content, they're modeled off the same basic curriculum. What's the value difference between the two in terms of learning? So you've written this paper in the last year that was published in the last year that I think is really interesting and addresses this question. You want to talk about what is the access hypothesis and why should we care about it? Sure. And like you alluded to, this all kind of started as sort of like a brainstorming exercise or a thought experiment really, which was to try and really identify what are the possible mechanisms that might be underlying differences, a difference between an OER adoption and a traditional commercial book. And my first open ed was sort of like a learning experience trying to get a feel for what people are thinking, what ideas are kind of being floated around and probably above and beyond the biggest... So anytime I go to a talk on OER research or something like that, I'm always the annoying guy at the end of the talk that says something like, but why? Like what's the mechanism behind this? And I do that just because that's kind of how I was trained to think, which is always like why? What do we think is underlying this? As a cognitive psychologist, what we're trying to do is, I don't really care about necessarily how many things that student recalls. I want to know why they recall that so that I can leverage that in the future so that they continue to recall more things or they continue to get more questions correct. So why would somebody, or why would adopting an OER book produce any kind of difference than a non-OER one? And there's this idea that people kind of implicitly were telling me, which was that, well, it has to do with access. And when you look at survey data and you ask students or you even observe what they're doing in their class, you'll find a lot of students choose to forego purchasing a book because it's too expensive. So they get to their course, they see that the book is going to cost 200, 300 books. And like with Virginia was saying that even with their own students is, you know what, I'm going to, they're going to do what, well, they're going to do one of two things. They're going to drop the course and just not even try, or they're going to keep going and they're going to do the course without the textbook. And the idea here would be, well, the students that do choose to do the course without the textbook are going to have a more difficult time than students who do have that resource. And so if you assume that having the resource is beneficial, and I hope that it is, otherwise we wouldn't be spending this much time like building these things, then these students are going to be worse off. And then what this is going to do is going to produce a deficit in performance for students using commercial materials. And that by adopting an OER resource that's free or more accessible, you remove this barrier. Okay, so this seems like plot. Actually, like if you really think about it, what this boils down to is, this is just the difference between having a textbook and not having a textbook. It doesn't have anything really to do with OER. It's just, do you have resources available or do you not? Continuing with the static experiment a little bit. Okay, so essentially what we're saying is some students in this hypothetical scenario, so you would, you are in a scenario, we have a commercial textbook, some of your students aren't going to buy it. This isn't going to be 100% of the time, not every, you know, it's going to depend on the class, but some of the students aren't going to have access. Okay, so when you go ahead and you adopt an OER material, what you are doing is you're essentially filling that access gap, but only for those students who didn't that had that access problem to begin with. So hypothetical scenario, let's say 20% of your students aren't going to buy the textbook or they don't have access to your textbook. The other 80% of students in your class are fine. Obviously, we would still like to save the money because they're students and they have lots of other things to do, but they're going to not have those same learning disadvantages. So those 20%, then you do potentially improve learning for those students, but if you look at the whole of research on OER, we don't really make that distinction most of the time. We look at it as we're going to look, give people this OER book and compare it to this commercial textbook and we're going to kind of make this implicit assumption that it's going to affect everybody equally. But like I just said, in this hypothetical thought experiment, we actually kind of don't think that it's going to. And so the quest, the problem with that is all of those students who I said are not necessarily going to be affected are contributing to our measurement of the effectiveness of this material. And even if your OER adoption is really, really, really effective or even like moderately effective for those students that didn't have access, all of those students that aren't affected to kind of wash out that impact. And so in the end, what you could potentially be seeing is nothing. And then the paper that Nathan mentioned was basically a way of formalizing this idea and we ran a bunch of simulation studies to kind of simulate hypothetical scenarios like this where some students in the class don't have access and some do, and if we were to actually improve their learning, would we actually be able to see it? And the take home message from this paper was you might be able to see it, but the vast majority of studies that have been conducted so far probably would not have been able to see any effect of OER, even though it very well could have been effective. And in a nutshell, that's the access hypothesis and that's the kind of the reality that it presents to researchers in this field. Yeah, so I'll stop there and kind of turn it back to you, Nathan. Yeah, I think that's terribly interesting and definitely if folks have any questions about that, write them in the chat. I'll be recording some questions. We can come back to them. I think it's an interesting, we'll talk a little bit about implications of that, but I think that's an interesting result. And then Virginia, you just published this huge meta-analysis looking at basically research over a decade of research in OER and specifically on the impact on student learning. And so you kind of did the big analysis that Phil was kind of was projecting or looking, thinking about, but what were the results, the bottom line results that you found? Right, so just to clarify, I focused on post-secondary students. And I also focused on textbooks and I'm not sure if the students had access to the commercial textbook. So for example, there was one experiment that actually was my own where we gave students copies of the commercial textbook and we didn't include that. So what I found looking at 22 different studies across various types of institutions and various content areas and over 100,000 studies is that there were equivalent learning outcomes in terms of student performance, which was almost always grades, either the final course grades or exam grades throughout the course. And I mean really there's, it was a robust lack of a difference. So this is really reassuring for anybody that's worried that an open textbook is gonna hurt their students, but it doesn't really fit in with the idea that they're so students are gonna be helped by a textbook. And I can explain a little, I have some more conjecture than anything, or hypotheses for why that could be. I really wanted to look at student characteristics because I think that there are certain students who not only may not afford a textbook, but who would really benefit from one, more so than just attending class and being really attentive and being really responsible with their work, which is also a major contributor to grades. But most of the studies just not providing enough information on student characteristics. And I'd also thought about like, well maybe I could separate based on assumptions of the general socioeconomic status of the community, but that just really didn't work out as far as the various schools that were in it. I did look at some methodological issues. So sometimes the instructor differed between the open and the commercial courses, which instructors obviously not only vary in their teaching quality, but the grading criteria. I looked at whether or not the exact same assessment was used. So oftentimes if you change a book, you need to change your assessments because the terminology may be a little different or certain concepts aren't covered. And then if they incorporated student prior knowledge or achievement into the metrics, which I knew from personal experience, which was important because in my own study, I noticed that there were slightly higher grades for the course using an open textbook. But when I got their high school GPAs, I found that they also had slightly higher high school GPAs going into the course. So once that was factored in, there probably was no difference. And if I group studies based on those methodological characteristics, it didn't make a difference. So I also looked at course withdrawal rate. And I found that in 11 studies with over 78,000 students that there was an impact statistically on course withdrawal rates where open textbooks had lower withdrawal rates than courses with commercial textbook. And that was a pretty robust finding. It wasn't driven by any particular study that was included. And half of these studies, the authors didn't publish the withdrawal rate. Their study was included in the learning efficacy portion of the meta-analysis. And I reached out to them and asked if they could share their withdrawal rate to those who did were included. And there wasn't any difference between people who published their withdrawal rate versus people who didn't look it up until I had requested it. So that part matters because it makes it less likely that people were just trying to make OER look good. And there's all these, what we call file door studies with no effects that people were hiding or bad effects that people were hiding. So my thinking is for that is that, and this is also just talking to students and getting their take on it. A lot of times after their first semester, students will wait to see if they need a textbook in a class in order to decide whether to buy it. And there's actually, you know, survey and interview data that shows that a decision to withdraw can be based on the cost of that textbook. If they realize that they do need that textbook, that can be a reason why they would withdraw if they just don't have the money. Another reason is convenience. If they're able to get an open textbook to be able to get an assignment done, they may be more likely to stay up in the class or catch up on the class than if they have to go and access the commercial materials, which is just a more involved process than clicking download from the course learning management system. So interesting, I would say two things I was just really impressed with Virginia study about. One is the fact that you reached out to all of these researchers and got additional data. I can imagine the follow up and work that took, you know, a little bit of time. And then also, you know, I know, I also stocked people at open ed. Stocking is a is a valid research strategy. I think that's great. So I just think that was really great. And John and his, and in his presentation at the last, John Hilton in his presentation at the last open ed, definitely shouted out your, your study as kind of the thing. If you wanted to pick one to take around and show your administrators and say, look, this is what OER does. I think this is the study. So just to highlight that. So given these results and given what you all found, I mean, what does this mean for us? I think we've talked to a couple of things, but maybe if we could just bullet very high level kind of, what are some takeaways that we should as OER practitioners, folks that are out in the field, what should we take away from this? I'll take a stab first. And then I'm interested to hear Virginia's take as well. So when I saw Virginia's results from her meta analysis, I kind of had the, like, yeah, this makes sense. Kind of feeling, right? If he based on the access hypothesis stuff and thinking about, like, well, what are the possible net mechanisms in which we can recognize the improvements, this really makes sense that these would be equivalent. The other thing that is important to take away is that if we have some other hypothesis too, and this is a real hypothesis that's put forward by, like, commercial publishers a lot, which is that OER materials are lower quality than the commercial ones. And if you believe that, and that very well may be the case, then you would predict in that case that OER would perform worse than the commercial counterparts. And Virginia's study, yeah, pretty much essentially ruled that out. One of the things that Virginia did in her study was really interesting. And I'm so glad that she brought it to the community, which is the introduction of these equivalence tests, which are not typically done in educational research at all. This is more of like a medical research technique, but it's used in research when you're trying to, you know, prove to the FDA that your generic drug is the same as the commercial drug that's on the market. And so her analysis is essentially that sort of thing, saying like, here's this thing. It's essentially the same as this commercial component. I will say like, there's still just, you know, I think we can kind of all rally around this to a certain extent to say like, look, we're certainly not causing any harm. We're saving students money. There's really no reason to not to go this route. And so we're kind of chipping away at all of the traditional arguments against OER. There are still a lot of unanswered questions though and things that we don't know. We also don't really even know what usage is like overall commercial or OER otherwise. So if, if all students are really choosing, you know, even if they're buying a textbook, if they're not reading it, it doesn't matter. We give them a free textbook and they're not reading it. It doesn't matter. And so in that case, we would also expect no differences. So we really need to start, you know, we're really chipping away at some of these deeper questions of, all right, well, how is usage different? And how does that contribute to differences between these materials? And I think that's kind of where we're sort of at Virginia. Where are your thoughts? Yes, I was going to add onto that one. I have a reviewer to thank for me digging up the non-equivalence test and finding out there was a test of non-equivalence. That was a learning process for me in developing my, my skills on this project. So thank you, anonymous reviewer, wherever you are. And I, um, a textbook doesn't matter. I personally am not in that. I don't, I don't necessarily agree with that. There may be courses that that is the case, you know, where you can have four level courses where you can have collections of readings or, you know, courses where maybe having that isn't important, but I know that students typically like having a cohesive document that kind of guides them through the course that helps them follow along with the course material. Yes, sometimes they don't read it. And obviously that's not going to make a difference, but, you know, there is a correlation between how much they read the textbook and how well they do in the class. Like that has been shown in the research course. That can also be because they're more responsible in general. But I do think, yeah, maybe bias because I researched text comprehension. I think reading is good. And I think, you know, having a good text to use to, as an ancillary for the class or to use during active learning or to use as a reference can be really useful for students. One thing I've wondered though is, again, this is more anecdotal and we don't have a lot of information on how the instructors assigned or used the textbook. You know, and a lot of these reports have just simply said like this many classes use this textbook this many classes use this textbook where there really wasn't a lot of detail and where there are assignments for you to use the books that they have to write and the instructions that they have to take quizzes before class because they have, you know, just gone to class and taken notes and been fine. And, you know, I know that there are some instructors will plot out tell their students the first day of class, you come to class and you take good notes and you pay attention. You don't need to buy that $200 textbook. So in that case, you probably wouldn't see a difference either. Great points. So just to kind of summarize it seems like the research has shown first of all that we can pretty much do away with that old publisher argument that was made to you Virginia so many years that that that if you're not paying for the textbook can't be as high quality right that I that was actively harming my students was right right that you're harming this to your students by providing them with a free textbook. We I think it's also kind of cool just to see how open research educational research is impact is interacting with educational research at large in some ways maybe open educational research is really influencing that that's kind of cool. I think the emphasis to focus on learning and the instructional practices that are involved and then also pulling student characteristics that those are really interesting important features to understand the impacts of we are there's a question in the discussion in the in the chat window that I I want to flag we are going to talk a little bit about human subject research but the question specifically talks about using data and analytics on usage and user data. I mean I think that's a big question that's out there so I want to flag that. But before so we'll we'll get to it. The before we get there which want to think about as a lot of the folks here on the call or are kind of in administration faculty at community colleges some of them are coordinating national programs and things and and so how how can folks do set up research studies for themselves and what kinds of things should they be looking at I have a I have a list of kind of possible questions here that will look at but I want just maybe if you all can address this area of thinking about setting up some good research at at a home institution. What should we be looking at. In keeping with the tradition of I guess me going first I'll go ahead and time in alphabetical. I think that. Like I was saying before I think some of the areas in which we really need to start chipping away at are some of the. Possible edge cases or. Moderating variables such as like how we are is actually being used Virginia mentioned things like well are there differences or changes in the tests from when you change from a traditional book to know your book are the are the exam saying the same difficult. Friend but also you know how are instructors using these materials differently is there anything different about using an OER material versus a traditional one. I think we understand that there are certain legal affordances of OER versus commercial books if you happen to be in Jeff Siemens talk at open ed he presented some interesting survey results from instructors to more or less acknowledge that they use a lot of commercial. Commercial materials in illegal ways that are not necessarily afforded by the licensing but those publishers aren't really going after the instructors to sue them. Although interesting tidbit our local school district Houston independent school district recently was subject to pretty high profile lawsuit involving copyright infringement from a publisher so I think they you know that is certainly something to be worried about but I think in practice what's happening is a lot of the things like remixing and taking out materials from textbooks and putting into slides and making making your own custom materials that's afforded by the license of open materials everybody's just kind of doing this any way with commercial books so is there actually a difference between them kind of some interesting. Less opportunity for a lawsuit I guess. Right. They're not going to sue the individual instructor but I guess if they can go after an entire school district then that's fair game because that's okay I guess. I don't know those are my thoughts Virginia what do you think. Related to that you know when I did my study on the commercial textbook you know I had some spaces for open end of responses and one of the comments was the students said I should have told everyone that I could they could get a free PDF of the commercial textbook online I'm like yeah the pirated copy. That's right. That's illegal I wasn't no I'm not going to stop you but I'm not going to encourage you to commit a copyright violation but apparently there are faculty on campus who do that. So. It's quite common I don't think they even know that it's illegal I think or or like Jeff Siemens said they they think it's fair use or whatever. It's on the internet it must be okay to use. I'm not profiting off of it so. So yeah I mean it's a great point I mean we don't know I think the point you're raising is that there's so much we could do in OER research looking at the specific practices that faculty are engaged in in the classroom and I think it's a great idea that the traditional sort of research questions that at least the OER research group has put out there that I think people tend to group around or the you know look at cost savings look at the past rates and completion rates the outcomes this is the stuff we were talking about and then this usage question I think this is some of the stuff you all were just talking about sort of how is the OER actually used how are traditional textbooks actually used can we compare these these things and then the perceptions you know what what are the perceived quality of the of OER. We've also talked then about the you know the idea of honing on subgroups student characteristics that would benefit most from this and how it's implemented and then broader learning areas are there any other things you all want to say on these sorts of big kind of research questions that people ought to be thinking about as they're as they're looking at how effective their OER implementation program is on campus. Well I think one area too is to you know relate to the idea of instructors changing their practices is to look at if using OER has has prompted them to maybe be more reflective or to think about their practice just the opportunity to redesign their course and how that relates to their instructional effectiveness that may be a consideration or even just how much they then feel comfortable requiring use of a textbook because they know I mean I know I feel completely comfortable requiring all kinds of assignments out of the textbook because I know they you know have it easily accessible available to immediately download so rely on probably a little bit more as far as assignments and things like that for that reason and I also think enrollment intensity and persistence and degree completion a lot of the studies have been relatively short-term but does this make a difference in the number of courses they enroll in and I know there's been an there's an experimental study that's just out I think Carrie Cutler is an author but where they randomly assign students to look at hypothetical classes and they would they weren't you know choosing general and more classes if the books if the course is used OER as opposed to interesting I haven't seen that one I mean granted that's that's a hypothetical that's a you know controlled experimental setting but there is some work I know John Hilton did a study a while back where he did look at enrollment intensity and I think that would be an important question especially when it comes to encouraging University administrators to bankroll a lot of this work yeah that was actually a nice there's a funding model called the intro model for funding sustainability and it was a tidewater I think it was Wiley Hilton and some others I there's it's a good article I do want to fill I want to ask you this question as well and then I want to loop back cable green's got a really good comment in the in the comment in the chat window I want to look back to the copyright issue but go ahead any thoughts on the some strategies or things that people ought to be studying I mean I think that starting with the why in your research is super super critical so what why do you think this is going to have an impact and once you once you come up with a hypothesis for why you think it's going to be impactful design your research around that so you have a hypothesis that well when you adopt OER more students are going to have access to the book I wouldn't otherwise have access to it then you have a you can set up your study to really test that idea and you can say okay well let's look at the students who don't have access and when we give those particular students access does that improve their grades and then when you can design your experiment around that hypothesis your results are going to be a lot more convincing and they're going to be a lot more powerful than if you kind of don't design your study with the hypothesis in mind so that would be kind of just like a general strategy I do think you know from the access hypothesis idea looking at just exactly that the types of students who are going to be most impacted by that OER implementation and trying to do you can to focus your research there that can be very helpful to the research and it is a general strategy too I know a lot of folks here are coming from the community college community but there are lots of researchers out there like Virginia myself who are you know itching to collaborate on these types of things and so I would also say don't feel like you need to become an educational research scientist overnight to do these things yourself you know we are happy to lots of folks in my field are happy to work with you to to design a research protocol and help hone in on some of those questions and so you can do your study in a way that's going to be rigorous and well received when it finally comes out yeah it's really an ideal collaboration because you have one person with the research skills background and you have somebody with the institutional knowledge and knowing the students and having that teaching skill and it's just a nice combination of skill sets that benefits everyone and can produce really high quality research I know personally and knowing about my colleagues we'd be thrilled if a community college would reach out to us and I've had some people who don't have research positions or extensive research training who have reached out to me and I have collaborated with them happily. That's out that's such a great point let's I want to talk about that and but just to loop back to Cable Green's comment about copyright we were talking about the practices of people of educators you know open licensing various covers them Cable Green makes a good comment he says that you know when educators violate their all rights the all rights reserve copyright they not only put themselves and their institution at risk at legal risk but it creates legal problems down for downstream users so I think that's a I mean I guess a great point that that in fact you know whereas you know using pirating materials might work for you or your class you know and maybe you can illegally adapt and remix those things you can't redistribute that and you can't help benefit the community in a way that that pays it forward so I think that's a that's a great thing to keep in mind one thing Cheryl Collier comments also that you know Cengage apparently is is is sending out information suggesting that OER content is compromised because it's openly available online and that their test banks aren't compromised I think Una's response is quite apt that you know I mean there's some website I don't remember the name of it but you can go on and get that you can find anybody's tests I would say my things are always that every test bank question in existence is online yeah so I think yeah yeah either you can just immediately Google it or I know that you could get a check account C H E G G and by purchasing that you have every test bank question from every publisher yeah that's great I mean I think so we should keep that in mind you know that we're even though it's going to be kind of a bit of a game of whack-a-mole here we're going to get credit criticisms constantly about open educational resources and sometimes research can resolve those questions and sometimes common sense can um cable green also points out I think this is something everyone should look into is this the 10 week certificate copyright certificate program that Creative Commons puts on so that's a pretty pretty rigorous and thorough course that people go through that it does require tuition payment but it's a really worthwhile thing I've not done it myself yet but I I know some folks who have and they have gotten a lot out of it okay um let's talk about collaborative projects since y'all brought that up I just I think it's a huge thing um thanks Amy um the Amy Lagers um the you know I know I came from this in a because I come from a background of PhD in philosophy which where we don't do empirical research with our empirical research involves sitting back in an arm chair um and um you know uh so we don't so when when I was thinking about doing some studies of the OER program I'm I administer I didn't know what to do and I was just lucky that I happened to be in the open stacks um institutional partnership program and ran into Phil and Phil said I want to help I can help and I said excellent and so we've worked together and I think we can talk about that process um but let's start with just some of these questions this comes this goes back to uh the question that was raised earlier regarding data and usage data Karine Olam um asked this question about how we protect students data and and rights especially in the broader ethos of open some of those values so we'll talk to a little bit who wants to talk a little bit about human subject research and what we should be thinking about their limitations on that and then and then we'll talk I guess we can go through sort of how we would embark on collaboration yeah so I guess I see Phil actually has actively uh collaborated with the community college I'd say first first step is to talk to your IRB and your institutional review board at your campus because um if you're not in the habit of doing human subjects research um a lot of policies in terms of student data vary by institutions and sometimes state laws vary so knowing what the guidelines are for your particular institution is um important going forward so for example I was able to get this my students grades from my particular course because I was I was the instructor um and I was able to use those in my research yeah but I needed to do it and however for their high school GPAs I could only get the course mean a standard deviation I couldn't get the individual student information without individual sign releases so like that's an example where the student data varies a little bit depending on what exactly I was asking um but I could also ask for self-reported high school GPAs that was totally fine and don't be scared of your institutional review board I know people have uh they're there to help they're there to help I I was kind of unsure about it but they're working with them has been really great so Phil you were going to say yeah no I mean I was going to echo essentially what you both just said which is that um it's not they're not only there just to review protocols that you submit to them but also to answer questions about research ethics and things like that that you might have and navigate some of the legal issues um and so you can approach them and ask like when when do like when do I need to um get informed consent for these things or how can I um obtain that data and they'll help you navigate those waters um beyond that obtaining student records like Virginia said it really depends on your your use um and how you're accessing that data and it's going to be um it's going to be dependent on your specific uh research questions and um how you're using that data um so you know if you need to identify that data versus not identify that data that those are going to be totally different things um or getting an aggregated means analysis versus getting individual subject line data um but yeah so they'll be able to um help you with those uh issues um and then you know um with the collaborative projects like we set up with HCC and Rice University universities are rightfully very protective of their students data so HCC who are um you know the students in our collaborative project who are working with their students at HCC HCC has the privacy of their students um in mind and so they want they don't want to just hand over data at any any institution and so there was a long um uh process between our two institutions putting together a data sharing policy specifically exactly how the data would be used how we would store the data how we would make sure that students privacy was protected how would we make sure to um protect against potential um releases of the data and so we specify how we actually do that and then eventually sign an agreement and then uphold those agreements and so these things can't happen um you know it should not happen or should not be taken lightly we'll just put it that way um and it's really important that you make sure you know what you're doing and that you're doing it um with um the approval of a possible owners of the data yes good good question I mean if you're going to have any kind of cross institutional collaboration you're going to have to go through the process you're going to have to get to know not only that have IRBs that approve that accept the IRB review of another institution or do their own in conjunction with it you're also going to have to consult with your general council and probably some higher level administration who need to sign off on that stuff it can be a process but once you navigate it it gets a little easier um and it does just it really does vary a lot by institution I've had some institutions and this is for a different project where we're developing a reading test for college students I've had some institutions being like oh you don't need any kind of approval to recruit from our students if an instructor is willing to share the recruitment materials I'm like okay and I've had other places where they refuse to let us recruit because they value student privacy to the point where they didn't want students voluntarily giving their email addresses to us yeah yeah wow that is extreme um some uh yeah so we're we're about five minutes out so I have one last question I wanted to just maybe you look back to Kareem is asking again about um usage data I think typically usage studies have been in the past um through surveys uh that typically surveys of faculty is that yeah surveys of faculty or surveys of students and asking sort of how they use the textbook uh is that yeah like I basically said do you want to share before the M's etc etc um Phil I know you're doing I don't want to I know you're doing some data with open stacks because you have a um you have a um learning um sorry you have some some software that actually provides homework stuff so there's some of that going on anything you want to say just really generally about how you about data privacy kind of considerations yeah yeah and and and to Kareem's point or to clarify uh what Kareem is asking about a little bit more I'm speaking more of like from a theoretical perspective here are the things that we want to be looking at usage and how that relates to outcomes and and how maybe differences in usage between OER and commercial there are a wide variety of ways in which you could actually address this um I think you give a specific examples of like okay well we could possibly collect um user information such as clicks that would be one way of doing it self-reported data would be another way of doing it um all of these things you're going to have to kind of weigh you know what are the privacy and ethical considerations for doing that if you want to go ahead and collect actual user data like clicks and things like that you need to be doing things like making sure your people are informed that you're actually doing it they're agreeing to this they can opt out of doing it from your institutions or you're doing the data anonymously you need there's all there's a million different levers and switches that you need to be considering and uh ultimately like um oh and then the point about yeah how does this fit into an open ethos I think all of this ultimately comes down to um being open about what you're doing uh so there's a movement now in the research community called open research um open research initiative and uh we are currently starting our process of moving everything that we're doing in open stacks over to the uh open science framework which is from the open research initiative um which will eventually kind of show all the things that we're doing or currently working on give information about our IRBs how we're protecting user data that sort of information um let people know what we're doing what we're working on and building in less of consent mechanisms and things like that but it's a very very super important question a very important issue I totally agree awesome um so I want to wrap up here by just uh sharing with you all some links um that you can find some misinformation I will I'm linking to three the three articles the two that we discussed and then also John Hilton's review of the literature um that gets published this is his most recent publication in August of this year covering research from 2015 to 2018 um in the chat window I think there was some mention of the open ed group uh this is a great website and all great research group um resource uh John Hilton is the director of that group it's housed at BYU they um they invite the OER research fellows Virginia mentioned that um earlier they also have some great tools on the website the OER research toolkit which includes even like sample survey questions and things you could implement um Lumen learnings developed this adoption impact calculator that I think is really cool addresses some of the enrollment intensities type stuff that we're talking about the cost factor and then John maintains a live sort of um review project that kind of feeds into his public published reviews um so as far as CCC OER just uh want to encourage you to stay in the loop um get on our email list the listserv um and um I didn't mention this but we are soliciting um blog uh posts case studies and and also anything on the equity diversity and inclusion topics if you're interested in that please reach out to Una Daily or Liziata um and um then our fall webinar series is wrapping up this is this was the penultimate um webinar the next one will be December 4th uh looking at conference recaps and reflections on open ed and OER global um and then also just announcement about the open education global conference uh taking place uh in a few weeks in um Milan um you know I'm sad to miss it but very much kicking myself that I learned about it too late yeah definitely so save up your travel money for next year and come to OER global that's the open education consortium um that's uh um and if you have any questions about the information is here um thank you all for being here thank you to our guests I really appreciate the conversation and look out for the recording on youtube alright thanks y'all