 Hi, everyone. Good morning. My name is Regina Gong and welcome to the Open Ed 2021 conference. Today we are going to talk about our OER feedback survey from our MSU students, its impact, implications, and lessons learned. I am the OER and student success librarian at MSU Libraries, and part of what I do is I lead the OER program, including our OER award program. I also manage our MSU Libraries open textbook publishing via press books, and I provide professional development to our faculty, our staff, our librarians on open education and open licensing, among others. And I am also facilitating the open pedagogy learning community. Hi, everyone. My name is Jason Elmarigi. I'm the director of assessment for the College of Social Science here at MSU. In this role, I do all sorts of things related to program evaluation and assessment of student learning and success in the educational services and programs we offer. My research interests are based on asset-based youth development and student voice and agency. So our MSU OER program at MSU is now on its third year. I am the full-time librarian leading the program. And in my team, we have a complete cast that includes our publishing assistant for print and demand services, a copy editor, an accessibility coordinator, and our student employee who does a lot of our covers for our open textbook. And we also have the OER award program wherein we have $50,000 every year that we allocate as funding for our faculty to adapt and create OER. And we are also an institutional member of the Open Education Network. And this year, we are also part of the OpenStacks institutional partnership program. So our OER program goals are, of course, affordability and access. We want to encourage adoption, adaptation, and creation of OER and other low-cost materials as textbook alternative for our courses. And in our OER program, we also strive to provide technical support for our instructors to implement OER in their courses. And as we do now with our Open Pedagogy Learning Community, we aim to engage in new pedagogical models that leverages the affordances of OER and open education. So this is just a snapshot of our OER adoption so far. So I've been tracking that since fall 2019. And this is a cumulative number of courses, number of sections, and number of instructors that are using OER. So that would be fall, spring, and summer for each academic year. And as you can see, it's really growing as we go on with our adoptions or our program. We have now a cumulative savings of more than $3.2 million as a result of this initiative. In addition to the adoption rates, we wanted to reach out to students and ask them a series of questions through a survey to find out what their experiences are, both with prior textbooks as well as their experiences with OER resources in their courses. And then our goal here is to, in the long term, have an understanding of does OER adoption improve student success? So to do this, we are fortunate enough to have access to institutional data, which allows us to identify all those students who are enrolled in those courses that Regina had previously identified. So we were able to identify those students as well as obtain their email addresses so that we could survey them directly. And so to do that, we built a 12-item survey in Qualtrics. And then with that, we were able to pair not only the survey data, but we're also to leverage institutional data to have a better understanding of how student background in terms of demographics would play into their results or if there were relationships. And then down the road, we'll be able to look at not only course outcome, but we can look at other metrics in terms of time to degree, persistence, and graduation rate. For that survey, again, we have a 12-item survey as just a sort of a visual diagram of what the survey contains. So we have two threads of information that we asked about. We asked about traditional textbooks in terms of what are the costs? How much have they spent in the past for traditional textbooks? Have there been any impacts related to those costs? We asked them whether or how frequently they purchased those textbooks. And then we also asked about open educational resources. Again, these are students enrolled in open educational resources using course. And so we asked them about the quality of the material within those resources in terms of accuracy, the clarity of the data or information being presented in those resources, and the extent to which they believe that there was a positive impact in the learning within the course. And then for both threads of traditional textbooks and OER, we asked about frequency of use and ultimately, what is their preference? Do they prefer traditional textbooks or do they prefer OER? And if they had a choice of identical courses, would they choose the one within a traditional textbook or an open educational resource? There's a lot of data here, so I won't go through all of this, but this is all the sample size in terms of the students across all those sections. Enrollment across all OER identified courses for fall 2020 was 8,739 students. Spring, there was 8,353 students. And you can see in terms of unduplicated student samples, so we had students who enrolled in not just one, but two or three different courses that used OER resources. So we deduplicated that sample for the survey just to make students' lives a little bit easier in terms of survey fatigue. And then students actively consented to complete the survey or not. So you can see that those consent rates. And then we have a small number of students who did not consent to complete the survey. And then as you can see here, we have a large number of students who did not respond to the request to complete the survey. Ultimately, what this means is that we have final response rates of 1,038 students in the fall and 218 students in the spring. I do just want to take a moment to talk about the spring results because that is incredibly low response rates. What we found is that in our second semester of being nearly fully remote here at MSU, that by the end of that spring semester, that academic year, students were burned out from completing surveys or online anything. And so while this is not an ideal response rate, we still use the data. We did take a look at the extent to which there were differences between respondents in the fall and spring. There were some minor differences related to things that you would expect in terms of students who would be more responsive to taking a survey. More females than males were likely to take the survey. Students with slightly higher GPAs were more likely to complete the survey. Students who came from an Asian background would be more likely to complete the survey. So on all of there are some differences there, but we wanted to move forward with maximizing the most amount of survey data that we could. And then in our question and answer session, we can talk about moving forward how we might increase our response rates and whether or not we might be missing groups of students. I wanted to present some of the sample results from the survey. Again, we have 12 items and so we don't have time to go through all those results, but here in a nutshell is some of them. One of the first questions that we asked was how often do you purchase required textbooks? And so you can see down here, these were the response options. Students can say, I never purchase required textbooks to rarely about half the time often or always. You can see the distribution here that most students about a third or that they often purchase their required textbooks. But what you see here is that there's a small proportion that say that they never buy their textbooks, required textbooks, or about 20% to say that they do purchase their textbooks either never or rarely. So we do have students who are not purchasing their required textbooks. The other thing that we wanted to look at is leveraging that institutional data. So we did a quick analysis looking at for those students who report that they purchased their required textbooks about half the time or less. So it'd be this group of students here. We compared all students. And so across the full sample, 40% reported that they purchased the required textbooks about half the time or less. You can see how this varies. URM refers to underrepresented minority. But you can see here that it's fairly consistent. Underrepresented minority students have slightly higher or slightly more likely to purchase about half the time or less. But that difference is quite a bit bigger for students who identify as international. I don't believe any of these results are statistically significant from each other. So once we wanted to ask students, to what extent does the cost of their required textbooks, have they caused any of the following? And so this is just a yes, no set of responses. And so students were asked, has the cost of required textbooks annoyed you, stressed you out, caused you to not purchase the required textbook because you couldn't afford it. Caused you to cut on non-necessary expenses. Things like entertainment, movies going out instead of eating in. And then we asked them, have you had to cut necessary living expenses or delay necessary expenses like paying your rent? And so you can see the percentages here across all students. At the very bottom, you can see that only 2.9% of those students reported that the cost of textbooks did not cause one of these feelings or behaviors. Majority of students at three quarters identified as being annoyed or being stressed out by the cost of textbooks. But you can see here, nearly a quarter of students reported that the cost of textbooks had caused them to cut back on necessary living expenses. So that's quite a few students in that sample. Yeah, and so this are the quotes that our students told us in our survey and they really appreciate not having to pay for a textbook. This one comes from an international May student taking Math 314 course. And as you can see here, they are very grateful for the fact that they do not have to pay for each textbook that they need to. Yes, and this one too is, I mean, I'm not going to read this for you, but it really goes to show that what our students are experiencing in terms of how the textbook is being used. So, you know, sometimes they feel that it's not necessary to use a textbook to get an A in the course. And so we also wanted to take a look at how its impact may have affected students differently or students from different backgrounds differently. And so again, to the left, you can see that what the question was has the cost of required textbooks or course materials cause you to cut back limit make late payments on necessary living expenses, food, housing, transportation, and the like. And so it's just that one option. And you can see here again these set of students, all students about a quarter as I identified before, but you can see as different types of students had differential responses here. Pell eligibility, if you're not familiar with it, it's, this is a family income based financial aid package. And so we use this here at MSU as an aproxy for social economic status. Students who are eligible for Pell grants tend to come from family backgrounds that have fewer resources. And so you can see here that I, it would make a little bit of sense that I, those students who come from lower means of economic means are disproportionately impacted by the cost of required textbooks. But you can see that it affects students from underrepresented minority status, as well as first generation status I differentially. And again, this are some of what our students are saying in terms of the impact of the cost of textbook on them. And as you can see here, we have, you know, we made it so that we have a profile, you know, of the students who are giving us their feedback. So, in this case, we have Hispanic female first generation and Pell eligible students and, you know, based from the feedback, it has really helped them this semester. Yeah, so our next slide, it's a lot to read, so I'm not going to read it. And yeah, so this one is all, you know, too familiar, not having to pay for textbook really relieve stress on on them and they wish that they, you know, more classes are offering courses that are using OER. Another section of the survey we asked students, how do they use their OER resource in their course and we, we asked three questions about how they use it so we asked if they print the entire textbook, I have it printed by an external printing company. Do they print out sets of pages or chapters on their own. And then do they copy and paste electronically parts of the textbook into a separate notes document or a study guide and you can see the proportion here, about 10% report that they have the open educational resource printed 15% will print it out. And 31% report that they copy and paste and so I think that what we've what we take away here is that part of the utility of these open educational resources is not that they can be printed and used as a printed resource but they're they remain an electronic resource students leverage that the digital copies of that for their study benefits. We next asked to what extent they, we asked about the quality of the resources in their course. And so, the three questions we asked are, I was the textbook presented did the textbook or open educational resource present the information accurately. I was it presented in a clear and understandable way. And then I did the open educational resource support your learning of the course material and you can see across the board very high percentages of students either agreed or strongly agreed with these statements. And then we asked a second question about quality, comparing the open educational resource to their experiences with traditional textbooks. And this is the chart on the left. And so we asked them, would you rate how would you rate the open educational resource was it worse than other textbooks or those required textbooks. Was it about the same as those other textbooks, or was it better and you can see here, majority students reported 60% reported that it was about the same and then combine about the same and better about what is that 6789 94%. So majority vast majority of students report that their open educational resource was as good if not better than traditional textbooks. And then we asked students if they were given a choice of two different sections of a course that had the same material or same instructor and the same course, but they had a choice and equally desirable time slots for those two choices, which would they choose the course with a traditional printed textbook, when they choose the course with the we are resource the open textbook or would they not have a preference and as you can see figures students do have a preference, they would prefer their the course with a we are resource. Yeah, and in here are students are telling us feedback regarding the resource itself. They love the book. It's very helpful it's organized well and it's very comprehensible and yeah it's just as good if not better than any of their the textbook commercial textbooks that they use in their other classes. And this one is from a female URL eligible student. Yeah, the course using an OVR for this course is really very beneficial for for her learning and education. When we when we step back and take a 30,000 foot perspective of the survey results, obviously we've not presented all the results here, but I hear some of the takeaways of what we learned from what students had to say about their experiences with traditional textbooks as well as their open educational resources. Students, I clearly identified that traditional textbooks are expensive that I even when they're required for a course they're not often use or that they're not used fully, and especially for older students, or students who are in their third or fourth year. Recently, they're less likely to purchase those textbooks they identify alternative means to access the information or have realized which courses I you can still do well and without using purchasing that traditional textbooks, students are not only stressed but they're irritated by the cost of traditional textbooks and part of that is because they're they're having to make cuts in other aspects of their life to in order to afford those textbooks, and then I often related to increasingly not purchasing textbooks. What we saw is that students, especially as they progress in their academic career becomes sophisticated consumers of those textbooks they do the cost benefit analysis of, you know, do I want to spend $250 on a textbook or, you know, am I will I farewell without it and increasingly again they do not purchase those textbooks. And then in terms of the open educational resources as students identify as as them having at equal quality or they're better, they're easier to access. Again, students often talked about the convenience of being able to use the find feature within an electronic resource that can search for a key term, or search for a particular term, especially if that resource is allowed during that test taking, and then there's more functionality within those open educational resources within the digital domain and there can be added functionality in terms of widgets or links the videos. And so students did identify those as helpful for their learning. We're going to continue administering student surveys. We're going to create and implement a faculty survey to see what faculty have to say, and then long term goals are to take close a close look at how open educational resources impact students metrics so things like, you know, not only course grades but their cumulative GPA, are they more likely to persist because they can afford college and then, of course that relates to graduation rates are time to degree. And then of course course drop rates, we did hear from one student in one of the quotes that state dropped one course that had a traditional textbook to take one with an open educational resource. And then of course, as we're doing today, we're going to continue disseminating the results to stakeholders, not only academic communities, but also students, students, administrators, and then folks in the field. Yeah, so thank you everyone. I hope you learned something from my presentation will be happy to answer any questions as we all collectively view this presentation. Thank you so much.