 Welcome everybody to the Community College Consortium for Open Educational Resources webinar on measuring the impact of open education. My name is Matthew Bloom. I'm one of the professional development team for CCC OER. And I am very excited to have a panel of folks today here to talk about this topic, which I think is kind of a hot topic has been for a while so we'll get into the discussion. So I want to let everybody know first of all to go ahead and introduce yourself in the chat so that we can, you know, kind of get a sense of all who's, you know, who's here. And we have quite a few people have who have already logged in so that's really fantastic first though I just want to real quickly run through the agenda. I'm going to do a real brief overview of what CCC OER is. And then we will introduce our panelists and have a discussion. The next section at the end actually I want to just let everybody know as participants we will have time at the end where you'll be able to ask questions. You can also ask questions in the chat and we might address those as we go. Also at the end we are going to be asking the audience if they have any particular strategies or approaches that they're using to collect data and to try to measure impact beyond cost savings. And because, you know, one of the things that we all probably can agree on is that we don't have the answers and so this is really a great opportunity for us to come together and try to, you know, talk through some different strategies and share some ideas about how we can look at the impact that open education is having with our students in it or institutions and even beyond our institution so without, you know, further ado, the CCC OER mission it's a nonprofit organization that is committed to expanding awareness and access to open educational resources, high quality, especially supporting faculty choice and development, fostering regional OER leadership and improving student equity and success and one of the ways that we try to do that is by providing these kinds of webinars these professional development opportunities for the public not just for our members. Our members are we have quite a few members and actually as you can see we're all over the United States and even a little bit beyond that northern border there. The CCOER is a is a kind of like the North American node of the open education global organization, which has a variety of different projects that are, you know, initiatives and nodes across the world. So that's something to keep in mind as well. I just want to go ahead and move right into this I'm going to try to I had some questions that I'm going to ask I've shared them in advance with the panelists and we've discussed a little bit of how we're going to do this. I'm going to try to stay out of the conversation as much as possible just maybe kind of help guide it along because we really want to hear from the, from the experts on this we have Ryan Corsting here. And Michael the Magna and Rebecca Ortiz and I want to go ahead and just let each one of them take a couple minutes to introduce themselves and provide a little bit of background or context as to their experience with measuring impact of OER. So Ryan, if you want to go first. Yeah, great. Thanks Matthew. My name is Ryan Corstange. I'm a director of academic affairs currently at the Tennessee Higher Education Commission. And so what I do here a number of things but I manage textbook affordability initiatives across the state. So as as we'll continue to talk about this conversation of measuring the impact of OER. I'm thinking specifically right now about measuring the impact and communicating the impact of OER to folks who sort of aren't normally on campus and who aren't normally thinking about higher ed in the same terms that we're thinking about higher ed. So, hopefully that will add some interesting components to the conversation before I was at the Higher Education Commission here. I was a faculty member at Middle Tennessee State University. I did that for about a decade. And in that time I moved from, you know, OER adopter in my class to faculty advocate to managing a program of OER adoption and training and did a lot of data impact measuring, you know, through that, starting with the real simple, how much cost are we saving students and then moving on to some more complex measurements that I'm sure we'll get into. As we talk today. Great. Thank you very much. Michael, do you want to go ahead and introduce yourself. Sure. Good afternoon. I'm Michael Magna. I'm the information literacy program and library service coordinator and a professor of library services at Delaware County Community College in southeastern Pennsylvania. I have led my institutions all text committee and our OER initiative for many years, and looking at ways to, you know, bring an adoption of open textbooks and OERs into the classroom through grant funded projects as well as funded initiatives institutionally. Recently I served on an advisory and review board for a statewide grant program that encouraged and supported faculty and academic staff to create, adapt and adopt open educational resources and zero course materials in the state of Pennsylvania for higher education institutions. And I'm currently on the executive committee for CCC OER, and I'm chairing the research and impact committee here. I've spent many years measuring the impact of OER at my own institution through grant programs. And I always like to have a discussion about the data that's required internally and externally and how that can inform any decisions that will make. And hopefully we'll bring that perspective to our conversation today. Thank you so much and Rebecca. Hello everyone and thank you for this opportunity to join the space. My name is Rebecca Vasquez Ortiz and I am originally trained as a developmental psychologist with emphasis in multiculturalism and data analysis. It's kind of an interesting match, but I started my journey in higher education as the daughter of an illiterate mother and grandmother and so it's so exciting for me to be contributing in spaces where typically you don't see people like me. I'm very interested in affordable processes for those entering higher education as well as community empowerment models that center cultural wealth. Aside from those areas, I am a proud member and advisor for Arlo, the regional leaders of open education, and I hold at my core a passion for both open pedagogy and dual enrollment. Thank you again and I look forward to our conversations. Great. Well, okay, so I guess we can just go ahead and get started here. You know the kind of the premise of this of this webinar today is that you know there is has been this movement, not necessarily away from calculating cost savings or focusing on cost savings but at least expanding the focus of the measurement of our impact so my question is, you know what brought about this desire to shift away from just gauging cost savings as a measurement of the impact of open education. I mean I think maybe we can also kind of address the, you know the role that cost savings still plays, but nonetheless I'm just wondering what from your perspectives really has brought about this shift. I guess I'll start. From my perspective, I think that the cost savings as a measure was always a limited measure and it was it had a lot of short term vibrancy. But but we are users and adopters always knew that there was more to we are than just saving students money. And so what has happened I think is that as more as we are has expanded. As people start using the materials in their classes students and faculty alike, start to see different value of we are, and then really get into the position of being able to communicate the value in different ways. There's a different phase that indicates the shift away from cost only or cost savings only the other is, you know, publishers have changed their cost model so that in a lot of ways you might could say that they've stole some of the talking points from the old we are movement and they've, you know, it's about five years ago when costs when we were talking about $1500 a semester we're now talking about four or $300 a semester for students. And so it's by necessity shifted the cost conversation. And we've had to think about the value of all we are past just cost savings. Yeah, so, so maybe that's where I'll leave things and let the other panelists jump in there. I think we can all agree that cost savings is an important metrics for measuring the impact of, you know, open content adoption at our institutions given our student populations at community colleges. And like Ryan was saying I think it provided a good splash a good way of gathering people's attention. Look at all this money we can save our students through we are adoption. And that's not the only issue at play here so the financial barrier that many of our students confront, you know, is is all too real and requires that we continue to measure cost savings, because it becomes an access issue. But really, as we see greater adoption to open content. It's time to look at other metrics such as academic success and the role that open access course material plays in that and really gather a fuller picture and really the overall impact for students academic experience, you know, and so we have to start thinking about other metrics that we can use thinking about student performance and academic success in the classroom impact on issues of retention and that is always a big issue for us at community colleges, looking at quality of material, thinking about diversity equity and inclusion issues, thinking about sense of belonging, and how students actually will see themselves in the course material, and looking at issues of teaching and learning practice. So I think all of this really will provide a better picture a better overall picture of the impact that we are has on our students and at our institution. And so while costs will always be an important measure. I don't think it's the only one, especially as we're trying to bring more people on board for this kind of initiative. Thank you, Michael for leading us into that that area of the discussion and I wanted to add that, although, of course, we all want to make education more affordable for students and and free up vital resources for basic needs. The idea of low cost or saving money really is arbitrary. Especially as we look across the state that I'm from which is California, given that the central area or the Central Valley of California is occupied by agricultural workers as well as various tribal nations across the state so if we look to cost savings. Those price tags, those price tags may be irrelevant for those who, you know, don't have access to clean water, right or safe soil. So, an important point in looking at open pedagogy helps a step beyond that model of economics that disproportionately impacts students of color and others marginalized, and allows us to step into spaces that allow for the community creation of knowledge and the transmission of that knowledge. And so as we look beyond cost savings to content in pedagogy and invite community members from the broader community as well as from our student population and and faculty in, we can begin to kind of unpack some of the disproportionate data that we see continuing to plague our most marginalized students and so again for me, cost is absolutely relevant, but the need to take a deeper dive into what it means to provide education and critical thinking content to all students is really imperative and to measure that through data analysis and disaggregation of data is even more so important as we start to move and and look for for ways to truly engage on those who have been silenced or or marginalized. Thank you very much. I think that, you know, this, I appreciate the comments in the chat about the both and approach to this you know we certainly do you know we had we very consciously made the decision that you know we didn't want to approach this as you know, cost savings isn't important because clearly it can be and it isn't many situations and continues to be so that's really great and some of the, some of what you were just talking about Rebecca when it comes to looking at the community creation of knowledge and and the role that open pedagogy plays in that and trying to measure that we definitely want to get. We have a chunk of time later on where we'll focus on that I just wanted to focus one more question that is related to the shift away from cost savings and that's specifically on the institutional initiatives that we have going right now. A lot of them may have been initially kind of founded on the idea of saving students money and full transparency. I did I introduced myself by name but I didn't say where I'm at I'm at the Maricopa Community College District and the Maricopa millions project and started back in 2013 was called that because they wanted to save students literally millions of dollars $5 million in five years and they were you know it was successful and it moved on but it was after a period of time we realized that you know that we wanted to do other things so enough about me the point is is that what kind of impact does this have on your institutional initiatives if you are making this transition where you're saying okay we're not just going to focus on cost savings anymore we're going to look at all this other stuff are there are there drawbacks to that or are there like is a positive side to that or it's a mix of things what are your thoughts about that. I guess I would say you know like funding has requirements and so if we have these initiatives that are funded on the assumption that we're saving money for students and we're marking the money that we're saving we have to honor those agreements I mean that's just like a starting point, but also we can do more than that and so certainly leaning into the non monetary value or non cost savings value and measuring that in of open and we are initiatives is valuable I think I think what it points to is a different future for the open movement and potentially an exciting and really interesting future for the open not that the current future is bad, but it points to new vistas that we can sort of get into where we're talking about things that we understand to be critically important the quality of learning the quality of materials, the stuff that happens in the classroom of Becca mentioned pedagogy earlier, right like that's where this all goes, so that we can get to vital learning quality learning that's truly transformative for students that for all students, you know, and, and that's where we get by moving past just measuring cost savings, I think. I like to chime in on an add on to what Ryan said and appreciate that that perspective on moving beyond cost savings and, and I think recent or something that I brought up briefly in, in what I was saying earlier, which is more of an economics based approach process and so for, for my institution as well as others. It's almost become a bit of a predatory practice of publishers coming in to, you know, present low cost or no cost originally to set set us up for kind of buying into, you know, the business model I think that keeping that in mind, especially as we're able to bring in more faculty and administrators and classified staff into the process of creating open pedagogy is particularly important in that not everybody realizes when they get trapped into one of these types of publishing types of systems. And so, for me, I think that means institutionally, as well as at a broader level, we have to establish this understanding of going beyond that so that as people might get pulled in to the idea of low cost or zero cost at the beginning, and then later it becomes a bit predatory. We can, we can help kind of shift people's energies away from that and focus on content that really is helping to empower students. I couldn't agree more with that comment. I think one of the important things we need to consider, especially if we look at issues of adoption at our own institutions. Is that there is a benefit to focusing on cost savings right external funding state legislatures, probably are interested in seeing that, but it may not connect with every faculty member so if we're truly looking for adoption across the board. We have to speak to faculty members who will be adopting those course materials. And Rebecca I couldn't agree more you have these traditional publishers coming in with these new models. And that speaks to a lot of faculty members because they're seeing the ancillary course materials that are included in that package, and they're seeing that value there for not only them as instructors, but for their students. And so we need to shift that conversation away from just cost savings, or what the publishers can and cannot do with it for you as an instructor, and really look at those other issues right. And so think about what is the strategic initiatives at the institution how does this tie in, and really trying to speak to faculty in terms that they'll understand so making that connection about, you know, the quality of the material that's being adopted, because it's always one of those sticking points for a lot of faculty when they're looking at adopting at large scale, talking about student performance and academic success in the classroom, and really having that data ready to have that conversation to talk about its impact on teaching and learning practices, even this move to to enhanced and more remote learning. How does this fit into that course that learning management system how does it easily fit into that course, speaking with faculty administrators issues of retention and completion. How do we encourage our students to ensure that they're registering for additional classes. How do we ensure those students are going to complete their program. And so being able to tie in the adoption of open content and how that facilitates retention how that facilitates completion is so important. And so these are major questions that we need to ask ourselves when discussing what kind of metrics so we're going to use what kind of how we're going to measure this impact. It requires that we consider the data that we need internally for these conversations as well as externally, because I still think there is a value to cost savings for a lot of external constituents and stakeholders. I know in the, the statewide grant opportunity I was involved in that was a major draw to talk to the legislature to say, here's the cost savings across the Commonwealth of Pennsylvania, but that doesn't necessarily speak for our faculty who are looking at these other things. So we have to make sure that we're addressing them. Yeah, if I could jump in there I think that Michael makes a lot of really good points. One thing I wanted to add is that it seems to me that when we're thinking about the impact of we are the impact looks different in different communities or groups or whatever so when we think about students the impact on students, there is a big cost savings impact on them, but also hopefully they also they get access to the materials on the first day, they get some flexibility with the materials. And hopefully if the materials are done well it transforms and provides different opportunities for learning. That's a really different set of outcomes or impacts, then there is for faculty. I remember sort of the first semester that I moved towards using an OER book in the classes that I was teaching. I moved from a publishing publishers book that had test banks and PowerPoints and all the ancillary materials with it so there was an additional cost for me in terms of my labor I had to recreate stuff I had to think through things, but the benefit what I got out of it that I wouldn't have known on the front end the benefit was, I was more connected to the content. Thinking about how it could be taught and how students were learning it. And so it really allowed over that semester it became a different instructor and I thought differently about what was happening in class, how students were using those course materials. And then I had better relationships with the students which was completely invaluable at that time. That's a different set of impacts and factors then you even get to when you think about campus administrators who are looking as Michaels mentioned several times, we're looking at retention, who are looking at persistence who are looking at course completion. And that's a different set of metrics entirely from people at a statewide level or at a legislative level who are looking at different factors. So I guess the point that I wanted to make here to sum it all together is the data that we collect can be used to tell multiple stories about the impact of oh we are because it is a rich impact. There is a lot to the impact of it so we can tell multiple stories with the same set of data cost is part, but it's not the only part. Well I want to say thank you very much this is going pretty well so far we see the chat is really been pretty active recently I wanted to read Paul's comments as cost savings go well beyond student textbook savings. I wish the cost savings conversation shifted to a bigger picture including additional student factors beyond textbooks cost savings to the institution, and to the government, rather than moving on from cost savings it might be better to take a more holistic and comprehensive approach to cost savings. I think that's a that's a really good great comment I think one of the things that one of the resources that we had looked at was the Midwestern higher education compacts resource on returning return on investment when it comes to open education initiatives and so that might be to all put a link in there later if I don't have it handy right now but that is something that that might be worth looking at but I think that that's a really great approach and I'm wondering if anyone on the panel has any thoughts about about that idea of looking at kind of like not just the cost savings of the textbooks but like still thinking about cost savings but more broadly. Let me jump in first on that. I mean my thought is. It seems to me that we need to think about costs that are monetary and non monetary in a holistic picture and and there is cost saving that goes past the students. There is cost on the textbook there is revenue that's recouped, because the data indicates that students who use we are in classes take more classes or enroll in more classes first semester or continue in their path a little bit farther than other students. It's not the new gains but I think that it's important that we catalog the costs clearly. Right, so we have, as I mentioned, faculty have different work associated with using how we are sometimes and sometimes, you know, I was able to make it work because I had a three course assignment and an administrative reassigned time if I was teaching five classes or if I was adjuncting and teaching nine classes between three institutions. It might have been a different factor entirely and the work might have been impossible or the cost benefit analysis might not have worked out in the same way. So, yeah, I think that it's valuable to think about the costs holistically and see where that gets us. I also think it's important to think about the benefits holistically because the benefits go well beyond the use of instructional materials if you think about our current system. We have expiring access to limited term subscriptions to institute to instructional materials, and that's not the best for faculty or that has a cost for faculty and students and administrators alike. And I think it's important to catalog all that stuff. You know, I wanted to chime in, Ryan, on on some of what you were kind of looting to at least that I felt is that and that Paul brought up in the in the chat around considering a broader perspective on what cost savings actually is. And, you know, as from my own personal experience as a student who received full financial aid as an undergrad and a graduate student. Of course I always appreciated free textbooks, but some of the initiatives and I think we've already made mentioned to this both in the chat as well as in in the panel spaces that some of those those are built into tuition right so even though we say oh we're saving students this amount or or or that and believe me I have also, you know put out that information when when I save students, you know, 10s of thousands of dollars. And as we look at at again, our most vulnerable students, those those debts that students acquire in the process of oftentimes just getting a career certificate or an associate's degree or a bachelor's degree, follow them for their entire lives. We build those types of practices into financial aid that students oftentimes have to carry right the burden of what appear to be free textbooks actually go on to follow people in terms of their credit, they're buying a building later and so I really appreciated Paul's comment on the idea of really looking at what it means to save students money, because just the idea of leaving higher ed with its incredible debt that has kind of been also covering the costs of instructional material and instruction. You know it's highly problematic and we even see the federal government trying to approach an understanding of that so I just wanted people to reflect a bit on what it means to appear free but yet carry an incredible financial burden. So if we think about how we access these open materials right there there's going to be a cost there in terms of technology in many cases and so we think about our student populations. There are always issues of reliable internet access in the home issues of device access. These students and I could speak personally, I know are taking classes on their, their smartphone instead of on a traditional laptop or desktop, and that creates some challenges, and also thinking about issues of access to resources at the start of the semester is a couple weeks in so if we think about that financial aid distribution that often happens it's not necessarily at the beginning of the semester but a few weeks in. And so students are required are dependent on that federal financial aid to pay for course material. They're not going to have it when they need it and that's at the beginning and we've all seen those studies that talk about that right, the importance of students having access to this material. I think there's a number of different ways we can look at the cost. And I think it's a very important way to examine this. And I also think we have to think about the other metrics we need to use so that we can speak effectively to all the different constituencies, both internally and externally to ensure that we're getting the maximum buy in. We jump in right there because you, you, you pretty much prompted the next question which is what metrics might we consider we've already been talking about this a little bit anyway I mean we got ahead of ourselves quite a bit but that's fine because that's really the core of this conversation really is, is what is it that we should maybe consider focusing on and how do we go about doing that now that's a really really broad question I understand so I have at least 15 minutes dedicated to this right here. So, if you want to go ahead and just jump in there with any thoughts. We, there's a lot of different ways that we could approach that question but what should we be looking at. You know for me I talked about it or rather we have to consider internal and external stakeholders what kind of data we need to speak with each of those groups, whether it's a state agency, a grant or funding agency, we have to consider what, what they're looking at here. Think about internally are we talking to administrators, are we talking to faculty are we talking to staff externally as we think about the community that's part of Community College, thinking about our students and their families. How do we speak to them about this. And so it's going to require that we really stop for a minute and think about what data we need to collect and more importantly why we're collecting this data. And so the story does it allow us to tell, you know, for me, I'm looking at it from a pragmatic standpoint of a Community College and really it's it's that student performance, academic success issues, issues of retention and completion, you know, quality of materials, looking at open pedagogy and professional development that's related there. And so, as we're thinking about this data, it's not just going to be that traditional data or quantitative data we have to also think about the qualitative data so our students are able to tell their story. If we think about, you know, a sense of belonging, how the course material reflects our student populations. So that qualitative data that focus group. We also have to think about the methodologies that we're going to use when we collect this data to make sure that we're able to have a reliable data set to speak from. You know, this is a conversation that I've been having recently about course marking so how do we ensure that we're getting the most out of our data sets. You know, so our do institutions, mark their courses, does it allow for easy quantitative data to be pulled out of our systems. And that's where I kind of start that conversation is really thinking about what data we're going to collect and why we're going to collect it and what story that's going to allow us to tell in terms of those different areas. Let me add some thoughts. So I think that what we need is we need a set of metrics that allow us to talk about student learning in the classroom. And what is challenging is that we have a lot of proxy measurements for student learning GPA course completion rate persistence right these allow us to sort of talk around learning, but don't always talk about equal learning. And so what I think we need to do is think creatively about how to represent student learning that happens in the classroom in open classrooms. We need to think about different ways of representing those and using the that data to talk about the impact of open. And so what that I think means is a lot of qualitative stuff. Maybe we can talk to faculty members a little bit about the pedagogical moves that they're allowed to make because of open resources that they wouldn't be allowed to make without open resources, or we can talk to students about the way they find themselves and find their learning in open resources that they haven't experienced using non open resources. Or we could talk about the different types of resources that can be used in classes that are using open resources versus the ones that are using standard published resources right because I think what we what we have the opportunity to figure out and I don't have all the answers here I'm not sure that anybody does I hope that somebody here does to be clear, but I think what we have the opportunity to figure out is a different way of cataloging and measuring student learning that happens in the classroom because at the end of the day I think that what we're actually concerned about is learning for all students and for every student and quality learning that impacts their life for for a long time to come. And so that's the set of metrics that I think is the most intriguing to me. Yeah, we can talk about some of the, there's some quantitative metrics we can use to sort of get around and triangulate towards that. But I'm most interested in those rich learning centered metrics that require a little bit more messy complicated quantitative data. And Ron, I think that brings up a great point about cost right because that kind of data collection is going to have a cost to it. But that the simple quantitative collection, you're running a report in your system. Hopefully it's set up well so you can, it's auto generated that qualitative is going to require a lot of work and a lot of effort on the institution to not only identify study participants whether it's students, faculty, community members, but then to actually go and perform that focus group that interview. So that would be another area that we probably should discuss when we think about that kind of that methodology. Yeah, yeah, and let me add another layer to that. I mean I think that there has to be some conversation of defining what we're measuring in each of these things and I'll give you an example and it doesn't talk to the qualitative thing entirely but I think that's part of a lot of conversations about what the heck we even mean by we are. Do we mean course materials that cost less than a certain threshold, or do we mean course materials that are openly licensed and those are two really different things but if campuses, you know I'm thinking across the state if I have different campuses were defining we are in different ways, and they're telling me a use rate or this class uses we are and this one doesn't. But they're defining a we are differently, we got a problem and the data is not saying what we think it says. And so if we start to think through a lot of these terms that we use finding agreement defining them carefully defining them precisely understanding what we're talking about, especially when we move outside of individual classroom or an individual program to campus or a state system, or, or beyond right those the issues of definition become really difficult and important, but also potentially add to that cost that you're talking about. You know, I wanted to reflect for a moment on, I think somebody came up in the chat but it's tough for me to keep up with the chat because there's such incredible incredible comments being made there so I hope that I can get a copy of the chat later but that's the side note I just, I just wanted to say that, as, as we're looking about moving more towards open pedagogy and, and kind of, I mean, in a radical sense is which I tend to bring is the idea of a kind of outdated textbook right and so when I look at this kind of approach I'm thinking, you know what what metrics can we use and for me it's centered on students and, and that's entering says, what, what are students saying, what are students doing, because I'm a developmentalist and so I see my students already creating content, you know their content creators, and I get all over the place and social media and it wasn't too long ago that we were frowning on social media and we were frowning on the different sites that we thought were less rigorous and, and less scientific and now we see students running wild with content creation and when I bring students into my courses and invite them to create content, whether that's video content or written content or assessment, can you imagine students creating their own assessments. And really, for me, creates this communal type of social construction of knowledge and learning and ownership and I think as a statistician that we could still look at those things via quantitative analysis. I think we, we, we kind of shift over to the idea that we need to use qualitative data and I'm not saying that we shouldn't because absolutely qualitative data is so rich, and it forms the basis of multicultural psychology. I also think that there are proxy variables and mediating variables and path analysis models that could help us understand some of the intermediary processes that go on between the creation of knowledge, the consumption of that knowledge, and the ability to take that into the broader sphere. And what I consider the broader sphere is the community and family. So, if we're able to look at metrics that say this content creation allowed students to not only consume and create, but it also impacted their immediate siblings, cousins, communities, neighbors, that those metrics are are as important, especially as we're looking to increase enrollment across higher education. Yes, thank you, Rebecca for that and I was just noting in the chat that it connected to what you were just saying but also what you were saying earlier is is more about this the kind of involvement of the impact that open education can have on the broader community. And I was just wondering, like, you know what you, what you think, you know, we should be looking at there in terms of open pedagogy and that kind of community generation of knowledge like how how do we. How would we go about measuring that maybe not. I don't know other panelists can feel free to chime in as well and people in the chat to but I'm just I kind of, I'm wondering like what that would look like you know how do we go about doing it. I mean, I would suggest that we look at a sense of efficacy among our students who matriculate through programs that center open pedagogy, and also center or we are. So for example at my campus we have some degrees that are we are degrees, and that means that there is not one textbook that's purchased right, but at the same time, we're still making the leap to open practices, in terms of creating particular what is interesting to me is assessment right because I am interested in assessing what the students create in terms of assessment. And so I would suggest that if there are opportunities to look at that type of data and parse it out around the measurement of student performance as well as their sense of efficacy over the creation of content I think that would be really powerful as we try to move away from you know these old outdated textbook processes because in the workforce we're not really seeing people using that type of those, you know, really static knowledge anymore so we're seeing, you know, really the cutting edge fields using dynamic processes of learning and so I think we even see some of that going on. And a more prolific sense in K through 12 than we do in higher ed. So I would center the idea of being able to parse out data that involves a sense of construction, and I think that could be done, especially in courses, and with instructors who are allowing the students the freedom to create and navigate their own education. There's a little bit of interest in the chat about the idea of open education impacting the community more broadly. And I think that, you know, maybe that is a subject for that could be its own webinar actually at some point in the future so we've taken note of that but of course we can continue that discussion. So just given the fact that we have, we're kind of approaching the last quarter of the webinar here. We can maybe like transition into the second to last question, which is specifically about resources now we've already talked we've provided, you know, discuss a lot of different ideas and but I'm just wondering from the panelists but then also from from the participants there in the chat what resources are available to help this community guide by community I mean the open education community. You know guide further discussions and further practices and efforts to measure these kinds of impacts so what resources do we do we think might be out there for folks other than this webinar apparently which is which is emerging as potentially one. Yeah, I guess I'll go first. I'm going to try. We'll see how this works to copy a couple of things into the chat. Okay, that didn't work exactly like I intended so I'll go back well another panelist is talking and copy actual hyperlinks in there. But there are a couple of things out there. Lumen has some stuff in their sort of we are playbook about measuring the impact. There's the coop framework which I think is becoming increasingly popular for thinking about the impact of we are. There's some chapters in some press books books about sort of getting started with we are that are helpful. And then the OER I student impact toolkit has a very a variety of resources about helping students measure the impact of we are these are all things that I think are helpful. I'm not sure they get us all the way there but sort of between them, they get us someplace and help us, at least in my perspective, they help me think about what we ought to be doing that we're not currently doing to think about the value of the impact and measuring that impact of we are. So let me get those corrected hyperlinks in the chat. Excellent. I know CCC OER and the Research and Impact Committee is going to be providing some some resources in your future. And really it's about going into the literature. Seeing what studies are being performed looking at their methodology thinking about how it applies to your specific campus your location your community, and thinking about what data they're collecting how they're collecting it and how you could replicate that study to tell the localize story for your institution and community. And so for me that's where I would spend my time is really looking in the literature. And Ryan these are great resources these are some that I was actually thinking about myself the OER I and Cooper frameworks are excellent resources to start with. I just wanted to chime in on on just local right local efforts and in particular, you know, again as as a child, you know the library was, was a place where I would often run to although the librarians were we're not never ever the nicest and now that I have jumped into open pedagogy and OER, I have, I have found an incredible resource in every librarian that I have met across institutions and so for me, I always partner up with my librarians because they're incredible researchers right and and they provide so many different on my campus there's so so much access to different portals around resources. And I also wanted to mention that the Arlo network, as we're moving forward with our next phase is looking to create spaces around establishing resource hubs for for those of us around North America and but I would definitely say in terms of measurement, there's an incredible Alliance, if you can establish it, both locally at your college and and with the surrounding universities between the methodologists who happened to be in the field of psychology and also the the librarians who really tap into the databases that we have access to and are actually very savvy when it comes to data analysis. So I want to make sure that we have the opportunity to solicit some, some feedback, not feedback but like you know some participation from the audience here, anyone in the audience who's interested in, in, you know you can turn on your audio you could I believe so you can turn on your audio your camera whatever to answer this question here but we're wondering what kind of data you are collecting at your institution. And how you're collecting it, and to whom you're reporting, you know because as we've kind of talked about that's pretty important right because you have a particular audience so if anyone has any thoughts that they want to chime in with, you know to just join the conversation feel free to turn your audio on, or even just pop it in the chat if you're not comfortable doing that but we have a few minutes here dedicated to to try and to address some of these some of your approaches. I hope I didn't step on anybody's toes by just unmuting. I won't turn my camera on because I'm on grandma duty right now, and the baby sleeping so. I've been posting in the chat, my thoughts about technician education or technical education now I'm at a two year college and the bulk of our degrees are centered around the petrochemical industry and refineries that are in my area and in fact I'm speaking from my own experience I am in training myself to become a solar installation instructor for our new clean energy programs that are coming on board. And I have in front of me the proprietary training guides that we're required to use if you want to be certified in those areas. And yet there there's a well, I mean, this this stuff isn't like, you know, just so cutting edge and coming out now that people don't know about it people haven't used it there's a wealth of information out there. But the will to, I guess book the system and tell these industries that hey you know there's a different model we could use to to train folk in these areas. They don't need to be spending, you know, so much money to to obtain books that you mandate. You know, for example if you if you want to be a process technician, you have to use the books that the company industry says are like the training manuals and, you know, the training Bible so to speak. And it creates a burden on students and yet you hear industry saying we have so many jobs that we can't fill. You have a lot of jobs that you can't fill because you're not you're not necessarily using a model that gives everybody equal shot at the at those jobs right. So, you know, I mean I knew about this stuff ahead before but now I'm really like in it because I'm doing it myself. And so, you know, we can create hundreds of open chemistry courses and change the way instructors teach those materials. And many times that to your colleges those classes just support classes to what the students are really there to get. And they have no way around using what they're being told they have to use. Yeah, that's something that we've talked a song about in Tennessee. You know my experience with we are comes from a four year where we were using sort of more, I guess I would say traditional and not technical materials in our classes. But it strikes me as something that's really interesting and important in Tennessee, there's a huge focus on technical training, but there's a huge expense to the proprietary training materials that you've talked about really well. I think that also the mechanism for converting to we are is different because technical faculty in Tennessee they teach differently so they're in the classroom like from eight until five every day. And that means that they don't really, we can't use the same levers we can't use release time we can't use extra compensation in the same way. The life cycle of we are development on the technical side is just incredibly different. It's something that I'd like to figure out a lot more. So, we can connect offline because it seems like you have some interesting ideas there. I look forward to it. So I've been scrolling through the chat and it's very interesting to see just just the range of ways that we're collecting data how we're collecting data. One of the big issues that I find today and and I'll, I'll, I want to give a shout out to Louise Feldman my colleague who's here who's doing a great inventory for our institution for faculty are using we are and we're currently walking through the process of institutional policy for course marking so I'm seeing a lot of individuals are working with their registrar's office to to identify those courses that are using open materials and then running those reports and it kind of hits on that quantitative data should be easy, especially if we have our system set up correctly, but in many cases it's not and requires a lot of manual things. So, from my perspective I think a big part of this discussion would be course marking so that we can maximize the use of our internal systems and really quickly and easily pull that data out. So that's just one of those things that I'm seeing happening in the discussion right now. And I would say, you know, a question I would have is, you know, would it be useful if we had some kind of standardized practice that we can use as a community so that we can benchmark against other institutions. As Ron was saying earlier I believe sharing that common language common definitions so that we know at one institution if we say this it's it's it's the same thing at every other institution just want to see if anybody had any thoughts on that. I don't have any thoughts on that, but going back to what Esperanza did put oil, that the proprietary nature of technical manuals, and how it is all, you know, profit driven to get these people educated, and even stepping into something that isn't OER or OER acceptable. I don't think that would ever happen, but changing their mindsets, that can happen. You know, I would, before the 2000s, OER really wasn't available wasn't really out there. You know, it took, it took stuff to happen. So, I think making technical aspect, unparatory, that will happen. You know, it's going to take some time and pushing out there and saying hey, look, we can do this in regular academic. We should be able to do it here but we can't because of these roadblocks. Does that make sense because that's kind of like how the OER initiative started. We see this, we need to make steps towards that. Susan, this is Esperanza, I'll respond real quick. My solution to this is that industry needs to be at the table. Yes. They need to be part of these conversations. They need to be represented here because that's a big hole that needs to be plugged in this whole OER movement and process. Well, not just industry. I mean, it's just like I say, you know, even when we're talking about, for example, training a certain number of graduates and then moving them into the workforce. Well, if workforce isn't talking to you, it's going to be very difficult to do that. Same thing here. If we want, you know, workforce and industry to use open materials and open pedagogy, then they have to be at the table when decisions are being made. You know, that they feel valued and they can put their input in and we can work together to make the changes. But if we're just trying to drive it from our side, it's not going to happen. True, true, but sometimes I mean, but look at how it happened. What was it in the 2000s how OER got started, or how creative comments got started is because something happened in government. Somebody said, Oh, no, this isn't going to happen. I mean, this is like, you know, the 70 years plus my history is a little big. Sorry about that. But it did happen. So I think getting industry to the table, getting big places like cats and Toyota and Lockheed Martin, those things, those people to the table will happen. But maybe they also need an invitation to an incentive to getting them to the table. You know, I know we're nearly out of time and I, I wanted to thank everybody for joining in for allowing me to speak but I also saw one particular comment. And like I said, I need to go through them more in depth later and I think it was Yoti, and I hope I'm saying that correctly is the idea of. Oh, sorry, is the idea that we should be looking to students and I think that I've made that point in particular and I had a quick example of a student who reached out to me and said, can I have access back to all of the things I wrote in my in my notes because I want to use it to inform my application to nursing school and so I just want to throw out there the idea that if you are already in the space of creating open pedagogy, and we're in the space of utilizing it's so important to almost form a roster of some of your students who are contributing at such a high level that you may be able to rely on them as you move forward and bring their voices back to the table and we're trying to do that at my institution, with bringing students who have gone on and and we're actively creating content and and talk to others about how empowering that process is. Rebecca, well I want to thank everybody for this fantastic, all the participation at the end here the great activity in the chat and everything. So I actually sent, I put a link in the chat but you can view all of the four fall webinars at that URL right there. And I encourage you to check out like all the archived webinars that we've done for years there's, I mean, I don't know how many there's like 100 of them or something like that that that are out there. I encourage you to do that. Please. If you're not part of the community email and you want to be looped in on on, you know, all things we are I guess I shouldn't say all but lots of we are and you want to get extra emails. Then please join the community email, you don't have to be a member of CCC we are to do that but it is really a great in itself a great resource not just for networking communication but people share things and ask about things on there so it's really great. We have a number of blog posts EDI blog posts and student we are impact stories on our website so I encourage you to check that out as well and please let us know how you think we did. And I encourage you to join us next time and and see all later thanks for being here. Thank you again to our panelists as well. Thank you this was a great discussion.