 Good afternoon and welcome to our faculty panel on open educational resources. My name is Christina Hendricks. I'm the academic director of the Center for Teaching, Learning and Technology, and also a professor of teaching in philosophy at UBC Vancouver. And I'm here to introduce you to the panel who's going to be talking about various open educational resources projects they've been involved in over the last, I don't know, year, two years, three years potentially. So just to give you some ideas of the really interesting things that are happening around open educational resources here at UBC Vancouver. And I'm going to start for introducing our panel. I couldn't remember where I was starting. So we're going to start with Elise Yeager, who is an instructor in the Department of Mathematics. Then we have a project that was led by Leonora Crema from the UBC Library, but she could not be here today. So we have Erin Fields in her place, who is an open education specialist in the UBC Library. We have Agnes Dantremont and Jonathan Voret from Mechanical Engineering and Chemical and Biological Engineering. And we also have Siobhan Miguel-Deff from the Department of Classical Near Eastern and Religious Studies. So we're going to be going in that order, which is not the same as this order, but we have slides. So it's important that we go in the right order to get the slides right. Each person is going to talk for about 10 to 12 minutes about their project, and then we'll have time for questions at the end. But first, I'm going to say just a little bit about open educational resources in case there are people in the room who are not as familiar with the idea of open educational resources. So we've got up here a definition of OER from the United Nations Educational Scientific and Cultural Organization, UNESCO. They just recently updated their OER definition, and this is the newest one. So it is teaching, learning, and resource, excuse me, teaching, learning, and research materials in any medium, digital or otherwise, that reside in the public domain or have been released under an open license that permits no cost, access, use, adaptation, and redistribution by others with no or limited restrictions. So there's a lot going on in there, but it is really any teaching and learning resource that can be anything from a text, a reading, a syllabus, assignment directions, to videos, to podcasts, to other sorts of digital objects that are licensed to be allowed to be reused by others without asking direct permission from the Copyright Holder, and often also allowed to be revised by others and free of cost to do so. So I wanted to mention also something that will come up in discussions of open educational resources quite often. This is what's called the 5R permissions from open education supporter advocate David Wiley. He came up with these in 2014. That's where the first blog post about the 5R permissions came up. So things that are as open as possible in terms of educational resources have these five permissions, meaning you're allowed to retain, to make own and control copies of the content. So that would be different than, say, a textbook that you have to rent and give access back. So it's something that you can retain. Reuse. It's material that can be reused in multiple ways. Revised. You can adapt it. You can modify it. You can edit it. Remix. You can combine one piece of material with something else or even a third thing and create an entirely new compilation. And redistribute. You're able to not only, let's say, make physical copies for students but also post it on a publicly available website. So how do you get these permissions? Well, usually, materials are given an open license. And the open license will tell you what permissions you have. And sort of the most open licenses will allow you all five of those permissions. And very frequently, open educational resources will have a Creative Commons license. Now anybody who's not really familiar with Creative Commons licenses on a website called the OpenUBC website, URL is there, open.ubc.ca, we've got a tool kit that is open licensing basics for instructors and for students. So if you go to the OpenUBC website and look up Open Education, then you'll find a link to the tool kits. So often resources will have a Creative Commons license that allows those five permissions. So that was just a little bit of introduction. And now we're going to move straight into our panelists talking about the projects they've been involved in. Oh, notification on the top of the slide. Sorry about that. Yeah, that's a big deal. Okay, so first off is Elise Yeager talking about clp calculus. Yeah, where you can use this. But you need a mic because we're recording. Okay. You know what? AV supports. Excellent. My name is Elise and I'm going to talk about, oops, our project called CLP Calculus. This is joint work with Joel Feldman, Andrew Rechnitzer, and me, and we're all from the math department. So the math has four series calculus course. So this is pretty standard. It's usually for semesters of calculus. And we have working textbooks for all four of these series. And these are some of the courses that they're being used in. The textbooks exist as sort of a standard PDF textbook. So students can grab them online for free. And then we're also in the process of turning them into web-based versions as well. So the first of the textbook, you have to worry about word wrapping. Okay. So this started when Stuart Calculus, which is the sort of probably most famous textbook, put out a change the context of a little, put out a new addition and a new jacket. But there's also the free-ism speech, which had less to do with the motivation to start, but a lot to do with the motivation to keep going. And then I asked my co-authors what else motivated you. And Andrew said unwarranted optimism. So Joel had notes that he already had that Andrew thought, like, oh, it'll be really easy to turn this into a textbook. And that wasn't true. But then when I joined, mostly, I mean, it was a good project, but I thought it would be super fun. And I was correct. It is super fun. And none of us had any. Right. Okay. So we started pretty much without student input. So Joel had the skeleton of the notes. Andrew turned them into a textbook. And I wrote the exercises. But then once we had a working draft, we started incorporating student input. So when these are being used by a class, you'll have easily a thousand pairs of eyes on a chapter at a time. And so we offered a little bug bounty. So if you're in Calculus One and you see an error in the textbook, you report it to a graduate TA who sees if it's the first error. And if you're the first one to report it, then your teacher in front of the whole class kind of gives you a pat on the back and gives you, like, a card for a coffee at Lofi. So that's a really good way to involve students and also to get really quality copy editing. We also hired a work-learn student who was an undergrad when we hired him and a master's student when he left to think about accessibility and clarity. So he researched a bunch of accessibility and he gave us a style guide for things like which colors are distinguishable to people with different kinds of color blindness, where should you put the text in your figures so that screen readers can read them, things like that. And then he also unified the design in our figures to make them easier. And it's just a nicer text for him being involved. Student reaction has been very positive, largely because of the free as in beer. Joel gave me some stats from the website. So in just the month of September 2019, we had 16,000 sessions, which is over about 10,000 from the same month in 2018. We had almost 8,000 users, which is up from 4,800 in the month of September 2018. And a lot of that growth is coming from outside of UBC, and we actually have no idea who these people are, because anyone can access it. I mean, people don't ask. So we don't always know. Yes. So things that would have helped. One thing that we're still hoping to find a mechanism for is a bit more critical feedback. So the bug bounty is really good for saying, oh, there's a typo here, things like that. But for more critical reading, that's a tougher type of feedback to get. Once there was a skeleton, again, the amount of work estimated and turning a skeleton into a book was vastly underestimated, and so a teaching bio probably would have helped. And then things like the bug bounty. This was before these sort of nice OER grants were quite as active as they are, and it took a little bit of time to figure out how to fund that. So there's four books, and the first two are stable. The last two, the text is stable, and we're still filling in some of the gaps in the exercise books. We have some updating to do now that we have style guidelines, just to kind of polish things and put things from PDF to the web. So there's a bit of polishing to do on the text itself, but then we've also branched out into some related projects. So my current pet project is something called Orchard. So we wrote a ton of exercises, and in PDF form, they're the same as a book, which is just number one, number two, number three, but that's not the best way we want our students to study them, right? So we're working on some software recommendations, so problem one, and then how was that, and then based on your answer, we'll recommend something else, and we also want to put it together with goal setting and tracking. So I'm going to write here that I want to study three times this week, and the computer is going to keep me up to date as to how I am on that goal, and provide feedback that hopefully will be helpful towards meeting goals and also maybe, for lucky, encourage a growth mindset. So one of my co-authors, Andrew, and also Sachkan DeMirbash in the math department are starting to develop a free version of that, and my co-author, Andrew, with too many people that I could name, is working on a paperless online marking system. So this is something like Gradescope or CrowdMark, but hopefully free and open source. Okay, thanks. Okay, and next is Leonora Crema is not here, but Erin Fields from the library. So forgive me, I'm working off of somebody else's notes and project, so I will do the best I can speaking to it. So this project began in the spring of 2018 with a series of adjudicated subgrants to endless faculty's interest in partnering in OER development with the library. So really what did the library want to accomplish with this? Well, the growth of OER seemed to be happening on our campus for quite some time, and it's trying to kind of understand the library's role in that space, especially since open education is so closely related to open access in general, which is something the library has been considering for quite some time. We also wanted with these projects to think about how students could support this kind of work related to publishing of OERs and specifically looking at how library students would be involved in this process. We wanted to have a better understanding of faculty needs for open education resource development and where those needs intersect with what the library can do in terms of service. And then we want to align with the UBC TLEF priorities in addition to the priorities from the library, specifically again with open access being a focus. But mainly we wanted to explore the landscape of OERs on campus and the library's capacity to support faculty in creating open education resources. So we secured a grant from the TLEF, the Teaching and Learning Enhancement Fund for about $39,000 for a one-year project. The process was really that the library gave adjudicated sub-grants of about $2,000 each to incentivize faculty partnerships. Half of the funds went to the faculty projects while the other half went to the library to support project students. So there was a hiring of two GAAs to shore up some of the supports in the library around open education development. We didn't decline any of the projects as long as it resulted in open content it was considered to be a project that we would support. The project resulted in ten faculty projects that were funded and it was a mix of new works and enhancements to existing works and it was across subject spectrum. So there was history, mathematics, geography, philosophy, social work, political science and physics. So it was a good kind of cross the entire university and it was with many formats. We didn't restrict it specifically to publishing an open text. So it was text books, there was videos, simulations and more beyond that. So we also short up some of the work that was using press books due to there's robust service from PC campus to support press book use. Specifically we use their iteration of press books that any person associated to an academic institution has access to. But we quickly found that other platforms such as Overleaf would potentially have served some of the projects maybe better than press books would. And there was a need for very specific knowledge specifically around LaTeX. We took a publishing life approach which means we accepted offer-proofed content with no copy editing and no indexing. So the real work from the library was about ensuring that the content that we got would fit with inside of the tools that we were using. So press books and that sort of supports. But the actual editorial work was with the faculty members. So the focus was on layout, open licensing as well as front matter. So these are some of those 10 projects that were supported. So they ran a gamut. So there was simulated case studies which was for social work which focused on improving lives of women and girls through integrating integrative health practices. There was also an accessibility open textbook from Calculus which was really supporting the creation of accessible open math text for students with visual impairments. So it ran the gamut in terms of the kinds of content that were actually published. So in addition to this project work assessment was a part of our Leonora's work, not mine. I'm reading the notes, I'm saying our. In addition to conducting interviews of six faculty partners who indicated the important work and role the library in their OER development specifically related to copyright, creative commons licensing, publishing, project managing, etc. There was also an evaluation of students in the classes that were using the OER so a survey was delivered. What they found from this was in terms of learning experience that over 80% preferred open textbooks to traditional ones and none preferred the traditional textbooks at all. I think we can understand where that answer is coming from. Also, when compared to other similarly demanding classes had the use of open textbooks reduce the overall cost of the class and about 75% agreed, some weren't sure and then none disagreed so there's probably a disconnect between the open piece having an impact on student saving. But one of the more interesting quotes from a student was since our prof wrote the book it was designed specifically for the class it made it really easy to refer back to the textbook when looking over lecture notes, concepts and problems so they saw a really integrated approach to their materials because it all was written and produced and delivered by the same person. So major takeaways the library can definitely add value to the process of helping developing open education resources we have knowledge and copyright definitely knowledge and Creative Commons licensing the ability to deal with the publishing standards the front matter as well as being able to give DOI so digital object identifiers to material. Accessibility through metadata and platforms that's what libraries do we make things accessible so we understand those processes around attaching metadata to content and then making it available through various databases and spaces. So the long-term archiving of objects press books it's easy to use and effective there's a lot of power to it like interactivity with H5P as well as the latex conversion and there was a lot of peer support so it wasn't just UBC library attempting this on their own there's local post-secondary institutions like KPU that are doing similar projects BC campus of course and open textbook networks. Some of the things that need to be considered is there's a format proliferation of open publishing tools out there and that is not going to stop so press books is just one tool but there are many others and many more that may come along that have added benefit so there's having to have an awareness around what is coming up and how to integrate that into this space there's interactive elements will likely outstrip that standard e-publishing software so right now we're still kind of running on the traditional notion of what a book is and what it looks like but as time goes on I think that that kind of model is going to change. The question of what is an addition is an area of constantly updating content is an issue so we often see traditional textbooks it will be addition one, two, three, four but if in an open education resource that can be downloaded, modified you can get well into hundreds of additions in a very short period of time so how to manage that is an interesting issue. Pedagogy and publishing have become increasingly intertwined so we're having students that are creating content and that's usually beneficial for their own learning processes and also beneficial for the world in terms of adding value back as well as more conventional kind of publishing processes our projects are becoming more challenging in this space because again that open it's hard to track something when it's made openly available like you're saying somebody can use the content but who and how it's an issue to consider. So next steps with the announcement of the OER fund the library is now boosting its direct support for faculty and working with CTLT so there's more support in terms of finding open education resources that faculty are interested in there's a service around that publishing using open journal systems as well as press books library has support there as well as well as making the content available through a variety of different repository spaces and assisting with metadata. So these pieces are now available in the library and is a direct result from this initial pilot as well as the OER fund being developed. There also is an expanded digital publishing service at the skullcoms.ubc.ca website so you can find all of the information about how to get that support from the library there. So thank you. Thank you Erin. I'm sitting in the front. It's incredibly uncomfortable to turn around. So if you guys want to move out to be able to see you're welcome to do that and we'll come back when we do the questions. My neck is tough. Next I'd like to introduce Agnes Dantouma and Jonathan Varrette from Applied Science. Thanks Christina. We also have a colleague who's worked with us for this project. She's away on maternity leave with her firstborn so that's great news. So we're here representing the project and I'll kind of give an introduction and Agnes will talk a bit about our evaluation afterwards. So as a background WebWorks an open source online homework system and what will happen is a student will log in, they'll be assigned a set of numbers which are generally unique and then they'll be allowed multiple attempts up to unlimited attempts at it and then they'll be able to evaluate the student feedback as to whether it's correct or incorrect. So the computer is kind of calculating what that answer is for their unique numbers. There's about 35,000, over 35,000 problems in something called the open problem library which is stored on this Web site called GitHub which anybody can access. There's these kind of code snippets of all these problems and they're categorized into different subject areas so you can see the subject math subjects and so there's only about 260 engineering problems and when we started there was only kind of two areas where these problems were located. So UBC engineering students they take first year calculus as an example of a course and in that course they use WebWorks so they're already introduced to WebWorks so it's very easy for us to implement it in our engineering courses but of course we didn't have questions that we could base it off of so our goal was to develop a bunch of WebWork problems in second year engineering because that's where we have a lot of overlap in different subjects. So this had been going on before we started this project with the TLEF it was going on in the electrical and computer engineering department and they'd run a student survey where students were relatively happy about using WebWork like the lack of cost of it like the immediate feedback. There's also a student survey run in MEK where you can see some of the results there. They compared kind of Blackboard as a system back when we were using Blackboard before Canvas where you could put in something similar but it would re-randomize the numbers each time which kind of frustrated students so you can see I think the result of that in their responses there to that survey and then in chemical and biological engineering when I was using it I looked at TA and instructor working hours before using WebWork in 2016 and then after using WebWork in 2017 and 2018 and there was a big increase in my hours which are there on the bottom course instructor hours creating the problem sets but then after that it's a bit smoother and quite a bit easier to use them and my TA's had more time to interact with students rather than grading paper assignments let's say entirely so before this project we had sort of these three departments working in isolation not a lot of questions available openly and no sharing of questions across topics that are similar between disciplines so one example of that is fluid mechanics which is in mechanical engineering and chemical and biological engineering so we kind of rounded up these second year classes that were in multiple departments and here's kind of a broad chart of what this looks like so you can see here fluid mechanics chemical engineering, civil engineering, mechanical engineering dynamics between a few biomedical engineering and mechanical engineering I won't go through all of these but just to give you an idea these are some of the topic areas a number of questions that we aim to create in each of those topic areas for these courses there are some that have a few less like mining that kind of jumped on afterwards so those are the difference between year one subjects was when we kind of went out and found a bunch of courses that were across disciplines and year two some subjects kind of came to us and it heard about us which was nice and wanted questions created so to give you an idea of what this question creation process looks like basically what we tried to model was to have guidance from the course instructor or course instructors in terms of the type of content they'd like to have the types of questions the areas, right a graduate student would go and develop a problem then they would create kind of a rough sketch of the problem so that's kind of the first picture you see there on the left at the bottom with the variables we'd have an undergraduate student who is working with us who would code all the problems from there kind of taking that rough problem and turning it into a finalized problem which is what you see on the left there was some graphic support either from that student or from somebody else doing the graphics so like you see taking those hand drawn images and trying to turn them into something quite a bit nicer and then we test it on the students the first year get all the bugs out of it and then upload it to the OPL the open problem library afterwards so now from that 260 questions what this looks like is we have about a thousand questions that are in testing or contributed from UBC also because of this work we've created new taxonomies so each of these subject areas we started with two now we have something like 13 up here and that's allowed us to take in questions from other institutions that have been created so that number has gone up from 260 to 327 with another 324 from UBC so let Agnes talk a bit about evaluation of the project so we did have 13 partner courses last year and in each term we had a pre and post survey for the students the pre survey was focused on what's their past experience with online homework and what do they find is important about online homework and then in the post survey we were more asking about our questions so I won't cover all of the results but a few highlights so we actually asked them if they use web work before and if they use other systems before and out of the 331 respondents the vast majority had used web work before and when we compared statistically web work went out in all of these areas so easy to use easy to navigate enhanced my learning motivated me to persist and I want to use this in the future so the feedback being clear and easy to access the other systems went out on that one when we were asking a post survey about our new problems well they noticed we had a few errors in the problems unfortunately we found that even the amount of testing that we put in was not sufficient we ended up putting in more testing for the second term they also felt they weren't quite the same level of difficulty and that might be just because of having one instructor kind of helping us create the problems and multiple instructors trying them they have different desires for having easier problems or harder problems or how they match but they liked our images and then we were asking about how does this motivate you to learn well we actually got pretty high marks in them being motivated to try and solve the problems to attempt and to successfully complete the problems so we felt that that was probably an indication of pretty good learning and we still got good marks in web work enhanced my learning we got quite positive instructor responses to we they saw the value of having the students get that feedback immediately rather than hand in some homework get it back a week later marked up and are they ever going to go back to that problem again probably not it reduced marking time it provided more feedback the fact that they were shared problems and the instructors themselves can adapt those problems say if they're using slightly different notation they just change a few things in the code and it's done we had on-call support so we had a co-op student on-call to help with errors in the problems and there were so we were having sort of a one or two day turnaround to fix errors and we actually were able to convert a whole bunch of problems from desire to learn and the iMathAS system as well it was long creating these problems we figure between an hour and a half to two hours per problem including designing the problem creating the graphics coding it and testing it so quite a lot of effort per problem and of course we have to make these professional looking graphics we don't have to but it really makes a difference in how the problems look and obviously we had errors in the code and we had a problem with tolerances in some cases where the students could be a little off due to rounding errors and they wouldn't get the right answer what would have helped more funds it was a pretty expensive process to create these problems support for instructor time to help us that was a real barrier with some instructors who just couldn't give enough time to even tell the students developing the problems what they needed to have co-creation opportunities with the students would have been great as well and better documentation for web work in general and some more local expertise on coding would have been great and we've got some acknowledgments and if you want to use any of our problems they're all available we'll happily help you with that so thanks thank you and we have one more presentation from John McElda from classical Near Eastern and religious studies and then after that hopefully you're thinking about your questions because we will have time for questions so unfortunately my computer won't go on this I do have to go to the whole school I mean classics so there's three bits what is the syllabus for the course which kind of explains the content this is my favorite and then what is the first assignment which I have to change during the course and one was my sample first assignment which really didn't solve some of the problems so this I taught a class last year called Un-Roman Romans so that's Bandits Exile Sex Workers Witches and Other Outsiders in the Roman Empire and it was a course where I suggested it and thought it's 2018 and loads of stuff I'll be able to find a book that and I went through about 20 text books readers and in every one I found something enormously offensive and actually shockingly obviously unaware of it I didn't go to that thing anyway it doesn't really matter so go back a bit and I don't want to the subject matter we were dealing with if you know anything about the Romans other than the ancient Greeks but that's a really low bar and so the material we were dealing with was deeply offensive the list of people they don't like goes from grave diggers through to lesbians through to people who cross dress people who have fringe on their tunics or maybe enchanting the richest widow in town to marry them even though they're 20 years younger and I didn't really so the students were being asked to read very difficult material and I was like well I'm not going to give them other material that's also transphobic or sexist or colonialist or just straight out racist without even being aware of it and I mean it saddens me to say this is someone who's in classics so I decided that I'd put an OER together for a course of spectacles in ancient Rome and I thought I'd do that but then I realised that the problem with that is that I work with awful Romans a lot so I'm often gliding past terrible things they're doing and someone will be like hang on can we go back to that bit and I'm like alright yeah they just did do that or you don't really realise that students like there's a lot of help that they need that has to do with them not know anything about Roman history nor should they have to so you have to explain some basic facts that like a lot of textbooks don't work with so I thought well if I had any questions I could add some notes introduce them and work on them in class and have students work to create an OER themselves so what they would do is they take on a very small area of text or image or something else that they wanted to work on out of a list it kind of expanded beyond the list and we would work towards in a three stage process them creating something that could go out as a piece of public scholarship primary source reader on how awful the Romans were were dealing with difference so I did that it was a student of about close to 34 students I would say I would not do it again with that many students because it was an immense amount of work it involved the first thing you have there is the first assignment which was like just to write up a proposal come up with a subject look through this everybody suggested way too much a response which was like no you can't write on all the terrible things the Romans did there you'll have to write on this one poor person or these three years in this one poor person's life that we know about so I had students work on a range of topics anything from transgender issues to grave diggers to sex workers, to actors to accusations of prostitution they got flung around a lot and each one of them even though very few of them knew anything really about Greek or Latin they worked on a section and by the end of it through a three step process they produced at the end a piece of scholarship that's going into the reader that will be eventually published and finished online it's still in the process because it's a lot of work and it needs a lot of careful handling because the subjects are really problematic I would say I would highly recommend if you have anything you can do with students I would do it like it's a great experience the marking was awesome because you mark things they have to read it for the next stage there was no wasted marking it was a lot of marking it was a lot of effort the students thoroughly enjoyed it they enjoyed realizing that they could contribute to scholarship they also really enjoyed realizing they know more about race, gender and whole sexuality than many academics like it was like very obvious our midterm was here's a bunch of passages introduced by various classes with their names removed for everyone's sake how do you feel about these and they were very perceptive they were very good scholars they understand some things very well they understand difference very well they understand how to talk about difference without claiming that your position is the only right true one because of that one time I don't know you saw a person on the beach or they don't claim things like slaves in Greece must have been happy because they had the parts in the hun so stuff like that this was part of my project it worked really well I'm building it at the moment it got a course release from the public humanities hub it will be published and hopefully some of it will also be used outside of UBC for students who are really interested in difference and the history of difference I mean it doesn't take much realisation to realise that classics are super attractive to right supremacists fascism comes from fascades and so there's also an issue in my field of the need for serious interventions to show people that antiquity was not all just white straight men it was a vast compendium of individuals who created the Roman Empire and so that's really basically it I have a lot more things what you have there is the syllabus which you will note only goes half way through I had to give up my right to control the syllabus I had to allow them to vote on subjects they wanted to do so this also means you have to give up a lot of control I had to talk about a lot of topics I don't know anything about because I allowed them to suggest the topics as long as they fell within a certain area and that means that you have to expose yourself I made a lot of mistakes if you do the first assignment make it credit no credit it eventually had to go the one problem though that the students have was and I think maybe we have done too good a job with this is they're obsessed with being accused of plagiarism for everything it was actually a serious problem they wanted to footnote everything and I spent a lot of time going no that's just life that's just dates asking them to transition from traditional papers to this type of scholarship that's open and public creates some ethical issues and creates some other issues with how to cite people that are not very well covered by some of our current materials and that was a big problem what would really help was time because if you do this and you decide to involve students please be aware it's an enormous amount of work the feedback has to get back within 10 days to be effective otherwise they kind of switch off so that was the downside so yeah more marketing help more student help would help thank you and I have a blog on Roman Roman so you can google that if you want anything it's up here on Roman Roman all the stuff is there a whole bunch of the readers prepare to be deeply offended I'm sorry I can't help it you will be offended okay by the Romans and from you all for anybody or all I have a question Elise first I want to get your permission do we need Mike? that's fine first I want to ask your permission to steal free as in beer free as in tea because it was great it's not like it's not like that that's something I'm very interested in in the library as well connected with the education faculty and very interested in instructor and faculty motivations so some of you explicitly talked about your motivations and you focused it as well some of it came out as your original motivations and then you found benefits as well as challenges so I want to ask the panel I know your motives about motives and if they stayed the same as you worked through working on your one of the big original motivations was just sticker shock and I teach a number of courses in the math department and some of them use traditional textbooks and one of them uses a really expensive traditional textbooks and students just don't buy it and the homework is out of the textbook and they just say I can't afford it I'm just going to eat that 10% of my grade and it's heartbreaking and it comes time to study for the final homework because you couldn't afford this textbook but I feel like the cost until recently has not been considered kind of a high brow motivation I really, you mentioned a study I think maybe your folks did where you had good data showing that a lot of UBC students don't buy this textbook because it's too expensive which matches my my anecdotal data but I'm up for tenure this year and when I was preparing my documents one of my committee members said well don't stress too much the cost savings because this is somehow not such a such a noble goal once so for the beginning that's a good starting motivation but then I mean once you have things that are adopted and you're using them I mean you just want to make them better and better you want to fix all the issues and you want everything to work and you want it to be exactly how you want so I think cost was a good motivation to start it and then the sort of personalization of it going I think for myself it started with student cost as well so we had we have these integrated courses in my program in second year where we have three or five subjects smushed together into one giant course and one year one of the instructors put on a paid homework system where the other homework was free and the students really didn't like that but also you know we silo content into these individual courses but really in engineering there's a lot of overlapping content and one of the reasons we have this combined course is to give students a problem in more than one subject area so they have to use knowledge from more than one subject to solve this problem you can't get that in publisher systems we have these siloed sets of questions and so this is an opportunity to also create questions that bring in more richness that put things in context that require the students to use multiple skills from multiple areas okay thank you any other questions so some of your projects hired students for pre-world listings from graphic design to accessibility how hard was it to find that with those skills that you were seeking unfortunately I can't speak to this particular project but I can speak to other open education projects out of the library and while they might not have the specific skills that are very tied to this tool or this pedagogy what is really wonderful is that the students are increasingly motivated because of not just the whole idea of the advocacy around cost savings for students and that but they see kind of higher value in open on a larger kind of global level that what's coming out of the public institution should be publicly available so they buy into those sort of motivations maybe more than some people that have been here for longer so if that's the case then you get students that are highly motivated which equates to really good work there is one question that I think depending on what you're working with and obviously this probably doesn't affect calculus and this is way more controversy in the world with calculus if you're asking a student to work on something that's problematic and the name's going to be attached to it you have to let them know like if you're hiring someone say work on a reader say well like I am I've got things and you can imagine a whole host of like art or other related subjects if you have to a reader on say Nazi Germany you could build that person's name so you need to like let them know that this is a permanent connection if you're using those fellowships or if they're working on the material this connection will remain all their lives so they should be aware of that entails if it's a technical thing it might not affect as much but if you're doing something that is slightly going to be particularly if they're from backgrounds where that might or professions where being quirky or slightly different is not a great thing so that's one of the things about dealing with wide-ranging students that you really have to consider is you are giving them a permanent record that can be good and that can be bad so that would be one thing I would warn anybody working in arts at least in our project we had a couple of really great students right out of second year all primed with this knowledge of our problem areas to the point where our first student worked for us full-time for eight months, part-time for four months and then presented at a national conference based on his work so really awesome just a question we will focus on kind of some strengths and good things today about who we are but I think just focus on learners' perspective did you get anything to make that a little bit more challenging for the learners ahead? Errors Errors and tolerances they were frustrated and we got a lot of requests that they would really like hints and sort of this adaptive platform which would suggest the next problem to try which as far as we know isn't available openly yet. No I have no problems my students actually really really liked it you had to set time limits because otherwise they would have gone crazy like finishing things you have to be careful if students have got impetus to finish all the things so they really liked it once we went for times they were very worried about if they were working on something for this they were worried what would count as good enough to be included and some degrees when I was working with students at least to be working to create it it was very important to give them out and to say if you spent 10 hours on this that's okay even if it's like even if you haven't done it very much because some stuff is going to be harder if you're using them as if you're assessing them on it that was critical but they really I didn't bring in valuations but they really liked it they felt like it was very useful and were pleased that they had done something that would be semi permanent a question from the library a question from the bank so the students also created or uploaded the questions there are questions that we created so those were created by let's say TAs mostly graduate students who had TA'd the course before then tested on students and then we uploaded them to the problem bank and so I mean anybody can go in and again this is a funny story about a student going in to correct one of our questions in github my students weren't that savvy unfortunately but yeah so anybody can then go and access it and use it in web works open source so anybody can use that as well anything that is posted in the OPL they can find and clearly one of my students did but ultimately the stuffs for practice anyway they can get the answer off their friend just as easily as finding it on github so if they're going that way they're not really getting the benefit of it and I think mostly the I don't know in my class it's like maybe 10% 5% something like that you get to the midterm and the final they're worth 60% so if you don't know the material and you can't perform it on your own so I think students with that generally most of them are swayed anyways how easy was it to handle multi-part questions did you have any multi-part questions where at part A, B, C part B uses part A and so on and so forth and it was an issue if a student gets part A their problem area part B is wrong is there a way to kind of like take their answer use it in part B or C and kind of make it still help them to get the points and maybe check their understanding of B's and C's or the following parts without understanding maybe getting the part B I had a few multi-part questions and I mean generally we've left the students to have unlimited attempts I think it's the same in Mech and so what that would generally signal the students okay I understand this first part of the problem but then the second part I maybe need to talk to a colleague or come see a TA for example and so that's how mostly I dealt with multi-part questions there is an opportunity to put in hints in the program but you have to program those in it's a little intensive and you almost have to predict kind of where students are going wrong which sometimes when you look at what their numbers are you can kind of see it but it's a little hard to kind of pre-format that there is a second method where you can lock out the second third and fourth parts until they get the first part right but mostly we didn't do that they could see all four parts and put in the answers and then get individual feedback on each of those parts but of course when you have a math exam aligned through web work you want to have multiple parts but there's always an issue of how are you going to be able to control how much they know, how much they don't because if they got a part A wrong even if they know everything then we just don't get any points which is kind of unfair so we're going to find a way to find such a problem how do you handle it? I know that one of our instructors wanted to produce like check off all the assumptions you're making in this problem that was the first part and then start giving answers so I think there are some ways to get the students to give you more information about what they're doing Next question When we just developed some of the courses sometimes the images are always relevant good images we need to paint so I'm wondering is not very good so is that this kind of we are funding is that it's possible potentially for the image you know the free resources something like that Do you want to start? We haven't talked about that So generally when it comes to licensing for an open context an open resource it's just it never goes forward mainly because often you license things for a specific audience and when you're making it open I can have an unlimited audience so I do believe that the funding could potentially be used it could also be used for graphic designers or students to help create images as well Christina would you have any thoughts there? No I guess I'm not quite sure if you're asking could you use the funding to pay for an image to go in We need a lot of images but all the relevant images we had to buy free images always the quality is really good for the course in that case I was just wondering if free open sources have a good the data like good image packages will be great for the course learning designer something like that with paying for access to images through the fund is that you could get permission to use it in that thing but they might not give you permission to make it public to everybody else or to let other people reuse it so that causes some issues Jeff has got all of you TLEF projects use the funding to create open source or creative common illustrations there's a lot of anatomy and documents have been created so people don't have to make the licensing so one thing we might look for is people will actually produce images and then create a common license of using this funding and there was also a question about whether there are good quality images available and that's where you could talk to the library where we can try to find images that could be used with creative common license there's still here to use it open education resources are everywhere so finding good quality is difficult but we definitely will help and I'm afraid it is the end of our hour so please join me in thanking our family