 Most of the participants here are actually professors that are currently working with us to put these courses together and you can visit our website sailor.org anytime and see what we are up to. I think we have about 220 courses that we've completed so far and we are beginning to put all of those courses through peer review and build additional assessments and expand to new areas of study. So we have a lot going on and I would encourage those of you who aren't familiar with what we are doing to take a look around. Really quickly I wanted to once again mention that you can chat us any questions you guys have in the chat pane. You can also raise your hand I believe and that will definitely let us know that you are having some kind of issue and we will work to address that as quickly as we can. And finally I am recording this session as I just paused to turn that on so that we can make this available later and you can send it out to whoever you want. I think we will probably upload it to our YouTube channel and we will also be providing a link to it from our sailor.org blog so if you go to the sailor.org web page you will see our blog link and you should be able to find it there. Okay so let's get started. This is the second in our course design workshop series and this is all about articulating and aligning learning outcomes in your course. And I am thrilled to welcome Ellen, Stephanie, Andy and Emily who are talented instructional designers from the Washington State Board for community and technical colleges. I am sure that they will discuss this but they have been working on a really cool project called the open course library to build a suite of 81 very low cost courses and they have been so helpful to us and we actually have a few of their courses posted on our site that we've kind of taken what they've done and put it into the sailor format. So I encourage you guys to check that out. And so I guess without further ado I'd like to turn it over to Andy who I think is going to be speaking first. Andy do you want to take it from here? We'll do. Before we get started I'm Andy Williams and I am one of the instructional designers on the open course library project as are my colleagues. In addition to that I work in the business division at Edmunds Community College and why don't I get Ellen and Emily and Stephanie to introduce themselves as well before we go forward. So Ellen? Can everybody hear me? Okay great. I'm Ellen Bremen. I am faculty in the communication studies department at Highline Community College. I've been at Highline. I believe I'm going into my eighth year. I seem to be losing track and I am an instructional designer on the open course library project. This year I'm also serving inequality assurance role and I've been on the I think all yes all four of us have been on this project now. This is our second year. We're in phase this is our we were on all on phase one and this year we're in phase two. So I'll move over to Emily. Sure so hi I'm Emily Wood. I'm going to have my webcam on for just a minute so you can see who I am. I'm a faculty librarian at Pierce College Fort Steilacoom in Lakewood Washington and I am also an instructional designer in the open course library. I primarily work with the math and science courses within the open course library and I think one of our one or two of the courses that I worked on with Tyler Wallace from phase one have been changed and formatted into sailor courses so I'm really excited to see what happens when you put things out and open and the creative things that can happen from that. So sailor's doing great work open course library is doing great work and I'm really excited to be here today. Hi I'm Stephanie Delaney. I'm also an instructional designer on the open course library and I'm in the e-learning department at Seattle Central Community College. I too am excited to be here joining you all. So thanks all. Andy here again before we start talking about objectives and assessments a little bit about the open course library and in our part in that as well as once again reiterating what Jennifer said and thanks again Jennifer for hosting us and inviting us. This is going to be a we want this to be a conversation and not just a one-way data dump and with the four of us we tend to be a little bit of you know for lack of a better word of free for all so please everyone feel free to jump in whenever you've got questions or want to add a comment. Open course library started a couple of years ago with the seed money from the Gates Foundation and from the Washington State Legislature with the goal of producing open courses that covered the the highest demanded courses in the state and the stipulations were to the faculty developers who are going to develop these courses could use course materials that couldn't cost students more than thirty dollars per course as well as courses that that met quality standards we used a modified quality matters rubric when we're evaluating these courses and they also had to meet accessibility and global education standards. As instructional designers we applied the quality matters rubric and a modified form to the courses to make sure that alignments between objectives assessments learning materials and resources were being met and as well as acted as project managers and I see that Kelvin has a question here are there plans to offer more open education resources for upper-level courses in which publishers do not provide at this point we're focusing on the the high-demand courses which also seem to be the ones that the publishers are targeting but I'm going to defer to my colleagues to see if they are aware of any other plans that are out there. Stephanie, Emily or Ellen? I don't think that's within the scope of the current phase and the previous phase of open course library which has been focused on first and second level university or university or community college level courses and most of those are intro or or survey courses but that would be great. And Kelvin I agree that that the the publishers see a ready market there the the big push from our project has been to provide alternatives to the the expensive textbooks that are out there the psychology 101 textbook that cost the student a hundred and fifty two hundred dollars and we're meeting with some success with adoptions but more so it's I think it's a good idea to just kind of move this idea forward and and give faculty the idea that there are alternatives or complements or supplements to the publisher courses. Anyway for those who are interested more in the open course library project you've got a link there on the slide to open course library.org and I encourage you to take a look there and see what what's offering we are we have phase one that is available to the the world. The phase two 40 some odd courses will be piloted this fall refigured and and retested as it were come next winter and then will be available spring of 2013 again to the world. So I'm going to move on forward here. And the basis of our instructional design starts from that quality matters rubric and really with the idea of developing objectives and teaching to objectives and and I'm going to be general about these and then turn it over to my colleague Ellen to be more specific but the general idea with with the objectives is to think about to whom the the course is being designed for the students and with the idea what do you want those students to be able to do to accomplish to demonstrate and at what level and then to design your objectives based on in essence that those kinds of questions. So you know for instance here we're looking at who are the learners and what do we expect them to be able to do how are they going to demonstrate it and at the level and so Ellen do you want to take take this and and go into a little bit more depth please. Sure sure sure I have an echo on my end okay it's gone so I just wanted to add a little quick anecdote my first degree is in post-secondary education and I remember I graduated from the University of Nevada Las Vegas and Dr. Clifford McLean and Dr. Paul Meacham who was the former president of the well it was formerly the community college of Southern Nevada which is now colleges of Southern Nevada. I remember that they took us through writing objectives in every possible way imaginable during that degree program and I just remember that you know that ABCD that I am that Andy's presented here and I'll tell you that more than my graduate work I have drawn upon this method and this what I learned in that program I feel like I've drawn upon that in my 14 years of teaching more than I've drawn upon my master's in communication honestly. I've been on assessment committees at the two colleges where I've been on tenure track and I feel like this has changed my teaching and altered the way that I've taught so much is just looking at just again coming back to the basics of who are our learners what do we want them to do what you know what conditions do we want them to be learning under and what type of measurement or standard do we want our learners to do we do we want them to be learning in and so I so this is just a very basic example and again I believe that when we put these together I went directly back to my undergraduate work because I still have those notes from so long ago and so this is the example that I was given way back when which you can read and every time that I need to write objectives now when we need to retool our course adoption forms and we just looked at those in my in our communication studies department not long ago for our 101 course I literally go back to this to just this very basic format and I try to just just do this process all over again and it helps me to use something this simple of this sort of a b c d so maybe this could be something that that helps you too and I'll give you another trick that Dr. McLean taught me in a minute that that can kind of or at some point in this in this presentation I'll tell you another trick that Dr. McLean taught me that kind of helps me think about things about assess they helped me think about assessment so let's look at let's look at another example so I thought this one was this one was kind of fun and it also you can see that there's a very specific there's there's it makes it very specific and also when I think about the student I always think about we always think about student centeredness if a student looked at at their objectives would they know exactly what is expected of them and I try to think about that too and what Dr. McLean used to say is if you turned these into questions if you started these off with can the learner so so could the learner if you turn these around and you turn these into questions starting off with can the learner if a student could ask themselves that question and they could understand it and and comprehend well he used to say that understand is a bad word because you can't measure understanding so I'm using the bad word but if a student could figure out what they were expected to do then you know that it's that it's a saw it's an objective that is it's an objective that is a viable objective Andy did you want to add anything at this point I love the Jimi Hendrix treatment the having it crystal clear to the students of what's expected to them and how they will be measured is at the heart I think of our instructional design here and I think that that you stated it well Ellen in that in how you presented the objective so nothing more to add than that thanks okay so moving on so now let's take a look at some actual objectives that we have seen so this is where I invite everyone to to weigh in so what are some issues that what are some issues that you see here Andy can't you can't you describe the nature of the work can the learner describe the nature of the work that's not measurable okay so not written from student perspective vague exactly introduce students which religion my we've got the idea um vague um not measurable um downright silly describe the nature of work and my comment was it's hard okay well just when you thought it was safe to go into the water here are some more I know it's as bad as understand so believe it or not all of these are actual objectives that have been seen in in in actual classes these are these are real these have not been these have not been fabricated do you think they get worse you ready what's really interesting and I think that I think that all of us as instructional designers would probably say the same thing that even strong I mean even objectives are just difficult they are difficult to write and there are a lot of courses out there and a lot of strong a lot of strong faculty out there who have just simply never a lot of people just have simply never been trained on how to write on how to write course objectives I have seen it over and over again where because we run a summer institute at Highline and on our standards standards outcomes and competencies committee we do a lot of work with faculty throughout the year and you just you just see you just see it over and over again and also being on this project getting to see objectives you know system-wide that are going on it's just it's just very interesting and these are not necessarily from all from throughout our system these are just from all over but it's just very interesting to see objectives that are out there that are that just that are just being used so let me move on so that's why this is so important it's so important because if you if you think we look at these if you think we get confused looking at these you can only imagine what our students think so now I'll go ahead and move move this over to Stephanie thanks Alan another piece I wanted to add just on your comments about people not being trained on how to write learning objectives a lot of times people don't even think to write learning objectives you often inherit them from previous syllabi that other people taught and and just sort of never really look at them again and I think that that for many people the the push towards alignment of outcomes is is a new thing to even consider and actually reading one's learning objectives I've often had faculty just who hadn't even really read their objectives so it's a way that to get people to really think about them so sometimes it's not that they don't know how which often they don't that that they haven't actually even really thought about it when you do think about it one of the things that can help is looking at the the verb wheel from blooms oh I see some chatter there in the chat is this is my audio not coming through very well is it okay okay I see people saying it's okay um I love this video this visual of blooms taxonomy and I'm sure you're all familiar with blooms and its layers of knowledge and sort of the base of information that a student should have and that that should grow in complexity as a student moves through a topic and it's helpful in in trying to determine whether the learning objectives that you have are appropriate for the level of your course but it's also a really this particular graphic is a really great tool for aligning your outcomes with what you're having the students actually do and what you're actually having the students show what you know so that you can see whether they actually accomplished what you wanted so for example um in the knowledge uh region of the chart um which is is the base of the blooms pyramid you've got things like draw identify locate uh label write repeat um just some basic being able to you know basic ways of regurgitating information really and then how might they demonstrate those things um in I mean what types of assignments might you give to let them be able to do that labeling or repeating or drawing or identifying well you could do and if you look at that outer ring of the circle uh you could have them do a recording you could have them create a dictionary you could have them watch a tv show and identify things from their right definitions or read definitions um so all sorts of ways and if you as you look into the more complex area so let's look over in the analysis corner you see um you know comparing debating differentiating and showing things by um creating graphs uh do having a a broken down breaking down an argument or writing a report um and I think that this is interesting because you often have people writing reports in intro level classes where you're expecting sort of sophisticated analysis for outcomes that are really based at the base level of blooms so this really can help you align is what you're expecting from the students actually in alignment with what you're asking from the students and if those two are not lined up um it's not surprising that sometimes people are disappointed with um with what students produce so after you sort of thought through the blooms you can look at your learning objectives and ask some of these questions and I won't read through them all but what it all comes down to is what can the student do and it sort of ties back to what Ellen had the question Ellen asked is is the student able to is the learner able to do x um if you can't say what the student can do um then your outcome is really lacking the detail that is necessary for it to be student focused and measurable I had a conversation with a philosophy instructor the other day and I was um asking him to be a little bit more detailed in the writing of his learning objectives and he was really quite offended and said that you know philosophy is not something where people can do something it's you know it's all about thinking he said I said well you know thinking is doing something and that you know you need to it's important to the student to be able to understand the benefit of engaging in the thought that one does in philosophy and if he can't even articulate that then how can he really expect the students to be able to do it and how can he expect to be able to measure whether the students can do it um after he got over being mad at me he realized that it did make sense and then when he finished um he was a lot happier with this class so here's uh some more comparisons and um I won't go through them all because I can see we're running low on time but I do want to just point out that if you look at the new objectives they've all got greater detail easily measurable and a student can really clearly see what they'll be able to do at the end of all that so let me let the mic go and pass it on to Emily sure so uh I'm going to be talking about aligning and and when we say alignment we really mean a couple different things and this is all based off of quality matters um which is a way of thinking about course design that was specifically designed for online and hybrid courses but which has value for all um I think all courses so when we think about alignment um I'm just going to refer to those standards um the course learning objectives describe outcomes that are measurable and that's something that um Ellen and Stephanie and Andy have already talked about 2.2 is the module and learning objectives describe outcomes that are measurable and consistent with the course level objectives so it's great if you have some course level objectives um to really guide the learner and say well this is what what you're going to be able to know or do at the end of this course but um in order for students to be self-directed learners having module objectives so they clearly know what they're doing week by week or in a self-paced um environment per module can really help them understand if they have gotten that module and they can move on to the next one and if they have a solid foundation 3.3 is looks looks at the types of assessments selected to ensure that they really do measure the stated learning objectives and that they're going to be consistent with course activities and resources so that's really making sure that what I choose for an assessment is going to be appropriate for those objectives and finding a way to align that oh and I saw Claudia says I usually include objectives in my assignment so even going further than just the module level to the actual assignment level and that's a really great way to make sure that everything rolls up um and is going to have the students progress towards those course learning objectives terrific so looking at objective alignment I have a couple examples here um and I think this one might actually be Ellen so if Ellen if you have anything to add to that but I'll kind of get us started on this particular slide looking at the course level objective which is to construct and present effective information persuasive public presentations so the module objective breaks that down a little bit she say what are the parts what are these small parts that a student needs to be able to do in order to reach that course level objective so breaking it down into those little steps and this is a really good way for a student to self assess their learning to say well I can do this number one identify the functions but I'm having trouble with this one and it's a really great way that the instructor and or the students can give um and get some pinpointed feedback on where they're struggling and then looking at well how will the student demonstrate the knowledge for this module and in this case we want them to do something and to produce something and that is construct comprehensive speech outline which really looks at the supporting material so that gets back to that idea of getting informative and getting supporting materials in there and then also using credible sources Ellen says it helps the students break down into smaller steps absolutely and I'm actually going to use an example this is one that is from my not my biology 100 class but the biology 100 class that is currently in development for the open course library and with the permission of the instructor who loves to share we took a look at some of the module level objectives that support the course level objective of identify and explain the basic concepts of Mendelian genetics and inheritance so looking at well how do we break that big thing down into smaller steps and and in the sciences sometimes there tends to be more objectives than in other disciplines so you can see that there's quite a few module objectives and all of these will feed up and roll into that that larger course objective how to make mental fun yes and then how will how will the students demonstrate their knowledge of this well we're actually going to have them do something we're going to have them use a punnett square to illustrate probabilities of genetic inheritance so that really gets that yes I'm I'm using I'm able to explain Mendelian Mendelian genetics I'm able to show you what that looks like and then to have them do some critical thinking explain which ones are dominant and recessive and write a short paragraph relating your findings to Mendelian genetics and inheritance let me just move on here quickly so this is this is a slide that really looks at another objective from a human nutrition course and I think some of the objective might be cut off for some people so I'm going to just read it the objective is discuss the roles of physiological cultural and environmental factors on hunger and satiation so if we look at all the activities and all the assignments and all the assessments that really support that large course objective we can see some different options broken down and this can be a really helpful tool for you to think of well what are all the assignments what are all the readings what are all the learning materials and the assessments that support each objective so looking at the assignment they're going to be reading some chapters that we're assuming are talking about hunger and satiety assignment is to view a documentary so they're getting different learning modalities here they're getting the videos and they're also getting some text and study guide and the great of assignment here is to prepare a table graphic or timeline to demonstrate how patient queue this kind of made up patient so that's kind of the condition is new information here may experience feelings of hunger and satiation from a physiological cultural and environmental standpoint briefly explain the related processes or phenomenon involved so that greatest assignment really does get at that objective and let's just imagine that there's also a graded discussion forum that says list at least one each physiological cultural and environmental factor that impacts hunger and satiation and here is a really key point which is why I put italics use scientifically reasoned real life examples to demonstrate your point so now if we took out this um this italicized sentence here and they were just going to go into a discussion forum and list one physiological environmental factor that impacts hunger and satiation what did that one activity be able would you be able to obsess that learning objective and why or why not so I'll just open that up for for a little debate or some perspectives from the chat oh sorry I need to repeat the question um so looking at the graded discussion forum question if we took out that italicized sentence to say use scientifically reasoned real life examples to demonstrate your point so if that was taken out of that assignment and we just looked at the graded discussion forum question would would what's there truly assess the objective and I see kevin kelvin is saying no um and I'll wait to see some other responses to so maybe we can think about why or why not everyone will see the same thing um Chris says I'm assuming they have learned what scientifically reasoned is prior to this assignment yes yes simply creating a list does not demonstrate understanding of a concept um oh great I love those that wording Audrey no because the first half measures knowledge the second sentence measures application that's absolutely right so if we want them to discuss something simply listing will not be enough we want them to see how they are relating it um to scientifically reasoned real life examples and um and that's why the graded assignment is important too because it also gets them to apply um nor does a list yep nor does a list need the objective of discuss and now let's look at the mid-term exam so just looking at the mid-term exam would would just the mid-term exam be enough for that objective to assess that objective kelvin is saying no and I think somebody already pointed out that um the exam on multiple choice exam does not allow for discussion so we're not really going to see it that um so it truly doesn't assess that objective well that's right and and um I think that and Jen Jennifer step in if I'm wrong but I think about a year ago is when I first had contact um with with Jennifer and other folks at sailor because um as far as I understand correct me if I'm wrong that um courses that are designed through sailor are um need to have need to be administered the assessments are administered in a multiple choice type format but that doesn't mean you can't get at critical thinking questions um and maybe this is a nice way to segue into questions is how can we create multiple choice questions that get at some of the critical thinking and that are aligned with our objectives and I'm just going to read back some of these comments here oh include situations um that's great and maybe maybe there's going to be um some example here like conceptual multiple choice questions I love that idea um and one one way that might be helpful for for others to think about how to write a multiple choice question that really gets at a critical thinking or an application skill is this might not be the appropriate level for sailor but if you look at um GRE questions the graduate record examination or even MCAT um or LSAT which are standardized tests for students going into graduate level those do a really good job at getting at critical application in multiple choice question format because they um they usually give the students or give the the test takers a situation and then they have them critically analyze it through multiple choice questions Dana says it takes a bit more time but it's worth it they are hard questions to write it's very challenging um and I I think we can transition from here to some of your ideas about um designing assessments that that um are aligned with the objectives but I want to see if there's any more slides after this one oh just talking about the next webinar so um I guess at this point should we open this up to other questions or if anybody else has ideas for how to make a multiple choice question align with objectives I would love to to hear some ideas um from our presenters and our attendees well one thought that comes to mind um stepping back from uh the specific assignment it's just saying to how do we write the questions is the notion that I think it was Dana who who uh expressed it this is tough work this is difficult for instructors who are trying to um teach a class uh and and deal with multiple uh uh demands on their time and resources to actually develop um this kind of material and it you know number one the it points instructors back to the publishers who've got um a ton of resources that are available for them that can do this relatively effectively but if we're going to have an open education resource in an open education community it also begs the question of or you know not the question necessarily but it speaks to the idea of sharing across disciplines and sharing uh within disciplines um to a much greater extent than we're all doing right now and my two cents I think it it speaks to the sharing aspect this is Ellen I just wanted to make a comment Andy when you mentioned the publishers um I just wanted to throw an idea out there that when writing module level objectives I was on um an assessment retreat a couple years ago and one of the ideas that we had is that when writing module level objectives textbooks actually can be really helpful um you know writing module level objectives can seem really really overwhelming because there can be so many of them but one idea that we had was when you use textbooks and going um the chapters always have those you know after reading this chapter you will be able to those can give you some great um great words and a great head start for writing those smaller module level objectives or at least get the get the terminology to start word smithing them and I love data's suggestion of getting the students to develop uh assessment material um the um one of the secrets of successful online teaching in my experience was to to do exactly that to get students to develop not just the the course materials by having them explain but um have them develop assessments problems I taught accounting so um students could do this that um uh you could use later you could recycle um again that's sharing this is Ellen again Pat had asked a question um what did I mean by word smithing them um what I meant was when faculty looked at those um objectives for that were written in textbooks for the chapters sometimes they didn't like the way those were written but they threw some of their own verbiage in there and then they felt comfortable with them but what they liked about using the chapter uh the chapter you know terminology as a spring board at least and then changing it up with uh words that were comfortable or word smithing them as I said is that at least they didn't feel like they had to reinvent the wheel the the concepts were already laid out um I just want to jump in here to second um the idea of using textbooks or using existing materials out there for those module level objectives I was doing an account for another project that I'm working on but in an open course library so far I've looked at anywhere from 650 to 1200 objectives um within the courses I've been working in if you consider the course objectives and the module objectives and certainly um not all of those were invented by scratch but um another great resource to go to um to get some module level and course level objectives is um national um societies um within the disciplines um associations um can be really helpful to see what other um scholars and experts um within a particular field what they think um should be appropriate for different levels of um of the courses um well I think that there have been some really helpful um ideas written here in the chat window and um I'd like to you know first of all I want to say this is gen shoot by the way um I will be making these slides available to everyone um who would like them I think that a lot of people expressed interest in that wheel graphic that Stephanie presented um and I think I'll also maybe put put some of these ideas that you've been sharing with us in the chat window down on paper just so we can um you know share them and maybe keep an open google document about it or you put it on Wigio or something like that um because I love these ideas that are coming in and um Gary yes we can uh we will be sending around the link to the last we can send around the link to the last recording there is a problem though with our technology at that time so you can only hear the audio which is so annoying but I can also send you at least the slides um if you'd like one more point I'd like to add is that I think in higher education in a lot of different contexts there's a big push right now for authentic assessment so to see um actual student products um so papers and um and presentations and and making reports um which which Andy was talking about a little bit but I do want to say that that when you write a multiple choice question or you write an assessment question and exam format um in a way that exposes students to a situation um and new information and they have to synthesize that information or choose the best options you can get us some critical thinking um but certainly yes that is one of the challenges of writing those um those multiple choice questions that get at critical thinking one other point I wanted to make quickly um for my sailor professors um so we do um as Emily said you know the only way we can really authoritatively assess students and issue a certificate of completion for a given course is through a multiple choice final exam that we administer in Moodle but we are open to developing whatever assessments you like throughout the course as long as there's some kind of way to give students feedback whether that's an answer key or a guide to responding um and you know we're much more flexible in that way it's just that since all the courses are for self-directed learning there's no way to have a professor kind of say yes or no this is right or wrong um or what have you but I would also say that the next um presentation and I'll just just a nice segue into the next webinar which will be in August um you will be hearing about our e-portfolio and discussion forms which we've just gone live with um and the discussion forms will be talking specifically about how you can build the discussion forms into the design of your course so that you know if one of your learning outcomes is you know be able to discuss XYZ um you can actually build a discussion form prompt into the course itself so that when a student hits that point um they will be able to participate and see what other students and peers are saying um in the discussion form so and that presentation will be by Joseph um who here at Taylor is our strategic technology manager and Devin who's our special projects administrator so I'll be standing around um more details on that front soon um let's see are there any I see there's a lot of positive feedback in the chat pane which is great um so you know if you guys have any other questions feel free to email them to me or um I can pass them along to our very helpful um presenters today um Andy Ellen Emily and Stephanie I am so impressed with what you guys have taught me I've learned a lot and I know that all of our professors have as well um so thank you guys so much um I don't know if you guys have any last words you'd like to say before we wrap this up I just want to say thank you um just reading through the chat comments you guys have come up with some really great ideas for challenging situations and like I said at the beginning I'm so excited to see what sailor is doing um and really taking education and making it free to everybody so I appreciate that and you guys have re-energized me thanks Emily I mean I I share the I share that same feeling I'm totally re-energized and I'm even thinking about how to plug some of the content um in these slides into our our trainings and sharing them with the rest of our faculty so I think this has been so helpful um and thanks again for being here all of you and thank you to all of our professors and for those who um are not affiliated with sailor I see a few of you there um thanks so much for tuning in and I will definitely make a recording of this available and we'll circulate them as well as the slides and some of the great comments that have come up through the chat as well okay great well thank you all um have a great afternoon