 Teaching like exercise comes from the right balance of challenge and support. No challenge, no development. We've said this before. UDL is not about dumbing down the challenge. It's about making the challenge be higher, if anything else. The idea from Dr. Lin at Berkeley is a wonderful phrase. Education is about posing desirable difficulties. Desirable difficulties. You don't educate kids by making things easy. You educate kids by making desirable difficulties. That is, you've made good choices. There are desirable choices about difficult things. It's in their zone of proximal development, which is to say not too easy and not too hard. Desirable difficulties. The central core using her language of UDL is UDL is about decreasing the undesirable difficulties. That are not what you're really trying to do, trying to teach, but are undesirable difficulties that get in the way. What Grace and I want to move to now is to look at assessment. Because assessment is a great way to look at the imposition of undesirable difficulties when you're assessing so that your assessments aren't in fact accurate because they're not measuring a desirable difficulty. The thing you really want to measure is measured poorly because you've inadvertently put in undesirable difficulties. Things that are irrelevant, that are non-construct relevant, and they're getting in the way. So we're going to do that. The first grace is going to go back to the guidelines, work through them a little bit to say, well, here's the issues. When we're looking at expression to pay attention to, then we're going to look at assessment, look at desirable and undesirable difficulties. Grace, you want me to open up that slide, shall we? Great. So please pull out your guidelines. You should have those with you. And the guidelines, as you know, are the guidelines for designing lessons or looking at assessment or purchasing equipment. So here we are in multiple means of action and expression. Thank you very much. And what we're going to do is just do a little review again of where we are. The goal of this session is really to apply the UDL guidelines for you to have the opportunity to apply the guidelines to assessment. That's always a major question, is what does assessment look like through the UDL guidelines? I want to just review again when we talk about the guidelines, what our goal is, and our goal is to develop expert learners, and by expert learners relative to principle two, is strategic and goal directed. That is the ultimate goal. You can see that on your guideline. And here are the characteristics. They formulate plans, and this is really just a synopsis of what David said previously. Effective strategies and tactics, organized resources in tools, monitor progress, recognize weaknesses and strengths, abandon plans and strategies that are ineffective. Again, you're noticing that we're not referring to one specific category of individual, really just referring to expert learners, and all students fall within that category. Again, reviewing just a tad. This is the principle level, multiple means of action and expression, options for physical action, that's the guideline. Here we have the checkpoints that go under those, and you know that when you go on the website, you can actually go right into the checkpoint, look at tell me more, and you can find resources that support. Don't fall into the trap of thinking. One resource equals alignment to the guideline in and of itself. Really, it's just a resource that supports. It's not one isolated event or one isolated technology tool that says, I've arrived. You really want to explore what your goals are and move from there. And here we have five point of the fifth guideline, which is provide options for expression and communication, and that's actually what we're going to be focusing on in the deconstructing and debriefing assessment tools. So again, this is available right out of our guidelines. And guideline six, this is external to internal, and here we are providing options for executive functioning, and David has spent time talking about that. So one more is just to refresh. This is what our guidelines look like. This is a major takeaway for you. When you go back sharing ideas with your colleagues, if you're understanding the guidelines, the structure of the guidelines, and how to apply them, you have done fabulously. From here, we're moving to, do you want to move to this example, David? No. No, okay, good. Okay, so this is the order. Go to the, there it is. Okay, great. So now we'd like you to be test evaluators. What I would like you to do is to look at this item, try to figure out what's the desirable difficulty, what is it trying to measure, and then I want you to look at the guidelines and decide what are some undesirable difficulties, that is difficulties it's posing that are undesirable. They're not about the thing you really want to measure. So first, George, what is it really trying to measure? Then secondly, what are the things it's measuring that aren't that, that are getting in the way? They're undesirable difficulties, okay? First thing, what's the desirable difficulty? What is it really, should be, remain difficult? And then secondly, what are the undesirable difficulties? Things that are difficult but shouldn't be, shouldn't be there, okay? So we look at the guidelines. Now this time we're going to use it as a summary for everything we've been looking at. So I'd like you to, right Grace, you want people to, you may. First, do it reflectively first. Just pause and you think about it on your own. And then you're not to be included by your peers at that point. So spend, we're going to have enough time that you really will have time to just think about this. This is what students are faced with all the time. And then once you feel okay, I get it, then turn to your neighbor and just share what your reflections are you sharing on. And then we'll do a share out. Okay, so a couple of minutes quiet. What's the guy called? Think time. Two minutes, think time from our guy on Monday. Share, we all know what that is. Okay, and then we'll share. So think about what is it, what's the desirable difficulty? What are we trying to really measure? And what are the undesirable difficulties that would interfere with our ability to measure that accurately? Is this a real item? Yes, but these are all real items from, you know, they have sample test items across the country. I'm sorry, you didn't say where this one came from. I think this was Tennessee. Okay, so now we'll be quiet. Now it's think time. I think I find the easiest thing is to just look around you to find where's the most attractive person and talk to them. No, just talk to someone right next to you, okay? Just one more minute. I don't know, do people use this kind of thing? What do people do to get? Do you do that? Okay, there's always a way. So that's the way. Good, thank you. Thank you, fabulous. Okay, so there's probably people who are math educators. So I'm less interested in what the desirable difficulty is, but so who has a sense of what would be the desirable difficulty here? Anybody want to venture that? That's often harder than the undesirable difficulties. What's that? Well, we'll get to the undesirable minute. What's the desirable difficulty here? Right there. Yeah, understanding patterns. Okay, that clearly seems, but it's a little more than that. Yeah? Square numbers? We need a math. Wow, are they really square? Okay, so it's understanding patterns, but patterns in numbers specifically in this case, how squares work, how... But the second one is that you're squinching up. No, no, her back there, she's squinching up. But you're squinching too. Okay, we've got the two squinches. Go ahead, yes. The desirable difficulty is that you could approach this in a variety of ways and it's not automatic, we don't automatically go, oh yes, two plus three. Oh, I see, good. So you're saying it's desirable that it's actually a challenging item, which I think is too. So this is a good item in that respect in that it's not just which of the four, you click on one of the four answers of whatever. It's got some thinking in here. Great desirable ability. What were you going to say? It's another problem that involves visual pattern there. So there's a couple of things you can play with. Good. So it does have some potential, as you said, alternate ways to solve it, but it does get us into actually mathematical expression. How would you express this pattern mathematically, which is great in saying we want a mathematical expression. But you first want to understand the pattern and then we can get to a mathematical expression. Okay, so we can see whether you can do that. Now the fun part. What are some undesirable difficulties here that we would be inadvertently measuring that have nothing to do with understanding mathematical expression on patterns. So what's an undesirable difficulty? Well, school over here first. What's an undesirable difficulty people here noted? Yes? Multiple steps. Okay. Multiple steps inside the single item, and so then you start to get into kids who have attention difficulties, all sorts of things. You're starting to measure can you sustain effort over a long bunch of items. But it does break them down at least. So I've seen worse in that regard, and you have to I think, right? But okay, so it's measuring in part, can you sustain attention and effort over multiple steps? Okay? Good. Yeah? I'm sorry? And B. B. You have to write. Okay, so the great one. Do we mean to be measuring how well kids can write? Is this item. That is an undesirable difficulty, because Matthew can't write. So what does Matthew score on this item? Zero. Okay, so you wouldn't, so we wouldn't want to say write an expression, what would we want? Obviously they can use, you know, if you just said generate anyway, say there's a lot of means, make an expression in terms of N, whatever, but you may not be able to hand write. Yeah? Okay, go ahead. Yeah? Yeah. I mean didn't you have trouble understanding some of these, I mean this is hard writing. And so you realize, well the writing is harder than the math. And for some students, the writing is way harder, I mean the reading is way harder than the math, the reading comprehension. So essentially, and that's what turns out, people have done studies and shown that 50% of the variance for an LD student, 50% of the variance in their scores is the reading. It's not really testing math, it's testing their reading. So if you give hard reading items, and you don't do other things that we can think of doing, now I hope after these three days, you essentially are confounding reading. We're measuring reading, and then we don't know whether the kid knows the math or not, because the reading is really hard, it's the hardest part of the item is to do the reading, and so therefore a lot of kids are going to flunk and we have no idea whether they know math or not. What's some other things, yes? Sure, explain how you got your answer, what about that? Good. Now what, I was going to, she's saying the show or explain how you got your answer. It's interesting, that one actually doesn't over narrow the means. It sort of is saying, you could do them a bunch of ways, you could draw, you could talk, you could, we've done a lot of examples of this where we literally have a draw button, a right button, a record button, so we say show or explain, but you choose the method. So that if it said draw how you got your answer, that's narrowing the means to one means and kids who are good drawers do well, kids who are bed drawers do poorly, so we're measuring drawing. This is at least saying show or explain how you got your answer saying, well it's a bunch of means you could use. So that one's pretty good, yeah. So if you missed B, you got B. So C is a dead, if you have trouble with B, you're dead for C. So what's the word? They're confounded with each other, right? One's dependent, yeah. Go ahead. Sometimes many times I'm really frustrated and say, well I can't explain it, it's very difficult, but then we reflect and really understand their thinking process. This is asking for really a lot. Yes. But this is not, I know it, so don't bother me making me, we can think about my thinking patterns and then communicate. Good, so what would you say is, he's saying that some kids could do this mathematically sort of immediately and just do the math. And what they would have trouble with is doing all this stuff in the middle and is the item want that? What if they just did it? What if they just went, oh, it's X plus, X plus 2X plus 4X plus whatever and they do it right in the first, then the second part becomes now right about it or talk about it or whatever and you can picture kids on the autism spectrum who would immediately know the answer and not know how to express it in words and is that wrong then? Okay, tricky. A little bit louder, sorry. Can they create an expression that the whole thing's pointing towards along with the algebraic expression that can come in different ways and show it to you. Good, now you're saying so the bottom one is mathematics, it's a form of expression, right? I'm not able to hear you quite well. You're strong, kids mathematically frustrated because they know how to do that. Yeah, right. Okay, great. So we'd have to be very careful to say what is the desirable difficulty we're really after here in order to decide which of these things in the middle are useful in which are just a pain in the neck. Okay, other things. Look at the guidelines. We've not covered a bunch of things that could be and you don't have to stay within expression. There are a lot of triangles of triangles and if somebody has difficulty with vision they might see triangles within triangles or just four triangles of different sizes. Yeah. So the fifth step would be another triangle. Great. Remember I showed you this? Oh, I didn't. Yeah, there's a great slide between some kids see only the big pattern, some kids only see the details. So in fact, that's a very different drawing to some kids than others. And some see there's one triangle, four triangles, and you know whatever. Okay, and for whom is this completely inaccessible? The triangles. Yeah, someone who can't see. Alright, so then you say okay, here's the item and it says use the triangles to make the first 4,000 pound and do that. You can't do the item at all. So if you look at the top row under just the representation you go is something visual also presented in a way that's not visual so someone could do this. Okay? So what would you do? How would you do that? I won't answer it right now, but I think you can probably think, but anyway, so you want to make sure everybody can perceive it. So putting something in a drawing is really great for students for whom all the text is a problem, but then we need an alternative for the fact that you can't see. I don't have a strong math background so I'm not overly familiar with the curriculum probably as much as I do with people in the room. Is there in fact a mandating in the math program for a child to have to write an expression, and if there is such a thing then how do I as an LC in my school ask that my math teachers kind of juggle around what's being asked of the student on an exam, and is it possible to assess them in a different way without necessarily looking at those described curriculum objectives as writing expressions that still know that the mark on a repair card is valid, as much as you need to always do it in terms of marks, but how can you represent that on a report card with validity? She's asking about this word. Do you have to be able to write in mathematics? And this is where we would say this isn't overspecifying the means. It's saying there's one mean, right. There's lots of ways you can generate a mathematical expression. Matthew can do it with his chin. He doesn't write. He generates on an iPad. He can do all sorts of things. There's a million ways, and some people this word is ambiguous, but a lot of people it's not so ambiguous in this this one looks like you're supposed to write in a mathematical expression. Where's the one that there was the explanation, or explaining how if you say write, how you got your answer then we turned it into a writing examination. But if you say show or explain in a bunch of ways, then a lot of kids can do it in different ways. But you were getting cranky in the back about this. Great. They required to show all three ways. Fabulous. So you're already doing Fabulous. But required to show all three ways is very different. They have to do all three. Okay. So then you'd have to worry. Okay. Great. What you need is show at least one of them. Yeah. The word required is the problem. I see. That's very interesting. Great. Yeah. And required and. Wow. Okay. Great. Well then what we want to say is you need to express this and here's the alternative ways you could express it. Right. But what we don't, everybody needs to do it in a second. What? That's right. All four are possible. And you'll be demanding. They need to show that they understand the math can't just draw pictures. But if you say you've got to do this, this, and this, then of course kid who can't draw Matthew all of a sudden he looks like he can't do math, which is ridiculous. In fact, who wouldn't be able to do math? It's wonderful because you can look at the people. My favorite story. Sorry. Great one. If you ever want to hear a guy give a fabulous talk about assessment, it's the guy who the movie Jurassic Park, remember was made around a specific guy who was a, what do you call him, not an archeologist. What do you call him? An oceanographer. Paleontologist. The paleontologist, he was in high school he was already into paleontology. He was a brilliant paleontologist in high school. He was studying dinosaur eggs and, you know, was amazing. And you know what? He won the state science fair three years in a row. This is a guy who's really brilliant. But when he took his tests in his regular physical science class and his biology class, he got out. He flunked. He would be flunking general science and he was part of the top level scientific exploration of mammalian development and flunking general science in high school. And it's wonderful to hear him talk but you think, was anybody paying attention in that school to go, maybe we've got something backwards here. He's a scientist. He's already contributing to the literature and we're flunking him. Maybe it's the test. And so you ask him, if anybody asked him, why do you flunk all the tests? Well, because they always asked him to write or to read and he was dyslexic. And he's still dyslexic. He still doesn't write his own papers. He works with other people to write his papers but he's brilliant. So what happened was all his tests were measuring his writing and reading general science. And if they took a little time to back to the way it said, can you draw what's happening in da-da-da, can you tell us about what's happening? He would have given them a long speech. He would have been fabulous. And he won some big presidential scholar award and George Lucas makes a movie about you. That's pretty successful. Weird that we've been flunking him as if he didn't know science.