 I think we'll go ahead and get started. I'm pleased to introduce you to Robert Talbert. He's been using a new technology, a web-based technology, in place of ice pickers in his class, and he's going to demonstrate how he's been doing that and share some expertise with you. So let's welcome Robert Talbert. Thanks a lot. Before we begin talking, let me just say that if you've signed up for the used learning catalytics through the email that I sent out the other day, if you would, just at a moment when you're not surfing Facebook on your phone or on your iPads, go to LearningCatalytics.com and sign in and join session 167499 and if you do that successfully, you should say wait for the question if it's going to be revealed or something of that effect. I've made it to forget my password and my login. Do you have this session coded? Oh, it's right over there. So this is a true demo of the class. Okay, thanks for having me do this. I wanted to give this talk for a couple of reasons. First of all, a lot of us are using clickers in the classroom. I'm thinking about using clickers in the classroom. I feel like it's really important for us to share what's happening and what we're doing in the classroom with that particular technology. So I hope to do that today in the next 40 minutes or so. And also, I received a grant from the QFTLC to implement the software. And since I'm getting all this money and since I'm messing around with one of the most important classes we offer in the curriculum here and just totally working with this, I feel like I owe you an explanation as to where all this stuff is going on here. Hopefully, I'll do that too. So here's the overview of this talk. We're going to talk about peer instruction, first of all, which is a pedagogical model that I'm using in that particular set. And a lot of my other courses, too, that sort of gives rise to the problem for which this technology is a solution. Okay? Then we're going to talk about classroom response systems in general and clickers in particular and the switch to this so-called bring-your-own-device approach, which is what you're doing now. You're bringing your own devices and do a little demo of learning calculus where you get to play around the software with your own devices. And then finally, I want to spend some time talking about how this is working in math 227 currently. We're kind of, obviously, a little past halfway point in the semester has some data. It's an interesting pedagogical situation to share with you and you can kind of role-play with this. First of all, what is peer instruction? PI stands for peer instruction. Peer instruction is an instructional technique that was developed by Erdman at the Business Department University for use in his intro physics courses. He found that, and stop me if this sounds familiar, but he found that his students, his physics students at Harvard could make it all the way through his class and pass beautifully through his final exam calculating everything known to man that had no idea what physics was because he gave them some questions about basic physics concepts. Basically, they were operating on a level pretty much like a fifth century model of physics. So they could calculate their way through anything, but they really didn't know any physics because the conceptual knowledge just simply wasn't there. So he developed peer instruction as a way to bolster that, to shift the focus of his class as to conceptual understanding rather than procedural understanding. A typical peer instruction session goes like this, a 50-minute course might be arranged around three to four main concepts, three to four essential ideas that are developed in that particular class. There's usually some degree of flipping that goes on where students have to do some reading and some activity outside of class. Each concept is presented briefly by the instructor, maybe a five-minute mini lecture. And then students are given a conceptual question, a question that is targeted specifically in one of the fundamental misconceptions about that concept, which, again, put yourself in the shoes of teaching calculus this way. And you can think about break your classes down into concepts of what do students typically misunderstand about those concepts. So a question is thrown up on the screen or given to the students in some way. It doesn't involve a lot of calculations. This is a conceptually-oriented question. Students, consider that question for one thing, a completely silently thinking to themselves. And then they vote. This is typically thought of as a multiple-choice question for students that have responses. And they vote using a classroom response system. Some means of voting on a question. That could be as low-tech as a colored note card. It could be as high-tech as something I'm going to describe here. But in some way or another, they vote on this question. Depending on the percentage of students that get the correct answer on this question, that determines what the instructor does next. If fewer than 30% are getting this correct, these are just round numbers that are just general idea, general principles here. If very few students get that correct answer on this, it's probably the instructor's fault. Probably there's something that needs to just be revisited in a different way. So we just loop back around, revisit the concept, and then try it again. If a middling number of students are getting that correct answer, then what happens next is that students are broken into pairs or threes, ideally with someone with whom they disagree on this question. Their task now is to argue in favor of their answer and against all the others. So a very rigorous, vigorous debate takes place. So students argue in favor of their right answer and against all the wrong ones for a couple of minutes, and then revote. What typically happens on the second round of voting is one of two things. Either we have more than 75% correct, or it's exactly 50-50. Very close to exactly 50-50. So it's an interesting, organic settling takes place there. If we have more than 75% correct on that first round of voting, pretty much you can trust that the students are more or less understanding this concept. And then we move on to the debrief, thinking of anything that's left over. And then we move on to the next topic. Yes, sir? Well, obviously, in the first case, you don't tell them what the correct answer is. That's right. Even in the second case, you don't even show what the votes were for. And that's it. Some research has been done on this, and if you do a first round voting, and you don't want to say, oh, 60% of your voting for A, so let's do a revote, because then, you know, almost 60% of them in the minority have better, you know, you just say, we don't have a strong consensus on this topic, so let's vote again. That's probably the best way to do it. So that's a typical peer instruction mode. And this might happen two or three, four times in a particular class you do. So it's very heavily driven by students with a few discussion. No instructor in a mention is taking place during the second round of voting, just answering clarification questions. Now, the key bottleneck here is right there. Typically, the classroom response system is one of these guys, one of these clicker systems that I think we've all seen around here. A number of us have a box, and he's sitting in the drawer somewhere. I have a box. That's a clicker, a typical turning point RF response card clicker. And it is what you see. It's a radio device, pretty much the same as your garage door opener. It's the same technology as your garage door opener. You can click a button on here. It releases a radio signal. There's a little USB device that sticks in your computer. It receives the radio signals and calluses. It's a very, very, very simple device. Let's talk about those clickers for a little bit. Well, first of all, here's a typical peer instruction question. This is actually, I gave this in a Calc 2 class, but it was obviously a review of Calc 1 moving into an application problem here. So you can see there's not a lot of calculations to take place on there. There are no calculations that you could do on this. But you can see where you might not get full consensus on this kind of a question. Some students may think like, you can't tell anything because there's no formula, for example. Often when I give this question, that's the number one answer. Well, I don't know the formula, so I can't tell. But in fact, you can tell information about this. Students can engage and teach each other on this. That's why it's called peer instruction. So five questions, multiple choice. Let's talk about the clickers. Oh, no, I don't want to talk about that. People always ask me, well, what evidence do you have that this is actually successful? It's actually about 20 years worth of scholarship and teaching and learning that weighs in in favor of peer instruction. There was a gigantic, epic study done with 6,000 students across multiple universities and community colleges by Richard Pate of Indiana University in 1997 in physics courses using peer instruction, some not using standard lecture. And he found that students in the peer instruction courses were showing improvements on this test of force concept inventory. It's a standard conceptual inventory for physics, showing gains of two standard deviations higher than students in straight lecture courses. And this is a massive sample. It's a seminal study in this literature. Students definitely across all genders, across all institutional types showing massive gains, significant gains over straight lecture courses. Students in these peer instruction-based physics courses, interestingly enough, to get these conceptual questions in, you have to take a lot of computation examples out, because you can't create more time in the class session, so you do less computation from the zero found, was that students improved on their computational skills with less direct instruction on computation because they're focusing on conceptual ideas. It's counterintuitive until you think about it, I guess. If you have a better basis for your conceptual understanding, you're going to know what to do when you're going into a computational situation. This also bridges the levels of playing field in a number of ways. This study here showed that, in one particular course, the gender gap between performance on the force concept inventory in a lecture course was effectively eliminated when you moved it to a peer instruction course. Finally, this pair here, this is a study done very recently, gave students a pre- and post-test on the NESI, the National Survey of Student Engagement, some of the sub-items from there, both before and after a peer instruction course that showed some significant gains on student engagement. They measured the student engagement using this particular survey. All the vectors are pointing in favor of peer instruction as a pretty handy and useful way to teach. But what about those clickers? That's the technological bottleneck between success and failure sometimes. You can use peer instruction, and people do all the time, with no technology whatsoever, just using colored note cards. A lot of high schools, for example, that don't have the funding can do this. There's a lot of pluses to these clickers. They're extremely simple. They're very robust. You can practically run this thing over with a automobile. It's still going to work. I have not experimentally verified that, but I believe that's probably true. And despite what they look like, they're pretty low-tech. Basically, you click a button, it produces a radio signal, the radio signal is received. It's very hard to screw up using clickers from the tech support standpoint. Well, it's got to be set up right. But there's some downsides for this, too. They are expensive, and one of these little guys here will cost $40 new. They've got 20-ish dollars views. I've seen them for as low as $8.99 on eBay. Still, you know, the student comes in, they have all this other stuff to buy, and they have to go buy a clicker. And that gets me to the second point. They're basically unit-asking devices. This thing has this totally useless outside the context of a classroom. You can't, unless you can get in here and hack it to actually open your garage door, which I'm not saying you can't, but maybe not. It's pretty much, you buy this for your classroom, you're never going to use it again. Let's face it. And they're pretty limited, pedagogically, too, because what sorts of questions can you answer with this? Multiple choice and true-false. Okay, and if you have a question that you feel is really important and conceptual, say something that involves a graphical response where you'd like the students to draw something or submit text for something, or a short answer, or even just a simply multiple-select question where you want to check all the following that are true, you can't do this. So you have to shoe more in every question that you ask in parents' direction into some form of a multiple-choice question. Okay, it seems kind of awkward. So what is the next evolutionary step in using clickers? Well, how about let's eliminate the added expense of owning advice and make it so that there's multiple forms of input? Okay, that would get us to something like this, or something that a lot of you have in front of you. Okay, now it's not necessarily the case that every student owns a portable device like this, but if a lot of them do, most do, I think, and that's what we find out. So here's what I want to talk about, learning catalytics, because learning catalytics is a classroom response system that instead of using physical hardware for clickers, uses a web-based platform. So I'm going to invite you now to participate with me on this, and while you make sure you're logged in, I'm going to do a little bit with my displays here. If you haven't signed up already, you're just showing up, you can go to learningcatalytics.com, sign up for a student, not an instructor, but a student account with that access code right there, and the session number is 167499. This is what my students do on the first day of class, or even outside of class, so it doesn't have to take place in a particular spot. I'm going to mess with my displays here just for a second. Come on, you're doing that. Take that off. And let's just put this over here. There we are. And let me show you how learningcatalytics works, and you will be able to follow along. Here it comes. So what you don't see, I have to set up now where, does this set up in a dual monitor kind of situation where I have one thing on my screen, and that's the other monitor, so I have something else showing up on this screen. It doesn't have to be done that way, but this way I don't have to keep shut on the window so you don't see the right answers to the questions as I ask them. So I have a question for you. If you're logged in, you will now see a question show up on your screens. Okay. So that, I have it up here on the board, of course, too, but that doesn't have to take place. So if you have your login, I just served that question out to you over our Wi-Fi network. Okay. So just as you're asking this fun little question here, here's the latest, largest known prime member fresh off the presses in January from Central Missouri University. If you took that number and printed all the digits on a long single strip of paper using 10-point font, it would stretch from here to fresh, Kirchhoff, Jemison, and Ski-Gun, or Chicago. So select the one, this is a multiple-choice question, so select the one answer you think is most correct. What I'm seeing up here, and you don't see, is as you're selecting your, making your selection, I'm getting a little histogram in real time as to who's selecting what. So I can see answers come in. You can see for yourself there's a little pie chart up there. It shows me what percentage of registered students for this session haven't answered yet. So I can get a sense of how close we are, about two-thirds of the way there. Do you see what I mean? I do not. Can you? I think I can. We might put fake names. Yeah. Like the kernel is one of the people. Yeah. I've never used that for all these years. You may not be here, though. All you have to do is be logged into this account. I mean, I had a student back when we had all the snow days a few weeks ago, a student who lived in, you know, near the Lakeshore, couldn't make it in, but she was still able to participate in this. She didn't have the benefit of the setup either, but all she has to do is type in a web address and you can do that for anywhere, right? Okay, go ahead and vote in your responses. I think we're going to have an opportunity to do a true peer instruction multi-round voting here in a second. I can see that there are 13 of you registered and 11 of you have sent in stuff. That might be John and I were both having problems with the Wi-Fi. Yeah, and then here in comes the... I didn't see your reply. Yeah. We're using the web... That's right. Is there an application as well for the Indian food market for... No, this is a pure web browser type of thing. Which has its pros and cons, right? The fact that it's running only in a web browser means that a student can use any device that they have that has a web browser on it to access this. So in my 227 classes, we have smart phones going, we have iPad, full-size iPads, many iPads, Kindle fires, and Barnes & Noble readers, laptops, a whole zoo of technology going on here. Okay, so I'm going to go ahead and stop the delivery on this and I'll show you the results. The correct answer is actually the ski. It's about 27 miles long if you pass the strip out. There is the anonymity, right? Now, in a pure instruction environment, I would say like, okay, I would probably not show you the results and to say, don't have a strong consensus on this question. Why don't you turn to your neighbor and argue in favor of your answer against all the others? Okay, I think we do everything though. So that was a straight multiple choice question. Here's a different kind of question coming your way right now. This is taken from a Cal2 that I've worked for has demoed this. And this is a check all that apply kind of question. So this is something you can't do with a clicker. Unless you make an extremely long question like you have A, B, C, D, and you have A, N, B, A, N, C, A, N, D, B, N, C, B, you know, list all the permutations. So check all that apply. Which of these is an improper integral? This is an actual question that I asked in a Cal2 class using this technology. As are all the remaining ones, Cal2 and 227 and 225. Is that because they're all missing DX? Does that make them all improper? No. What are you using? Latex. You use a straight latex. It also uses markdown or straight text. So you can just type latex directly into that. In 227 this is important because I want to ask questions about matrices and it's just straight plain latex. No frills. Does this upload them in a TAC file that somehow impacts the idea? I'll show you that in a minute. The basic answer is when you author a question it opens up just a text with a type of writing. Type of writing. Or yeah, you could, what I usually use, type it in the editor, get it looking right and dump it in the browser. Alright, so I'm going to stop the delivery on that too. We have a strong consensus on that and it shows us 75% correct. No one's going for the wrong one, but some people are not going for the right one. That's a little bit more finely brained information that you normally get from clicker questions. I think that's kind of neat. Getting more into stuff you really can't do with a clicker. Here's a question. A standard calc one question. What you have served out to your screens now is a picture. And this is a question we always ask about bread and butter calc one question. You know, give it a graphical function, draw the graph as derivative. And what you can do now is to simply draw and stick your finger on your screen or take your mouse and just literally draw it. What I'm getting while you obviously can't do that with a clicker. This is a a big gainer over standard clicker devices. Now what I'm getting as you submit your results, somebody's submitting it, but as I get them I have little thumbnails of what you're seeing and this is going to generate a composite sketch for the whole class, which I can then turn around and show and we can see the commonalities and discuss some of the differences. I'll show you that composite sketch as soon as we have a few more people. Well people can ask a quick question. Sure. On the previous screen after we submitted our response and you showed us I got on my on my device, I got a little there's a little prompt that said I get it now or I still don't get it. Is that just an opportunity to give an additional feedback for the instructor? That's right. That's right. Especially helpful when there's two rounds of voting going on because as soon as we can click there's two rounds of voting and they still don't get it. They can click I don't get it, I still don't get it and that will show up for me and I can go drill down and see who said that and they give that to me later and say hey I saw you still didn't understand what was going on with this question and respond. So I also noticed that the percentages were coming up but didn't you say earlier that you don't like to show those percentages? Yeah, let me just give I'll show you a few more things what I can do is I can stop the voting and read the results so I can say there's no consensus but I can say I control when the results are shown but I can re-deliver a question without showing the results. Okay, let me go ahead and stop that delivery and I'll show you what this looks like. Here's the... So you can see some strong consensus in one smiley face and okay I'm very nervous about it Yeah, you can see that in a lot of people I can see this is multiple, right not one person that's dragging a finger back for many times but that's okay that shows me some consensus in a quicker situation I have to create several potential solutions through this as graphs and have students click which one you want that sort of gives the game away but you want to do is see if students can construct that graph without having a prompt without a derivative just a very natural memory so this takes the place of the pencil and paper Next question here is another sort of graphical question Here's a graph tap or click on the point where the derivative is maximized a sort of bread and butter calculus question so this is a heat map question what's going to happen is you just go and try to click on the correct area and what I'm going to be getting up here is a little map that has like the clicks it shows me the answer where the clicks happen and to offer this question what I have to do is I have to go and draw a little box around a region but I'm going to consider it to be the right answer okay and so any clip that falls within that box is going to get a green dot any clip that falls outside that box is going to have a red dot on the previous one did you have to enter what the right answer was in some way did you map the boxes or something did you have a white answer to it that was just gathering data and then I turned that back around to the class so that doesn't have a right or wrong answer what do you think this is yeah as a student you can go back this is all archive every answer you give and every question is archive with the right answer so you don't have to post the quicker questions later it's helpful for review for exams and that sort of thing I'm going to go ahead and stop and click into ahead actually we can do this we don't have a strong consensus on this particular question so I'm going to stop delivery and deliver it right back to you for round two so I want you to turn to your neighbor and discuss are you vociferously or your answer against all the other ones I have, um technology I have I have I have two people could we have two things and we talk them My answer, and we talk a little bit about it in 20 years. I still haven't talked to the smart phone or some of the smart phones, because you know, I see you guys in the area, because I don't know, and I also don't have any work I'm doing, so I'm in a case that doesn't manage work, and everybody has to go out. Okay, we have to reload it. That takes a little longer. I'll have to read to one of the best people. All right, so here's the meeting. If you're really taking place here, let me show you the results from round one. First of all, this is round one, so you see the right answer is quick. And 55% of you got this right. But here's one. But after debate and discussion and review with each other, here are the results of round two up to 77%. Okay, so something magical sort of happens here, right? We have not a real strong consensus on it right now. I don't think any of us would be satisfied with 55% correct on anything. If you just turn it around to the students and let them discuss it, then usually they can work themselves out. Right, and the two that are coming up right here, what's going on with that? The negative side. So we can sort of have a discussion about, okay, do we mean by maximizing we mean more just negative or only more just positive? For the most part, I mean that's a conceptual thing that we now have picked up. Okay, we can repair, we can consider that to be a misconception. Is it a misconception or just for the right question? Okay, so we can talk about that. But it's something that we can pick up and repair now rather than the exam. Which is sort of like the main thing. Last question I want to serve out to you is a text-based question. And this is, again, something else we can't do. Just what's the question you currently have about all this? And you can just make out your devices and text it out. It's not really texting it out. It doesn't use SMS. It's still a Wi-Fi network. But it's still, you're entering in a verbal response of some sort. Once this is over, I'm going to do a quick demo of how I'll offer these questions. And if you can see some of the options you can have, I'll answer you. Just looking at it. Okay. Okay. Most rings have kind of a macro problem, but I don't think that's the case. No, I don't think so. We had this discussion before the couple started talking about that. Any questions for me while we're entering the text here? Well, I answered it. You get to see it. Come on, see these. Okay, let me go ahead and stop. I'll show you what I've got. There's a number of ways you can add the text information. What is this through a straight paragraph text and so this creates sort of like a Google form. Are we asking to type the text in Google form? But what I have going on here is it's going to create, I'll show you the answer here and stop. It creates a word file. So I guess we've got a little cloud here that the font size is proportional to frequency. So like the bigger words and the bigger issues we can sell off. All in there lives the kernel. All right, all right. Use this sparing thing. Okay, so that can be useful for like an end of the class one minute paper type of thing where you ask the students here, what's the most important point of the day? This can just look at this and see like, okay, a lot of people say second derivative or something like that. Well, second derivative is obviously a lot of people's minds. So let's plan something. I'll change my teaching plan accordingly or you can do this in the middle of the class or the front end. Okay, let me talk a little bit about, you might have some questions now about how to author some of these questions and how to put it. So let me get back to single display mode here. This is kind of dashboard that I see. If I wanted to create a new question, well, let's say in my department seminar if I want to create a module, this group of questions that I've given to you is called module. Say I want to create one, split the module and let's say like a demo module date, sure. I can have, you just do a synchronous which means that we're all answering at the same time. You can also set this up to be self-paced or self-paced and rated, which is the self-test thing. I experimented with this in 227 with having students do quizzes outside of class. I have a few things to say about that. It didn't work so well. I'll talk about that a little bit later. And this is a team-based assessment. It's really interesting. You set up a module so students will come in and answer a bank of questions individually first and then go into a team and answer them as a team second. And there's some rules for doing this. Stick to something simple. Synchronous and this. I can change this up. I can change how the credit is assigned. If I want to create a new question, it's two ways to do that. I can do it manually. And this is what you're asking about. This is what you see when you offer a question. Up here all the question types. I can enter in text and have students highlight the text that's up on the screen. Like what's wrong with the following definition, for example. Interesting enough. I haven't had a chance to play with this, but students ask a question and the answer is you take a picture of your phone and submit it. It shows all the pictures. So what could possibly go wrong with it? I feel like I can go here. Long answer and any choice matching questions. Let's see. Multiple choice, numerical or your answer. You can enter in numbers and you can give it a tolerance level around the number that you like. 18 kinds of questions per region, which is what you guys play with while we're down. Sketching, word cloud, short answer. Or you can just create a slide and information just to set things up. So I think there are 17 types of questions at this point. If you have composite sketch, confidence level, collecting data. One that's very helpful in Calc 3 is direction we can draw a vector and it specifies what the right is. You have to get the magnitude in the direction right. You can plot a vector in R2. You can even enter in latex expressions and also latex and tell you if you have it right before. What's the derivative of cosine to 3x? You type in the latex and of course that out in terms of whether it's right or not. That's only enough. You enter in the prompt, you have an option. You enter in your answer options here. You have an option to share. I'll mention that in a minute. You can tag your results. Why should you share and tag your results? Because another way to enter in the question is there's a library of questions available through learning catalytics that it's kind of small enough because this is a soft piece of software. It's just getting ramped up. But every time, for example, I offer a question and tag it and ask it to be shared, it gets dumped into a common library. So if I wanted to say, let me go get a question. You don't have to actually offer the questions in other words. You can just sort through what other people have done to pick one. So, for example, let me name my name from this and let's see something like physics. New word requirements, anything. So here are a bunch of physics questions that I'm just on. Absolutely good. Let's see. Here are some more. Yeah, draw a picture that best describes your attitude. Okay. Here are some things that are just been offered by people using the system. So it's pretty nice that you can search through. I really need a question that's a word cloud question about solids revolution or something like that. There may not be anything, but at least you can search for it. When you created this graph question, so do you create the graph on the system or do you have to create the graph elsewhere? I create that elsewhere, but you upload it as a simple jpeg or a pane. I just created that and I'm seven. Yeah, maple or whatever. I just export it as a pane or a PDF and upload it. How long per question do you roughly take to offer a question? Anything? I don't know. It's really hard to make a good multiple choice question. You have to be very mindful that the foils in the question are not totally obviously wrong. That you have enough of them and so forth. I find that it doesn't take me very long at all except in a few cases where I really got to get this one thing right. Because you only have time in the peer instruction class. So I've got to really nail it. So you have to think very carefully. This is what the majority of my prep looks like. I have to think really carefully about what students normally get wrong about things and the right questions about that. And it's right at the seams. So let students wrestle with it in class. I don't find that I take any more prep time that I used to do when I was doing the lecture. This is very, very different stuff. There's also there are several problem things that you would apply to one time. Yeah, the huge Halit textbook I think now actually has a peer instruction problem to it that's specifically made for peer instruction. So they're really important to us. And of course physics is loaded with this because this is really arising out of physics and research. So that is a quick tour of learning catalytics. Any further questions on that? All right. This sounds like such an amazing thing. What are the drawbacks? To get to the drawbacks I think you have to put it in context. That's where I'm going to kind of have an answer with this. Let me talk again about implementing this bring-your-in-vice furniture. So I'm first trying this out in May after two or two last spring. Instructors can get a 30-day free license to try this out. And of course in the spring course, that's perfect. It fits around top in the spring semester. It's four weeks long. And the students loved it. We had regular clickers for a while and the 30-day license ran out before the semester ran out so we switched back to clickers. And students were mad. They were mad that they were not using learning catalytics anymore. It's quickly terrible. So that was pretty successful. And some of those questions you saw were from that course. And so in the fall, I wrote this grant to the FTLC for a Pew technology enhancement grant. That got us 60 licenses for students. Students purchased a license for the semester or $20 for the year, I think. And it's unlimited classes. So if you're taking a form course load and everybody's using the system, you pay it once. You've got no technology left over to do nothing with later on. And we also, being sensitive to the fact that not every student owns one of these devices, we also use the money to buy nine of these. These are the Nexus 7 tablet devices from Google. They're the cheapest thing that I can find and they offer one that really works. So we bought nine of those. It was supposed to be ten, but I'll get to that in a little while. Ask Lori too. The student licenses, are they only for the semester then? You buy for the whole year too. But then they expire after the year is up. Yeah. And so right now, get my two sessions of the 227, that's what we're using. We're using peer instruction, we're using catalogs. So what I want to do now is just kind of walk you through some, maybe not all of these, but some of the more interesting results that have come up here. This is totally anonymized data that's real data from both of these sections. I just want to ask and invite you to put yourselves in the shoes of the instructor and like when you see this data, what do you do with it? Keeping also in mind that at least we have the data that we can do something with. This is pretty important. So the question was, look at these three vectors in R4, do these vectors span all of R4? And the options there in case you can see are yes and I'm sure, yes but I'm not sure, no but I'm not sure, and no and I'm sure. So it's a yes-no with a confidence interval wrapped around it. And this was a section two. In the first round of voting, we had about 57% of the students voting the right answer. In the second round, only where more students answered correctly, they were answering more confidently too, which I think is a pretty nice effect there. And again, in the second round of voting, there's no instructor intervention whatsoever. I just stand in the middle of the class, say if you have clarifying questions, you need to ask, ask them. Otherwise it's all done. So that really I think illustrates the peer instruction effect, both on the correctness of the answer and on the confidence that students have. You can see the 10 get it now, Robert, how often do you have from standard on that criteria? One minute for the first round of voting, two minutes for the second round. Right, so if you add in like a five minute mini lecture, of course about five minutes for the whole deal, maybe two minutes of debriefing, then you can get about three or four of these out. And that must assume that the students are correct. That's right, that's right. So there's a little bit of flipping that goes on in my class, so they have to be reading some more of the work outside of class. Here's another one here, for section two. So they have the system that's given an overview special on form, and the question is, what can you conclude about the system based on the following? And so we have, this is a select, this is a multiple choice one, this is just a question about how many solutions does the system have. You can see that students are really understanding this concept, I don't need to explain this anymore. I might have some beautiful pristine lecture set up to explain this some more, or something else really. I mean, there's very few students who are not getting this at this point. Now if there's a follow on question for this, I use the same system, and the question was, which of the following does describe the solution set? The options here are, the solution is a single point, the solution is the line y equals x, the solution is the line y equals negative x, the solution is a plane, or none of the above. The same system, but not a lot, I mean, I guess about two thirds of the students who do is if you look at these data, you're up here, the students are out there, it's real time, you see the data come in, what do you do? You now have the opportunity to do something. What is the basic misconception? What do you see here? 64% of the students are voting for the correct answer that the solution is, the solution set is the line y equals x. It doesn't label the percentages on here, because they're too small, but 21% are voting for the line y equals negative x. So what exactly is in this conception here? Where the equal sign goes, right? They want to put the first column equals the second and put an x in one and a y in the other. This isn't really a problem about linear algebra anymore. They understand that the solution is a line, that's a big idea. So I can conclude fairly certainly that about 85% of the students, all with the same number they got the first one, not coincidentally, are really basically getting this correct. The only thing is how do you interpret their outcome? That's what I've got to focus on next. I might have had some idea like, oh man, you guys still don't understand why this is a line? You must have understood this is a line. Which line is it? How do you interpret the equation form of the system that's come up? So it's a very different sort of decision tree that takes place when you have the data in front of you. This was maybe one of my favorites here. So this is one from quite recently, just a few weeks ago. So I'm going to show you a comparative data between two, same question, same preparation, or at least same assigned preparation, level of preparation, don't know about that, between the two sections I'm teaching. So they had a web-work quiz to do prior to class that gave them one of the questions on it was a 2x2 matrix A and asked them to compute A cubed. It was like 1, 0, 2, 4, or something like that. And a lot of students were coming back saying 1, 0, 8, 6, 4, and it's not right. What happened? Like, yeah, what happened? So I decided to ask this question is to see if they've gotten it between the night before and the day of. I didn't plan on asking this but let's just throw this out there. So A is a square matrix and a compute A to the fourth or is each of the entries A to the fourth power true or false? Here's the first section and here's the second section. Big difference. Now this is the same group, not the same group, essentially the same size class, different times in the day I guess but why should that matter, right? 91% is getting, of one section is going to arrive in only 83% of the other. What's going on with section one? That's my question. Here's what I discovered about section one. They were the ones who were taking the initiative to get started on the web-work early. And because they started on the web-work early and so they never took the time to get it right. I think the section two people had a little bit more time to think about why this was not coming out right because I think they were jumping on that web-work question so quickly they were getting it wrong in greater numbers. So it's interesting that their personal initiative is sort of working against and I figured that out to the students they think that's maybe why this isn't. I think so. This one is really good too. This is a stock linear algebra question on matrix operations. This is actually following some lecture that took place on this. When I stand up there and look at people writing on it and say matrix multiplication is not commutative. I wave my arms and everything. Then I ask this question. Here's the first section. Good students, right? Here's the second section. Okay. What's going on there? The class is the same way and you just can't do it. At this point, I'm thinking, I don't need to say another word about commutativity and matrix multiplication in this group of 30 students. About a quarter of these students I need to know what's going on here and what turned out as long as I didn't do the reading. And this has been an ongoing issue with that particular section of a low completion of the reading questions and that's where it came from. So you get down, you drill down you can find some things out. Let's see. Do I want this one? So here's another question. I'm going to do the same bunch here. This is a select all the following and apply question. So if you have a matrix that's invertible and D is the inverse of A, which the following is true. And I'm pointing to that third foil there because I'm trying to see if they moved on in the A to the fourth question. Right? Because I'm thinking like, well, A inverse means you just flip everything over. That's got to be how it works. So you should answer C and D both correct anyway. Here's the first question. So that's good. Everybody's going for the right stuff and hardly anybody's going for the wrong stuff. Here's the second one. Okay. So again and so that's interesting because that section two although it was a section that got almost all the A to the fourth question right, but not this one. So what do we have to do here? Go ahead. So on the left where it says the one gets it now is that one one of the C or D selections? No, it's one of these. Actually, I don't know. I think that's not tied to your right answer or wrong answer. Yeah, that's okay. I just want to remind you. I'm not sure. It just showed up. So what I had to do with this section here was just like, okay. So those of you who are saying who are not voting for that question, why? Why is it not the case that the entries of D are the reciprocal of the entries of A? And so somebody says, well, look at this two by two matrix. I just want to see D brief on that. I just want the students to handle it. So those of you who are the 84% of you who are not voting for that incorrect answer, which at this point they can see it's incorrect. We've got to understand why. So tell me why. What do you got? And so that's what they gave me. And then everybody seemed to be okay with that. Finally, here's a question that was really fun. Every non-zero matrix has an inverse. Because non-zero numbers have inverse of non-zero matrices. This is going to show the interesting effect of the round two voting. So this was kind of disinserting around one. First of all that in my section that normally gets a lot of stuff right, only about two-thirds of the students were voting correctly on this. But after round two everybody got it right. And I said nothing about rightness or wrongness or inverses or anything. I just sort of said, okay, talk to each other. And round two everybody got it right. And round in section two it was 50-50 and only 75-25 roughly on the second round. In terms of normalized gain. Normalized gain is defined as the ratio between the actual differences between pre and post test and the ideal difference between pre and post test. Like this is a percentage. Everybody who could have changed the right answer did. And only 42% of people who could have changed the right answer in that section did. Okay, so this calls for a little of an agile teaching, right? You have to sort of think on your toes. You can't just come down with a set thing to teach about. These guys, this is the data I'm getting from now. We can move on with this section. This one, stop. And let's probe. Let's see what's going on here. You can begin to see some patterns for this section too. We begin to sort of emerge as well. Not bad students, but they have a lot of more misconceptions along the same lines in section one does. So this is good data. Let me just kind of end here by talking about three things. What's been good? What's been sort of challenging? What am I going to be doing next for the remainder of the semester? First of all, students are really engaged. We don't see very much the Facebook effect. I do am very sensitive to the fact that I have a room full of 30 students with mobile phones up in front of them all the time during class. And I told them at the beginning of the class I consider that to be on me to be more interesting than Facebook. Let's take that as a challenge. In a very few instances of people I had a FTLC person come observe the class and she saw the one person would vote in and quickly do a Facebook check and then come back. But that person wasn't really disengaged. They were just sort of passing the time. They needed the answer and they just wanted to do a quick check. It's not a big deal. It's great to be able to catch and fix these misconceptions almost as they are happening. So I don't have to wait until there's a crash on the exam. They go back and say, whoa, what happened? I can get that before they leave. It allows me to focus on conceptual understanding of class. And in 227 that is the key. The calculations in linear algebra are dead symbol but they are so hard to figure out which one you're supposed to do next. That's the hard part of linear algebra really just about in the class. And I really haven't had any problems with students resisting this method of teaching versus just doing lots and lots of examples. The further on we go into the semester, students want more more examples. I say these are examples. What are these? These are examples. So there we go. I've mentioned agile teaching which is a way I particularly feel very comfortable in using to be able to fix these misconceptions as they happen. What's been challenging here? Attack problems. Wifi dropage. Mackinac has got the strangest Wifi configuration I've seen. You can be in like sitting where Ted is and it's fine. You can be sitting where Gary is and it's not fine. I have no answer to that. That's been a problem. One time the analytic server crashed mid session and we were done. I had to go to the write the letter on a piece of paper method and hold it up in the air. We had to sort of revert back to what he said. Nothing is fixable. This is just ongoing issues. Some of these learning capabilities features are hard to implement while like a team-based assessment didn't go well. I'd like to give another shot looking at it's cool but it didn't work all the one time I tried it. So I just don't think I knew what I was doing. Obviously selling students on the central reports of conceptual understanding a lot of students will come into math classes thinking it's all about calculating stuff and it's not and you have to convince them that's hard. Students didn't do well on their exam on the conceptual questions I'll just say that and I'm not really sure why because they have very similar quicker type questions given as multiple choice items wasn't so great. Finally dealing with Google has been a massive pain on the rear end up there. They do not know how to work with higher education institutions and Google if you're watching this on YouTube I'm talking to you. The reason we don't have 10 nexus families is because they still haven't refunded us the tax for being a tax-exempt institution. So we had to cart out 300 bucks to pay for the taxes on these things and it's been four months they still haven't given it back to us. So talk to Lori. She'll be meaningful about that. What's next? I'm mixing in some more, some much more delivered in vermin classroom approaches where women's very little lecturing role at all. Learning Catholics is taking more of the role of a conversation server. So students have a ton of questions coming into the classroom to do those but if they don't I have learning Catholics questions rounded up and ready to go. We're currently meeting on Eigenvectors and Eigenvectors this is a prime target for visual questions so we're going to be seeing a lot more of those. We're going to try the teammates assessment one more time and we're trying to see where learning Catholics fits into the grand space of GVSU. GVSU is thinking about moving in this bring-your-on-device direction but thinking of a piece of software called Top Hat Monocle which I think you've heard of them before possibly. They sent some emails around and so right now we're kind of thinking that IT is leaning one direction and I'm leaning another direction we're trying to figure out which system to go to but basically everyone at Tech seems to be agreeing that these things are at the end of life at this point. These are not going to be around for much longer. Maybe that's for the best. Okay so thanks for this and I know I don't have a lot, I'm kind of over time a little bit but if you have any quickie questions I'll take a nail chart. Socrative? Do you know about Socrative? I do know about Socrative. Socrative I auditioned now and I was thinking about using this in 202 I didn't care for it. It's very slow very buggy. I didn't like the way it felt. The big thing about it is it's free. It is free and it works on text message mode too if I recall correctly which is kind of a nice thing for when the Wi-Fi drops out but I just didn't care for that. I didn't have a lot of question choices so it's an okay problem but I felt that learning Catholics is better. Yeah. Do you just talk that monocle friendly with latex? They say it is but I never tried it. So I asked them that specifically. They said you just do latex normally but I've never seen exactly how it works. This is sort of a horrible equation that you have to deal with. I like learning Catholics because the text is nice and plain. Yeah no. Oh yes. What was the cost to you to use this? Zero. And start for your answer for you. Okay. That's my question. How do students react to 12 bucks? I asked well these students aren't paying anything. Oh that's right. I remember asking the 202 students about this and they said yeah bring it on we'll pay it. It's worth it. Maybe we'll have to roll it into a horse feed for example. Especially if there was some great free capitalist textbook that they could be given. Yeah. Alright thank you again for your time.