 So, as we get started, it's right here at 11.30, I'll explain who I am, but I would love for you guys to post in the chat your name, where you teach, what you teach, and what your favorite ways are currently to assess your students' learning. I am Charlotte Duncan. I am the AI program architect here at the North Carolina School of Science and Math. I have been an educator since 2006. Prior to that I was a computer programmer for and I have a degree in learning and teaching from the Harvard Graduate School of Education. I've worked in K to 12 settings, both formal and informal. And right now I develop curriculum, resources and professional development for teachers, as well as teaching just a small bit of my time throughout the state through distance education through interactive video conference. So I have taught in this format since 2018 to a little over two years. I've been teaching online as well as lots of other teaching experience. So that's me. If there's a question, please, you can just unmute yourself and ask partway through. It's not necessarily a lecture. We can have a conversation. You can also post in the chat and I'd be happy to check that as we go. So I wanted to start with this idea of the learning pyramid. Have you guys seen this before? This idea of the way students learn best is not necessarily by lecture. And we tend to default to lecture just like I'm doing right now in this webinar, which is not the best way to learn. So that's why I would like you to discuss. You'll notice it as we bring it to the discussion level, 50%. You'll get a lot more out of this conversation. I'm not going to make you do that because you're not being assessed on this and you probably have some intrinsic motivation that's coming to play. So if I just talk at you and show you some things, they'll probably stick a little bit better than than a student's maybe extrinsic motivation. But you'll, you'll get more out of this if you take some things away and then practice by doing or teach others about what you learn here. So starting with that idea that learning is generative, that students learn best by creating content, discussing it out loud, making things. The first thing I would say about online or distance education that it's very easy to back away from the student and focus on the content. And I think that's a mistake. I think the connection to students and acknowledging their unique situations and getting to know what they're passionate about. Encouraging them to be a part of the classroom environment is really key to successful online learning. So that's my plug for tomorrow's webinar, which is at the same time 1130. And it's about that first couple of weeks that where you're getting started with online learning and how to create a classroom culture that actually feels really collaborative. One thing that I like to do in my classes is if you can turn your web and your web cam on, and we have more of a group experience and so I would generally ask that for my students. So if you're able to Megan, Lacey, we'd love to see you while we're here. Any questions so far about the learning pyramid. So I want to talk about why we do assessment. We do that for two reasons. The first is that we want to check the students progress. We want to see if they're moving toward the learning objectives that we've given. And then the second reason that we assess is to ensure that they've mastered the content that we expect them to have have gained at that time that we're offering the assessment. Neither of these things are the learning process, unless you design your assessments very carefully. So oftentimes we're creating assignments or other work, and we are then assessing. But the learning process is what happens within the student. And so the more that we can give them opportunities to practice what they're learning, the better those assessments will come out for you guys know about formative assessment. And do you use it. Okay, would you mind sharing how you use formative assessment. So my students spend a lot of time in every class doing, as opposed to listening to me talk. And one of the ways that I do that formative assessment is they have to pick somebody in their group that's going to be the speaker for their group. And then they move to another group. So the speaker gets up and moves. And what I'm able to do then is to see how that transfers from one group to the next, because I can arrange them in my classroom that given a teacher eyes and teacher ears from years of I can keep track of what each of the groups is sending to the next group. And then when they go back, I know, what is it that they didn't get. What's the next problem that they need to be facing in order to pick up the part that they didn't get. So I have a long collection of problems that go with each of the various. I'm speaking from a math point of view. That go with each of the various pieces that I'm going to need them to get out of a set of work that they're doing, and that helps me choose what's next. What's the next problem that I'm going to do with them. Great. So that doesn't translate so well to online, right? Like maybe that's your concern. Do you have the opportunity to meet like this with a Google Hangout or a Zoom room? Probably. So I'm in the western North Carolina mountains and so bandwidth access, even sell reception for phones is not close to guaranteed anywhere. And I may drop out of this meeting in and off off and on because of the internet that is available in our region. So I'm in Macon County, but I have students from as far away as Cherokee and Clay and Graham and Wake County. I mean, I've got them from all over the state. So I'm a little concerned that the classroom culture already have established on assessment isn't necessarily going to work the same way. So I'm hoping to hear a lot of new ideas. Sure. Okay, so I do teach remotely just like this via Zoom. I have students in Jackson County, Transylvania County. I just did a webinar in Henderson County. So I do understand there are especially some of those kids that don't have great access. One thing you can do to continue a conversation is to just have everybody stop their video, which really slows down the amount of bandwidth needed and allows them to keep talking. So you can put them in breakout rooms in Zoom through like I you have that option. And usually my rural kids can still have a conversation even if they can't see one another. So the other thing that I do, much like you is I have these, I'll just show you what I use. This is my, sorry, I have a whole window of things but like I have this space in my LMS I don't know which LMS you use to use canvas. Sorry, that was slow. We have blackboard. Okay, so I just keep a bunch of handy links at the top of my modules and what I have here are these shareboards and these work like your your small groups. These are permanent Google Docs that just link. If you click on shareboard one, you'll see that students have used this and it's, I haven't cleaned it up. So we were talking about the definition of affective computing, and there were groups working in boards one, two and three. They were they they're labeling what they're doing and I'm popping into these and an offering feedback as they write them and then they're discussing like here's some examples of the things that we're doing and you can move students from board one to board two to board three. You have that same kind of like talk about the thing that you're doing now because you're doing math that the doc might not be the right choice for you, but you can use either Lucid chart, which has more of a like a flow feel to it, or just some sort of online whiteboard that might be more conducive to doing some math problems to it. Does that help at all about that that first bit. Well, the real thing is, so I'm teaching pre calc which is totally different than teaching number sense for elementary teachers. The, the entire goal is what you're doing has a very different kind of feel to it. And so for one group doing the chairboards is great for the other group the whiteboards are going to be critical in terms of what's going on so yeah those those are certainly ideas of things that I've thought about and it's kind of nice to hear somebody who knows what they're doing in this area, talk about the same things. So one of the options that you have right here in canvas is a whiteboard, I mean in canvas in zoom is a whiteboard so if you have access to zoom. Sean maybe you have a question. Yeah, I just want to, to ask you a question. You showed like the group, a group board 123. And I can see if you if you want to see group once you cannot see group to a group three at the same time right. If you, if you make that windows, different windows right. I'm wondering whether, whether a padlet is is a substitute for those. And I do, I do use padlet to, and I was going to show you an example a little later but I'll show that to in. So, sorry, it moved you guys right where I'm trying to click. So I do use padlet also use it sounds like in your in your chemistry class. Yeah, yeah, I use it yeah. So here's an example of a padlet that I created. Have you guys do any of the rest of us padlet. Okay, so you're already familiar like. Yeah, big thing with padlet for me is that they, there has to be requirements for what they post. So here's an example of just a conversation that students were having that was great. But they also built a collaborative timeline of this is an AI course so we were talking about AI and they had to. In the rubric for this assignment, they had to post a picture, they had to have a heading that was meaningful with that included the year, they had to keep it in the right part of the timeline and it had to have a description with credits at the bottom. So it was just, you know, a collaborative they had a certain hadn't they start each had a certain number of topics that we built out in a Google spreadsheet. First, and then they went and researched those further to build this, which I thought came out pretty well like really cool examples of things. Sounds like you're already on know about padlet you guys are great already, you're all doing all doing great things. Megan I'm sorry I didn't catch what you teach. Oh physics, physics great. Do you have any special tools that you use online for for some of physics that you do. I know because we're not where I have 10 laptops for 30 kids. Okay, when we're in session. So I, I mainly do quit like many quizzes throughout as my formative assessment that they can retake. I don't get better as they go through but I don't have any digital resources right now. Okay, so there's some pretty good simulations at net logo, net logo web. And they are free and they don't require any downloads so if your students are able to access through a Chromebook they'll still be able to see some of those physics simulations which may be helpful and applicable to both chemistry and and physics. So back to formative assessment we're going to check student progress. I want to show you how I do that in canvas with exit tickets. It sounds like some of you use some sort of version of exit tickets you're not unfamiliar with them. But I so I just want to show the way that I do it so that I can aggregate the information online very quickly. So this is the topic so I would give like I mean they're they're designing their topic. So this is one thing that I do in online assessment so that they can't cheat because cheating is a real problem for online learning. This is a junior at a local UNC system school and she demonstrate she shows me group chats of how people cheat and students are way smarter than we are about being able to answer questions do quizzes. Any software you use that you're trying to, you know do a multiple choice question or something that you're giving all students the same answers. They'll take pictures on their phone. They'll call each other. They'll sit next to each other. She has there are whole chat boards where they share answers. So your your ability to simply give a quiz that's ABCD or that has the same mathematical answers is no longer. You just simply cannot trust that they are going to be honest. It's not realistic they will use social media to share. So I ask them to customize their work for their individually. So for example a good chemistry example of this is like osmosis is something you could teach about and so they could apply the osmotic process. I'm not a chemistry person. But like to how carne asada is made or how does osmosis work in the desert plant or some way that they're going to show their learning in their own context so that they can't get around the being able to do what Fred is doing because they have a different question. So I ask them about their own topic. What clarifications they need these are these are my exit tickets that I give. And then reflecting on the discussion for today this is how I take attendance online to where I ask a question where they think about what happened in class and then reflect on it. And then this is the part that I wanted to show you right here is after the class. This is a multiple choice question that they get credit just for answering and it helps me gauge the level of understanding in the course. So after they've answered I can see a bar graph of if I'm on target and anything where like they're feeling good or they're beginning to understand but they still feel good is really helpful for me. When I see even more than one. I need help and feel overwhelmed. I know that we have some content to cover again. And for those kids that are saying this was too easy that's a place where I can give additional support and enrichment. So this bar graph is super quick. And do you have you made these and can't do you guys use an LMS that can do these types of surveys. Okay, so I'd like to show you how I create these because I think it's important to know like how to formulate it so I go over here. Oh, I didn't think that was going to do that. Let me make this window a little bit bigger so I can get my menu. There we go. And I go down to quizzes. And this exit ticket all my exit tickets are a certain type of quiz. They aren't moved. Sorry about that. It's this quiz type right here that's called a graded survey. And so when a student does a graded survey. I give them like four points or something if there's four questions I give them credit for doing each question, but they automatically get 100% grade. Once they've submitted the survey. So you're not grading this thing. You're just able to tell if they've completed it or not and then you can read the responses and reply appropriately. The other thing I do if you're using canvas is I turn off that they can see their quiz responses, and I keep all of these other things off. So I just keep all of this blank I assign it. I make it the date the due date of the class. So, you know, if this is our class date I would make it due by the end of the day today, but then I leave this until button until, you know, like the next class day or two more class days, depending on whether students are having some technical challenges and getting online or I know some of my students have challenges with sharing laptops with siblings or needing to care for siblings that are out of school. So giving them some leeway like making an expectation that it's due today but giving them several days to complete it. This is how I take attendance in this asynchronous way is like, if you have completed the exit ticket and I can tell the questions that you've answered here that this is where we would create our quiz. Then I know that you were here attending class today. In Canvas, if you're often using the same questions, like that question I showed you about feeling good or whatever you can look for your own questions. So once you've created the question you can find it here. And you can just look like I have an exit ticket how are you feeling about this today's lesson, or what questions you have I use these all the time. And so I just add them to the quiz and then they show up here and the multiple choices already built in. So, if you create this once, then you can use them over and over and for a system administrator to create these for your whole organization and you can have every teacher use them and implement them. You can see like there's unfiled questions which are mine and then there's course question banks, which can be created for the whole organization right here. So that's really nice to give teachers a tool if you're serving multiple teachers right in that question bank space. Any questions about graded surveys and exit tickets. Go ahead, Jimmy. I didn't use the service that's that's just really for service like that. Have you had chemistry before and have you have you taken AP chemistry exam if you have taken that what's your score, something like that because we just use it really as just a survey survey not some sort of assessment. One question I have is if a student leave a question blank, say I have five questions they answered four. And will they get five points or four points. I can go back and if they didn't answer I reduce their grade. And the way I do that is in my syllabus for online learning. I have a section here called collaborative participation, and that's where those exit tickets go but it's also where any formative is submitted. So any daily classroom work. I don't give them credit for it because they're still learning except that they've completed it to the best of their ability. And then they're only really graded on their unit projects final exam and final project for a grade. Because I really want formative assessment to be formative. And I just want to make sure they're participating. It's funny what they'll do for four points. So if so because you said if they as long as they submitted they get say five points if you set it five. That was just wondering if they only just did one thing and randomly and then as long as they hit the submission and they will get five or the system will give like every question say one point. I'm not sure I've never seen I only use the survey and I didn't care about the grade. So I wanted to show you I don't think I can show you without showing student work, and I am not able to do that for privacy. But what I can do is if they answered only three or five questions or they didn't, you know they just didn't write a sentence they just wrote like one word instead of X X X. Yeah, if they do that then I just take off points and there's a little bar at the bottom where it's called fudge points minus the points and and you can even just say like you're required to fill out all five you get zero and and they'll learn very quickly like you actually have to answer the questions. But really they, once you come to class the next time, talking about the things that they reflected, they know that those exit tickets actually matter to the classroom. In the same way that we're talking right now and it's different than if I was just lecturing at you probably about stuff you didn't want to know about. When you bring the points that they bring up or if you say you know I saw that we didn't really have good clarity about natural interaction last time we met. Let's go back over that again they know that it's worth their time to do well on the exit ticket and and give feedback because they know it's going to be benefiting their own learning. So, yes, you will have occasionally a student that'll skip a question you just fudge point it off. You can change it. So how hard is that to integrate within a zoom class like and so I'm thinking that your exit ticket of how things are going would be really effective at various points in a math class. I haven't gotten part a going to part B is probably not a good plan at this point and I'm assuming so I know this is within canvas and that's fun the kids are good at working in blackboard and we have a similar capability. How hard is it to get them to actually move to that and come back. So I only do them and you could change this but I only do them at the end of class. And it is the way that they show attendance and participation in class. So I do have students that have technical challenges. I taught through the last hurricane where I had seven weeks of students who were not able to come to school and they had various situations one was would go to grandma's house and there were like a three year old in the background and grandma tottering by and another would have to go to McDonald's in order to connect and so it wasn't an everyday thing. And so the exit ticket was the way that they said hey I've I've watched the course video or I've done the work that you have assigned and I've you know collaborated with my peers on some document and and here's what I here's how I've reflected so it was the way that they checked in as being present. So but I only did it at the end of class I've never tried it in the middle I think for me that would be kind of overwhelming in the middle of the day. And I hope that help. Well yeah I mean have you used within zoom have you used even like the thumbs up and that kind of stuff to give a similar to that last one I can't imagine doing the full exit ticket you had. It's much more the sense of doing portions of that keep track because I do that every day in class where I'm doing that check even when they're sitting in those wonderful long rows that colleges don't seem to understand is not the best way to teach kids but I can do that as I go and I'm just trying to figure out ways that I can maintain that. Yeah I actually do I do you know thumbs up thumbs down and I do this middle thumb to quite a quite a bit. And if you don't have video they can do that through the chat as well they have like little icons. The students will find them quicker than you will. It's reactions so there's a little smile with a plus and there are like claps and thumbs ups and there's one right. So I can I can hopefully take that off. I don't know how do I turn it off for myself. I guess there it is. Okay. Yeah there you go. So those are helpful. I do think it's important to have students speak as much as possible so if you have a very large cohort. You may just be checking in per group, taking the temperature, you know one to five kind of thing. So that's formative assessment and we're checking their understanding and their engagement with the material using those exit tickets I try not to make them take more than five, maybe 10 minutes. But then summative assessment is at the end of the learning processes which is what we normally think of as assessment, but it's not the process itself it's that how we give credit for the completion and mastery of the material. So exit tickets are for attendance and questions and discussion. And that's that collaborative participation but the assessment that we're going to talk about next is that ensuring the students have completed the learning objective. And I want to take a quick moment and talk to you about book that I left across the room. It's called Science in the City. And it's a recent release. And it talks about how we reach urban and and other under resourced students who may not have the same rich vocabulary and culture that that we as educated people have been have had had had access to. And I really love what they talk about in terms of disaggregate instruction. So this is where with you when you have a difficult concept like a function or a chemical process that you first teach the idea. Without any technical vocabulary at all in a in a way that a student can convey their own knowledge in the vocabulary and context that they speak in. I'm going to show you a quick example of this. This is one of my exit tickets. So this was still a formative assessment. But I was I was asking them to describe what we were learning about natural interactions and how artificial intelligence systems try to interact very naturally with people. I wanted them to share their own experiences once they've learned that concept. And then I had them give a definition of a term that we had just learned in their own words. These are all disaggregate so they're they're not trying to be technical. They're just giving examples and this is something that we did in class about facial expressions and the movements that they make and what what emotions that might be expressed. So these are and then I asked them and you know what's interesting to them about this unit so nothing technical here, even though the next concept the next day will be technical. So I've disaggregated the content from the vocabulary and the technical piece. So I'm going to check their understanding of the base concepts before I give them the vocabulary. So the next day that we would do this I would ask them to do a similar type of assignment only using the rich vocabulary and more technical maybe mathematical concepts that we talked about in a more technical context the first time, and then most importantly in the third application of this disaggregate methodology, you have to have them use the vocabulary over and over and over. So they would do it an exit ticket they would talk about it with their peers, they would do an online discussion, and they would have to explain and then make arguments and then extend their learning into a new field so if they were first talking about it at home now they're talking about it in a public health setting, something like that. And then, because this conversation is so important and relevant to the learning process in that disaggregate. I use an online discussion forum, which is built into most LMS is. So this is a graded discussion. And I'm very clear in the description of like, you're going to explain this happens to be a project but explaining. You've already read your, you've already done your reading you've already written your your reflective essay, you've already started building with scratch this is our very first coding assignment, and then they describe their plan. They talk about who their partners are. And then they write back and forth to one another using the outline that I've given them. So, this is an, do you guys do online discussion yet. So even a math class a chemistry class they should be able to describe in words, talking to one another, or maybe asking questions with one another in this format that where you've outlined very clearly what they should talk about. So look, I'm, you know, I'm going to make a darts game or a Ferris wheel. Oh, have you thought about a merry go round or a popcorn machine, because they're building games in this, in this example. I do have a rubric. I'm going to share it just one second. Great question. Thank you for bringing up rubrics. Let me see if I can find it here. Okay, yes. So the one that I had an example for Timothy S for rubric is for a project. Because I use a lot of projects in my class, as you saw. So this is just one project checkpoint rubric so a project checkpoint for me. I've disaggregated the creating of the project with the final result of the project. And so this is a project checkpoint that students have to use as they're working on their final project. In order to show that they're moving forward, and their group is functioning in a healthy way so they're listing goals so in this class they have to use the agile methodology and they like burn down a task list it's very it's very like, I don't know techie. So that they know what this means, like clearly defined goals are like small bits that anyone from their group can pick up and do. They're explaining how they're testing with users they're they're thinking about their future goals. They're posting to their project log which is a blog. It's a public blog that they're explaining how they're where they're getting stuck and they have to post to that. They're describing how they've used their homework time and they, they're showing a summary so this is just a rubric that I used for that I do have. Let me see if I can find the rubric for for discussions. I have to look for it, I will find it for you Timothy and if I can't do it right now I'll send it along. Any other questions. So far about disaggregation. About the discussion I have used it for my, so prevent discussion mainly for safety concerns because this is chemistry class and we sort of really emphasize the material safety data sheet. And so, in order to, to encourage participation in the face to face class we created a discussion type, so every because that says everyone you have to post first, and then you can see someone else work. So this is some sort of sharing. They are allowed to sort of copy paste someone else's work. This post a fake example. Oh, either the major concern with HCL is you cannot avoid inhalation something like that. So you need to be to stay away in the distance. So eventually they, they, if they themselves did not have this kind of thought in their own mind and after the discussion they learned it so they can copy it to their place. My question is some student asks them because the discussion function it has something like they can talk. Have you ever experienced like just let them talk or what's the value of talking I can see in a face to face class, and it's not, they have already talked to this they talk this and at the same time they type. So with the remote discussion, do you think talking might be better than just typing. I think. Yes, a lot of times we just talk in class, and I do have them reflect on their talking. So I was trying to pull up a an example of that. Let me see if I can find one. It is. Yes, so this is an example from when they did a presentation. Let me just, I'm moving my windows around a little bit here. So this is where they are reflecting on a presentation but they could you could do the same thing for a discussion this is a graded assignment, which I'll show you the rubric for in a minute but so like when you were talking what would you have changed or included to improve your presentation. Focus on your content rather than your delivery so it's not really important if you did a great job as a public speaker because this isn't a public speaking class I'm really much more interested in what did you realize you left out. Let them talk about their strengths, and then I asked them to give themselves an overall letter grade based on the requirements of the rubric. And then have them explain their answer. It's really interesting to see what students say for this particular question and you could do the same thing for a discussion. One of the questions or one of the norms that I start with in my class I call it one over N. So if there are three people in the in the room, then you should be talking one third of the time. And that means if you're a little bit quieter that we set the expectation that you need to be included because all voices matter. And if you're a little bit more likely to jump in then then you may need to rain it back so that you can make sure that everybody has enough talking time. And just setting that norm and then enforcing it like reminding people before a group discussion starts. I sometimes talk privately via chat to students who are more likely to keep talking and ask them to listen and amplify a quieter voice. So I would say to a Steven, you know, hey Steven. I know that you're a leader in this class and you have a lot to offer. And I really need you to be a leader in this space and make sure that Peter's voice is heard because he's often a little less likely to speak up and it means a lot when you say Hey Peter, what do you think about this or Peter you've really offered some great stuff in class and I'd love to hear what you have to say about this and and so when you take that strength of that speaker and you turn it toward hearing other voices. It sort of becomes not an issue, as long as you set that norm. And then when you ask them to reflect in their exit ticket about their group participation. So this is a, I do a lot of these assessments of students thinking about other students work but what would you have asked about other students work what were the strengths of their presentation and then I have them send all their constructive criticism to me so that I can filter it, and share what might really be helpful and shareable. So they are required to turn this in for three students they don't have to evaluate every presentation that'd be far too much content, but they evaluate themselves and three students that they've been assigned throughout the presentations. And that allows them to even go back because you're recording the class. They can go back and maybe fill in those questions afterward to, but you can certainly modify what I was just showing you for a class discussion to say, were you amplifying quiet voices, did you were you were to speak up. And I asked my students at the start of class like, are you more comfortable less comfortable speaking out loud and, and then I use their responses to tailor how I speak with them in class. Questions so far. Good. I'm going to see I'm still going to try to find that rubric that I have. It's hard for me to do it while I'm talking to you guys but if I don't get to it I'll send it afterward. So disaggregating instruction is where we were that you've taught a concept then you've added the vocabulary and then you've given lots of time to talk about and speak and explain and make arguments. And then you're grading a conversation related to the topic. And you're using collaborative documents so that's the Google Docs or your whiteboard, and you can have them screenshot to show what's happened. I recommend that you, if you don't have something like Google Docs that shows a history that you asked them to set a timer and you can take a picture every five minutes and see how evolution is happening. Another resource is Miro. If you're doing any sort of story boarding. It is like post it notes, like if you use post it notes in your classroom and you're like, having everybody do stickies murals does the same thing and you can actually take pictures, actual pictures of, of post it notes on the wall, and they become virtual. So it's pretty cool tool. Let's see if I can pull it up for you visual collaboration software is what it's called. Miro. So if you were looking for that like whiteboard or, or some way to collaborate, you might like this tool. I also use. Oh, oh dear. Prezi I use Prezi on occasion too but I like this just because you can use it a bunch of different types of frameworks, and you can move things around super easily so here's brainstorming. You get here's a bunch of stickies, and you're just, you know, you're able to move them around just like you would in a classroom so if you're anybody's sticky fan, like your, your post it notes great. So this thing is exactly the same. And it, it's just really easy students can access it I think it's free for a while so you don't, you don't have to worry like I don't we're not paying. And I use it. So, I don't think you'll have any trouble with that. Okay, question so far. All good. All right, I want to talk about the problem of cheating. I have specific concerns about cheating in your classrooms. Yeah. Okay. Unsurprising. I have had lots of incidences. I'm pretty. I mean, I really don't think that you can use an online quiz application, like we've been seeing for those graded surveys, and have it be unmoderated and have the student learning be shown and assessed correctly. So if, if you were in a space like an online school where there was a facilitator that could watch students. I wouldn't be concerned but if your students are all at home, you basically don't have a moderator for them. And so, I hate to say this to you but I really don't think that you should use graded tests that are multiple choice, or some simple answer. The way that you have to show that makes it cheap proof is that your topic is unique and customized to your class so that they can't go download anything. And that you give separate questions and people in your class different work. Same types of problems, different problem sets. If you're giving a paper that there's a, you know, topic, yeah, project based we're going to get to real next. Yes, for sure. But you need to look on whenever you create an assessment, you need to Google your assessment questions and see how quickly that you can find the answers, because if it's one second, your student is not going to put in the time. I only use projects and presentations in my class for for summative assessment, because I feel like it's the only way that I can ensure that students are doing the work. If you have to have some sort of, you know, you given each student an individual problem set, make it at least harder for them and make them write it down and submit a written copy in pencil. I know that sounds crazy, but I teach a computer science series of courses and my written my tests that I give very few tests but when I want to know that they understand content I can do essay tests that are written in pencil with a moderator, because I could not find a way to ensure that those simple yes no a b, you know, show me what if it fell statement is I couldn't find a way for them to not find a way to cheat. So just take that like straight up you you just have to throw that out the window you you can't control that classroom environment. You're basically letting them go home and do whatever they want. Sorry to be such a dammer. So I do use projects, as I showed the project checkpoint, and that I have them do a plan that's separate from the execution that's graded so that their plan is a separate step. Then I give a very clear rubric about the project that's being assigned which I'll show in just a minute. That as much as possible those projects have a real world application. So I want to show you an example of a computing course. This is about a three minute video. It's a computing course that I developed so that students had a real life application, a real life application for why they were making computer science projects. It didn't feel right to me that we would just like tell them to go learn some coding like for nothing. So this was my attempt to make it really relevant and the more you can do this for your students better. So, here, here's the video. Have you ever considered what it must be like to see the world through the eyes of the American River Otter? Or why the bee is always so busy? Or even what it must be like to live in fur? STEM scholars in the North Carolina School of Science and Maths interactive video conferencing program created an opportunity for visitors of the world's largest zoo to experience the answers to these questions in New York. One of the best parts about being at the zoo today is to see high schoolers teach elementary school kids about science. And you see these kids light up with the excitement of knowing how a polar bear insulates itself or how a fish can survive climate change. Amazing. Students use computer science and computational thinking to build apps and also create websites and wearable devices that further enhance the public's understanding of creatures in the wild. So our project consists of two apps and a B suit and a helmet and it aims to create empathy for honey bees. We expected 10,000 visitors today at the zoo. We've seen several thousand already, 10 projects. Students from five different schools and they've been able to share their knowledge with the people here at the zoo. We expect to do it again next year. Hope to see it again. The social philosophy underlying this course is the belief that technology's highest purpose is to promote empathy and understanding among its users. So, took the learning objective, which was, let's learn some computer science and thought about a way to make a final project that was really motivating and relevant to the students, and we had a client. So if you can do the same thing with chemistry or math or like, I have a student who is super into baseball and so we've been talking a lot about data science and how the applications of, you know, what his passion is have to do with math. So he's learning a lot more math because it has to do with the synthesis. So wherever you can do that and make that connection. It makes the output of the goal of your class that much more engaging. Now I know y'all are stuck at home, right? You can find competitions that do the same thing, Kaggle competitions or whatever in your subject. I'm sure there are competitions and math Olympia ads and stuff, but if you can connect your learning goals to some deadline, it's super motivating to students. Any questions about that idea of like, getting the project into the real world? So I actually think that's great and I do those kind of projects with my students. Unfortunately, in a course where the goal is to, so pre count for example is a pathway course. It's not enough for them to learn some pieces of it. They really have to be able to show mastery of all of those things or their calc one course and their calc two course and their calc three course are going to be devastating. So while I do include those types of projects, they can't be the basis for them showing me that they have some technical knowledge of the various techniques of things that we're doing. So one of the things that I am working on to do with pre calc the number sense course is entirely different and entirely harder in many ways, but I've never given multiple choice tests that's never nothing I've ever done. And I think that in this particular instance, I may be forced to do some of that. But in addition to that, they're also going to have to take pictures of their work on their problems and put them in a document. And we're going to practice that to where they'll have 10 minutes after the test to put that on Blackboard or their test doesn't count. So they can go out. I mean, obviously they'll have a time to mount a time for that test, but sure they can go out and they can get somebody to give them the answer. But their answer is going to have to have the work supported behind it through some kind of document of their work which is exactly what they would be giving me on any math test. And there are, I don't know about for chemistry, but there are lots of applications for math that will give every kid the same problem with different numbers. And so yeah, they might be able to talk to Susie next door, but Susie is going to have to solve her own problem and their problem in order to get that done. And so there's less likely that they will do as much of that. There's no expectation that they're not going to find ways around it. They find ways around it. But there are some things that are certainly available for mathematics that we already use. We already use a system called WebWorks that is an open source system used and developed by mathematicians from around the world to create those very kinds of problems. So I can give them a timed WebWorks that they have to provide me with the documentation of their work on to help them. But if I were developing an online class, I would be able to do that project or anything to do this starting on Monday. That's probably not going to happen. I've got something faster for you that will address that issue of like, I need, they need to learn 19 concepts. It's required that they know all 19. And so the way you can do that is to gamify it. And so Badger is a way to issue micro credentials. You create your own badges. There is a pricing model. There is a free version right here. So you don't have to pay anything to start. And what you can do is, I don't know if it'll let me do this. My Badger version is in Canvas. This would interface directly with your Web software, whatever you're using. And basically you would make these little badges that there's a whole bunch of pre-programmed ones. But you can literally just say, okay, you have done the four requirements that show me that you know, whatever, to multi polynomial something or other. I don't know. I can't think of any math concepts right now. They're all flying out of my head. But you could show those 19 concepts that they need to know. And then you could award them with these badges. And that's one way to like, okay, you're 50% to that badge goal. You need to complete these three other assignments. It's kind of a cheat. It's a fast cheat. So it at least gives them some milestone to look forward to and shows their forward progress. So we have five minutes left. And I just want to make sure we have enough time to share with one another and any other resources or ideas, suggestions that we could share with one another. Oh, and Flipgrid. Yeah, Flipgrid is great. Go ahead. I used cohort used to be, you know, for the face to face class, it used to just for fun. And because cohort has this music has this jumping something just for, you know, making just to show relax. So we I have used the cohort for like exit ticket or entry ticket. I thought we have a discussion with our teaching teams on Thursday we're talking about pretty much like in mass we cannot quickly develop an appropriate project oriented assessment. So we simply just lowered the weight of the test. And we decided to just sort of like, like what the mass teacher just said, and give you some time is timed and they feel free to open book feel free to look for whatever resource is and then you shoot you shoot the photo of your test sheet and upload it to me like give them like extra five minutes after the test to upload. But we need to talk about maybe we can use cohort for like quiz because we also our syllabus also include weekly quizzes and the limitations cohort is it can you can only do multiple choice, but it's very short time is one question by another question they just simply do not have that much time to do get help from some somewhere else because we can decide like okay this question is only 20 seconds. So 20 seconds is not enough to Google search anything. So we were talking about this is what we are going to try. Because with chemistry we we have some conceptual content we have some calculation mass issues so we were thinking about half of the quiz will be code and say I have like three questions five questions which are conceptual multiple choice. And this can mimic the close book feature. And then after that we say a synchronous quiz is okay you get this problem solving and like like apply gas laws to solve this problem. I'm sort of feel free to cheat within a given time or something like that. And, but at least we will build on this philosophy is when you show me your own hand written test sheet, even if you copied it from someone else that you still your brain, you have to think through it. So there are some scratch in your mind. So we were thinking about with that half closed and half open and feature at least so we can dilute the artificial artificial effect of students cheating with the open book. But we still do not have a good idea with our because chemistry we have a final exam. And we were just hoping we can come back and do the exam. But it looked like that's not, not feasible, we still do not know what we are going to do with that. So currently, we decided to just try this have covered have open book quiz on campus with the 10 the quiz, which is low stake. So, right, that means it doesn't matter too much. But we don't have good idea with test and exams. It's, it's hard to make that shift. Certainly the more you can can connect it. So as we close. Thank you all for being here and for sharing. So thoughtfully, I did want you to know that there's a really good discussion rubric from Central Michigan University. And I'm, I just searched for rubric and I'm having trouble bringing it up in the full version here, but I love that it has like, there's a criteria for the original message. There's a criteria for the follow up, there's an overall confusion. You know, this is insightful, it made a serious effort, it encourages generation. And it's, you know, there's no spelling and grammatical errors that sort of thing so maybe that's helpful to you. I liked this one myself. I could not pull up my own rubric fast enough to to bring it up in class but this quick search did it for me. And for me the rubric is one of the ones that I can download from my school so anybody who's at NC SSM should be able to get to it through the repository of rubrics that we have but I'm not able to do that unless I'm creating an assignment that I couldn't just pull it up from an assignment that I'd already used I tend to customize mine really specifically for the content. Thank you for being here and appreciate all of you and please feel free to reach out if if you need anything more or want to collaborate. I'm charlotte.dungan at NC SSM.edu.