 Hi and we've been on over this morning. We've been to the east coast of America, the west coast of Canada, all around the place. Now, sunny Sunderland. Yes, sorry, I couldn't resist it. No, but Sonya, honestly, I'm very welcome from the University of Sunderland and apologies to anybody who opened the door at the east of England. But anyway, I'm really looking forward to this one as well. And I think, you know, joking aside, I think student feedback is always really important. But walking in these pandemic times, we really need to make sure our assessment is only as good as the feedback. So I'm really looking forward to this. So that's enough of me being nice. Let's get back to counting our scale again. Are we ready, Marin? Yes, we are, Tom. Sonya, are you going to give this a go? Sure. Thank you very much. Good go. Okay. Here we go. Ahem. Ahem. Ado. Ado. Watch, Sonya. A tree. A tree. A car. A car. Akuik. Akuik. Gustav. Yay. Take it away, Sonya. Oh, hello. Thank you very much for that fabulous introduction. And thank you for watching this presentation on how we adapted nominal group technique for student feedback for use with a cohort who had to complete the module online as a result of the pandemic. So obligatory biography slide for you, since you probably don't know me. I've been a learning technologist now for a little over 10 years. And I've been at Sunderland since 2014. I've got my social media and my blog links on there for anyone who wishes to contact me. And anyone who asks me on mastodon wins a Mars bar. In 2018, I took over the module introduction to digital learning and assessment, which is part of our post-grad certificate in academic practice. This is, of course, targeted at our early career academics and module tutors. And I teach other things on an ad hoc basis. So we all want student feedback. We know that we need to know that students are getting what they need. And we need to know how and where we can improve. The university has a number of systems in place for obtaining feedback, including a module feedback questionnaire. And that's fine. The questionnaire is good. I don't dislike it. But it is an automated and centrally controlled system. So it suffers from being a little bit generic. We use NGT because we want to give students the chance to give direct feedback to us and then to have a discussion with them about it. So NGT is a group exercise for obtaining a ranked order of items. First, people work individually to come up with their own answers or ideas. This is the nominal part. The nominal part. That's then collated into a group list and then items are voted on. There's a great quote here on the benefits of NGT for student feedback. In particular, my first experience of this was in 2016 when I was myself a student on the PG cert. And because a photo speaks louder than words, here's one we made earlier. This is from a 2019 cohort and note how the exact same cohort of students have saying that they both enjoyed learning about Canvas and that we spent too much time on Canvas. So we used to do this in a room with pen and paper. But after introducing the NGT process, we would give students three questions along the lines of what worked well, what didn't work so well. And then we would leave the room for 20, 30 minutes so that the students could discuss things privately. And the pandemic hit us last year, just as we were getting rolling with a new cohort. 12 students from our London campus, they had one session and they had four more taught days scheduled for April and May. So we had to find a way to adapt NGT for them. Those teaching sessions, they were taught via our Canvas, our VLE, and a big blue button, the conferencing tool that we had within Canvas. It was, however, poll everywhere that was crucial for enabling nominal group technique to work. And this is the question I drive my students mad with. Why poll everywhere and not something else that does a similar job? Well, I teach the systems as part of the module and I knew that poll everywhere had a question type that would allow free response and voting. Looking back at their answers again now, it's striking how each response has exactly one vote. Technologically, I know this feature works. I've done it in other contexts. So possibly just the activity wasn't explained or understood well enough. It could just be coincidence. For the third question on ideas for improvements, I used the open-ended question type with the results presented as a text wall that allowed students to see the results coming in live and to write freely as much as they wanted. From my point of view, this was a really successful experience. The one big difference is that normally with NGT, people are given a number of points to assign to items. This is how they rank them. But with poll everywhere, it was a much simpler up-vote-down-vote interface. I also asked the students about what they thought of the experience. And here's what they said. Two of them had previously used NGT as students and three had used it in their own teaching. On the crucial question about poll everywhere, all of them said that it was either somewhat or very successful. And here we have confirmation that they still found the feedback exercise useful and that they did feel that their voices were heard as individuals, something that I had been worried about with doing their activity via a big blue button. Finally, we had a free text question for any other comments and feedback. Just a note of caution on the survey size. There were only 12 students on the module and of those 12, only four responded to my survey. So definitely something I would want to repeat with larger cohorts. A shout out and a thank you to my colleague, Dan, who let me bat ideas around with him and we put together the poll everywhere solution. And thanks to my colleagues in London for their eminent enthusiasm.