 Felly, rydyn ni'n bod yn fath o'r fath o'r cyfnod i'r fath, sydd yn ganddyn ni'n ei wneud. Fy hoffi'r ddaf yn ddod o'r rhaid i'r ysgrifennu newydd, oedd ychydig yn ddod o'i ddau, a'r bydddoch chi'n ddau sydd yn ei fath o'r cyfnod. Felly, rydyn ni'n ddau'r Gren Macalony a rwy'n meddwl i'r Gromwch Matt Robson i'r Unedig Llyfrgell, a'u ei fod yn ystod i'r byd ar gyfer y wahanol Bring in rich media feedback into the assessment process at Sheffield. So I will give a bit of context and then we're going to dive in with the case study that Matt's been doing from his own practice which is great. So I'm going to try and go fairly quickly through these in fact so much so that I'm going to set a timer on my phone, isn't that something? So I realise this is context that a lot of people are aware of, but just in case people aren't is just to set the agenda. So, this is the classic quote that comes from... I saw Lisa earlier on, from your document, really saying that assessment is so important, but yet still it represents a major source of stress and dissatisfaction from our students. And there's a real sense that we haven't quite got this right yet. We have very high aims about wanting to use assessment as a really good tool for learning rather than just the certification, perhaps student's anxiety may have even become a bit more an increased sense of introduced fees. Then the challenge is that we face when trying to actually change this is that while our institutions may have very strong and noble policies they want us to adhere to, we still find that there is a huge resistance to change practice. I sometimes say to people that if you want to talk to academics about changing assessment practice Mae'r teisio'n stimulws sy'n cael ei wneud bod chi'n ei fod ymlaen o'r cydweudu'r swyddfa. Mae'n rhoi rwy'n cael ei ddatgen, i'r cystafell ar gyfer yr MB2, allai ddisglu'r ymgyrchu? Peithio rydw i'n cael ei ddweud ymgwrdd eu gweithio'r cyffredinol. So mae'r rhywun i wneud cael eu ddefnyddio'r cyflonfdon. Mae'r cyllid, ac mae'r cyllid, fel ymgyrchu'r cyllid, i chi'n cael ei ddweud o'r cyllid. ac yn ei gynnwys i, yn ystyried, os ydych chi'n deall. Felly, mae'n gwybod bod yn gwybod i'r cynhyrch a'r cyfnodau a'r cyfnodau sy'n ei gwybod i'r cyfnodau yn rhan o'r cyfrifiadau arall. Mae'r cyfrifiadau eures hynny yn cael eu cyfrifiadau yn yr oedd y cyfrifiadau er mwyn ar y brosid. Onol ydych chi'n bywyd yn ddiddordeb, oherwydd mae'r gwybod i'r cyfrifiadau, bod y byddwch y cyfrifoedd, rydyn ni'n dechrau'r cyfrifoedd. Yna bod y gallai cyfrifoedd yn ei fod yn ddull, yw ymgyrchu dylunio'r cyfrifoedd rydyn nhw'n iddyn nhw'n edrych. Ieithio, mae'n dweud ei fod yn cyfrifoedd o'r cyfrifoedd, byddai'n cyfrifoedd o'r cyfrifoedd o'r cyfrifoedd o'r cyfrifoedd. Yn ni, mae hwn yn ymdill, mae'n fawr o'n meddwl sy'n mynd i'w meddwl am meddwl mewn meddwl, a we've been trying to promote rich media feedback for a decade and there's been some really sort of foundational projects like the Sounds Good project that Bob Rotherham did in about 2007. And there's quite a lot of consensus that students really do engage well with technology enhanced feedback or rich media feedback. The kinds of things that we get back from student evaluation is that students really love the fact that there's a personal dimension to it. Practitioners, and we'll probably hear a bit more about this in a moment, it's actually logistically possible to give a higher quantity and quality of feedback. People feel they can understand the comments a little bit more and for specific kinds of assessment and we'll look at this in a bit more detail in a moment, it can actually be more suited to the very assessment type that we're doing and I think that's a really important point. So, it's a great old thing but it's also probably fair to say that certainly a place like Sheffield, as a concept or as a practice, it's not exactly flown off the shelves, can we say. So, what are the barriers? Well, in the absence of an integrated system that enables you to do it, typically case studies I've come across, they tend to represent really kind of complicated and fiddly workflows. People having to do things like record some audio with an audio recorder, save it out as an MP3, implement some kind of file in the magnitude system which enables you to map the piece of work against the student. What if it's an anonymous submission? That's a problem, isn't it? How do we then get it back to the students in a safe and secure way? And all of this adds up to something that isn't really in any way a properly scalable solution or indeed sustainable. So, in Sheffield, we made some inroads into this start in a couple of years ago by implementing the Caltura family of products which are, there's many fantastic products in Caltura, one of which is the way that it's integrated within the Blackboard virtual learning environment and that's really kind of made a huge difference. And so what we did when we first implemented Caltura, before we rolled it out, we started a number of pilots with a number of departments. They had different aims but four of our pilots really engaged with the fact that they could use the tools presented to them now to do and provide rich media feedback. So what I'm going to do now is hand over to Matt who's going to talk a little bit more about what you've been doing. Thank you, Graham. So yeah, I'm Matt Robson. I'm a university teacher at the Department of Journalism Studies in Sheffield. Specifically, I teach the very practical parts of TV, filming, editing, producing TV news items. So for me, the assessments that I do, it's really helpful to have this kind of rich media feedback available. In the bad old days when I was doing written feedback, I might say something to a student. I might write them a comment that would say, in your interview, you don't seem to remember the rule of thirds. The headroom on your shot is not particularly brilliant. So maybe you understand those terminologies, that jargon, but if you're the student and you've forgotten those things, which is why you didn't apply them, then what does that feedback actually mean? Whereas if I can put their interview on the screen, I can point to it on the screen, I can very much more easily help them to understand what's going on. But again, I'm trying to describe something with words, which is much better watched. So hopefully this is going to play. No, it didn't. Let's click it on there. There we go. And then when we talk about running Nicola Squires, we see a shot of Nicola. So again, this is probably the right choice of shot, but it does go on for quite a long time. And here, she doesn't quite leave frame before you cut to the interview. So you just needed to let this run on a couple more frames, you know, two or three more would have done it and she would have cleared the frame and that would be a much less jumpy edit. And I think part of the reason you can't do that is because here you've got all of your material on the same video and audio track. So really, you need this to be up here, needs to be up there, you need your sound to be one track lower down. And then if you need to, you can just extend that a little bit. But because everything's on the same tracks, it's not going to let you do that. So that is part of the problem. If you have stuff on the wrong track, you can't use the full capabilities of the editing program. So there that clip, I don't know, 30, 45 seconds long, but I've been able to give some really very detailed feedback and specifically for this student to help them sort of improve what they might be able to do the next time they do some editing. So I think that's very clear. I don't think there's any way that I could write that down and get that across. So I've been doing this for a few years. But as Graham was saying earlier on, it was very, very fiddly trying to use different apps, different ways of doing it. It wasn't working particularly well. So, yeah, couch era when that came along and when Graham told me about that, I really wanted to get involved because it has made my life certainly a whole load easier. Students can submit their work using this system. It goes straight into the blackboard system for submission. I've also been using another part of the system to record these bits of screencast feedback. And then once I've recorded it, I can just slot it straight back into the grade centre so that when the student looks for their mark and looks for their feedback, it's just there on the screen. Again, they just click on one piece of, click on the icon and then they can play that feedback. And that's been really, really helpful. For me as a teacher, the best thing about it is it's made it quicker. It's made it, I'd say, at least five minutes per piece of work faster in terms of integrating that workflow, getting it all done. And five minutes may not sound like much, but if you've got 40 or 50 students for each assessment, then that's as much as half a day that I've got back, which always gets filled up with something else, of course. But there you go. At least I've now got the chance to do those things. I'm less likely to be marking well into the night, which is always good. So, of course, every little case study pilot needs some evaluation. So I've been asking my students what they think about this rich media feedback. And I've been asking them to compare what's on a module where they get some written feedback on one assessment and my rich media feedback on another assessment. And so these are the results. You can see the numbers. That's nice. So students on the whole found video feedback much easier to understand. A big preference for video. A slightly smaller preference for receiving their feedback in VIN video, but still a large majority. Part of this is to do with accessing the videos. I've made these screencasts using a big display. You've seen it on a big display here. Students, of course, are watching it on their mobiles. So some of that fine, fine detail may be a little bit lost there. So it's something that I've got to think about there. And then a big preference for using more video feedback, please. And then some comments that I hope back up what we've been talking about. Video is a great idea. I think it's very good that students actually understand where their grade is coming from because grades are very important to a lot of students these days. And they really understand why everything is coming back to them in the way that it is. And then someone wrote something nice about me. So forgive my vanity for including that one there. But yeah, again, it sort of proves what we've been talking about. Thanks very much for that. So in conclusion, obviously there's other pilots and we know now that we're very confident that this is being a very highly effective and highly valued way of feeding back to our students. Thinking more broadly from I'm going to sort of take a perspective of from a learning technology management sort of central position. And in a way this actually echoes what Melissa was saying on Tuesday morning is that because we've got a process which has been vastly simplified by the type integration between in this case, Calchura and Blackboard. What we're actually seeing is a series of marginal gains and we're actually getting the benefit of a summation of those marginal gains. A little bit like the sorts of things at the British Olympic cycling team which all about five years ago. And as Matt said, these really add up. They add up to provide whole new opportunities. And this is what the people in our other pilot departments who aren't here today have said. So to paraphrase what Melissa was saying, here's a big thing which is actually lots of little things and I think that's really valuable. I think there's one take home from that, from those of us who are in a more learning technology management or provision role, might say. And then of course perhaps harder to answer right here and now are some of the broader questions. So at this moment in time certainly the University of Sheffield rich media feedback might be a bit of a novelty. It's a new thing. If that was the norm, would people still comment on it? But of course if it was the norm, why would they? Because that would be the norm of course. And then again we've got the broader questions about do we really need to assess as much as we do assess. How does all this broader pattern of assessment fit with for example adopting more of a programme level of view, programme level approach to learning and teaching our courses which I think is again a very contemporary issue for many of us. So that's pretty much what we had to say. So thank you very much for listening to us and very keen to take any questions you might have. Thank you very much, Matt and Graham. Are there any questions here in the live audience? There's a wee mate coming to you there. Thanks. Martin Compton University of Greenwich. Question for Matt. Like we were trying to say before, I really favour this. I think it's absolutely brilliant. There's multiple ways of doing it. You've showed us one example. But how do you account for the 12 to 20% that are less than keen and are you going to do anything to perhaps accommodate them or is it going to be, no, this is better for you? Well, I'm not in a position to make anybody do anything. I can only show people what I've done and say I've found it easier and maybe you want to have a go as well. Do you mean your students? All right, sorry, okay, not staff. So I don't know about that. I mean, I don't think they don't like it but I think they prefer written stuff because obviously we all have different learning preferences. For me, I still believe in its value. I'm going to carry on doing it. That may be a bit mean of me but maybe I need to think about that as well in terms of allowing another way of doing it for them. But for me it works and hopefully they will eventually come round to enjoying that as well. Was there another question from the floor there? No. So I've had a few in from Vvox so I'll take the most popular ones first. So why do you think that audio-visual feedback is not as prevalent as traditional and more time-intensive methods? Well, I wonder whether it's because whether we've been very fortunate to have access to the tools that enable us to be able to roll this out now in a sustainable way. As I said before, I think when you read some of the early literature or if you've ever been to conferences with people talking about this, some of the logistics behind doing this is quite frankly a little bit onerous and has probably been a bit of a barrier to people. So I do think that is a real issue. And I think because of that it hasn't scaled up, because of that it hasn't become a normality for people to do. And maybe that will change. I think certainly it's been very well received when we promoted it internally in Sheffield, colleagues like Matt and from our other departments as well. That's architecture and law and education, sorry, economics have been doing it. So I just think it's because it's still fairly a new thing to actually be able to do in a reliable way. I don't know if that answers the question. That's certainly what we've found. Another question there. Have you tried sharing the students' media feedback with other students to give them a wider range of resources to look at? And if so, how did it go? No, I think it's the simple answer. And I think there's difficulty with anonymity and personal data and all of that. I mean, if I start sharing someone's feedback then I'd have to be very careful in terms of anonymising it and adapting it. Probably simply put that's quite a lot of work, which I may not have time for, but it's a nice idea in terms of a way of learning. But I think probably I do learn things about the common mistakes and maybe can put forward things the next time I see those students about how the common faults, if you want, are the common successes. Certainly the way the technology is configured, that would be in affordance. So the beauty of having it tied into the same ecosystem into which the students are submitting their work, it means we've actually got the work and all maths recordings and or any other rich media feedback doing that could be extracted and repurposed in that kind of way if we felt that was appropriate. I suppose one question is, would you do that with other written feedback? So another question here about grade mark, because it has the option for audio feedback. So do you think that would be an option to overcome the challenges or why was that not? Yeah, I'm not at the worst. From what I know about grade mark it has up to historically imposed a certain limit on the amount of audio that you can record and it is only audio. I think what maths really clearly demonstrated is when we're doing submissions that may be visual or in maths case also quite technical because they're diving right down into the process of doing some editing. Audio, whilst it's great, certainly wouldn't give that degree of resolution, would it? But no, we would certainly acknowledge the fact that grade mark does provide that but the limitation I think that some people found with the limit on audio recording has been a bit of a barrier. The other thing about grade mark is not everyone in our institution for example uses Turniton all the time. Quite a lot of people use traditional blackboards because we are a blackboard house. We use the out-of-the-box blackboard submission point as well so again that's another reason. But grade mark has been a game changer in its own way because grade mark has an iPad app and in the broader campaign to get colleagues over into electronic management of assessment that's been a really important tool as well. So folks I think we'll maybe stop there because I know there's only a short break now between this session and the next one so apologies if we didn't get to your question. Hopefully you can maybe get a chat. Idina's work with learning technologies helps to develop skilled data literate students who can change our world for the better. Teachers and students can develop and share coding skills with multiple our Jupiter Notebook servers. Our DigiMap services deliver high quality mapping data for all stages of education. Future developments include a text and data mining service, working with satellite data and machine learning and smart campus technology.