 Good afternoon, everybody. My name is Chris Thompson. I'm the Programme Leads for Teaching, Learning and Assessment in JISC's advice team, but that's not important right now. Thank you for choosing this session, this afternoon session. It's two sessions of half an hour. In half an hour's time we'll be handing over to Anshela Lowe who's going to be talking to us about Staffordshire University's approach to upskilling, roadmap for upskilling staff. But before that, I'm happy to introduce you to Samantha Stark and Simon Carrey of the University of Leeds, who are going to be talking to us about design frameworks. As with all the previous sessions beforehand, there will be an opportunity for Q&A at the end of each session, so we'll go for 20 minutes or so, have some Q&A and then we'll swap over for a financial session. But without further ado, I'm going to get out of your way and it's all yours. Great. Thank you very much, Chris. Good afternoon everyone. Thanks for coming along to this session. I hope you're going to find it useful. What Samantha and I would like to do today is just give you a bit of an overview of a bit of a project that we've gone through the University of Leeds as an online learning team. We designed and developed fully online master's modules at postgraduate level and this is a piece of work that we've done as a large team and Samantha and I here today to explain a little bit about it. So the first part I'm going to talk about the process we went through to design the framework, a little bit around the challenges why we wanted to do this and how we went through the process and what the output was. And I'm going to hand over to Samantha who's going to tell you a bit about how we implemented that and the evaluation work that Samantha in particular has done around the framework. So, in terms of the design framework itself, we have been developing modules for quite some time and the approach that we've taken has been very much led by the individual learning designer, who has taken what they think is a good design process and have taken academics through that to design and develop the modules, but we've found as we've scaled and we are growing quite a lot and developing more and more programs that we are starting to get a lot of inconsistency in the programs from module to module because different modules have different academics and different learning designers. And that's creating an experience for students which is a bit inconsistent and they find they're finding they're having difficulties with that. From the academic perspective, working with different learning designers gives them different experiences. And those experiences vary a lot. And it's like starting with a blank sheet approach and that's something that they've found quite difficult so we thought perhaps by introducing and creating a framework for how we design the modules themselves that would help to scaffold the process for the academic so that they don't have to start with a blank sheet of paper they've got some sort of roles or framework they can work within. We'll be sharing these slides afterwards so I'm not going to go through everything on them but you can see some of the other challenges we've had along the way. So in order to start this framework we needed to think about what we want the student experience to be. So before going into designing a framework what we did is we came up with the design philosophy. That design philosophy is about how we, what we're creating for students and the process that we want to go through to create them. So you can see we want to create learner centered experience, active learning, compelling learning journeys and also very importantly part of the process of creating master's modules is that the academics have some resources that they can reuse afterwards so that they're really important part of what we, what we do and the approach we want to be collaborative, we want to innovate and we want to use people's time as effectively as possible because, as all of you know, academics can be very, very busy people so we want to make the process as useful for them as possible. So once we had that design philosophy in place, then we went through a design process and that's where I brought together the team of learning designers who brought together their ideas of what they thought a module framework should include some of the pedagogical ideas behind it the theories. We brought those together in a workshop setting, and then we created a framework and I'm going to explain that to you in a moment or two, but just to say once we've created that initial framework, we then reviewed it internally within the digital education service of needs, and then we took it out to the academics and got their input as well. Those are academics who've worked with us before and have experience of online masters, fully online masters programs. And through that feedback we've prototyped and iterated it. So here are some of the, some of the major influences I'm sure you'll be familiar with with a lot of them. Obviously, these are postgraduate levels, so we need to make sure we've got constructive alignment. Students are working towards that final summative assessment. A large conversational framework is something that we've used an awful lot at the University of Leeds, particularly in the early days when we were making MOOCs, but we've seen that come through as a really useful tool for the learning designers but also really useful tool for the academics because it helps them to make sense of the different types of interactions that students can have. Something called curriculum redefine, which is a new university wide initiative at the University of Leeds, which is about bringing digital into as much teaching and learning as possible. So that very much influenced us. And I'm going to skip down to the bottom because one of the major theories that we brought in is around situated learning theory and actually making the learning for students as authentic as possible. Because it's at master's level, most of the students that we have onto the programs are professionals, they have been out of education for some time usually their time poor. So what we want to do is make sure that we're creating a learning experience for them that they can immediately take away and take back to the workplace or to their own context. So that's very much underpinned the framework that we that we have. Just give you a quick overview of context about how the framework works. The modules that we have within programs take place over an eight week period so they are very condensed. That's 150 hours, potentially of learning in eight weeks. The students over each week, they have six core units. So six over six weeks and each unit each week, they do 20 hours of learning. Then in the final week for with the revision assessment, they've got potentially 30 hours to revise and do their final assessment before they have a weeks break and move into the following module. So it's really condensed. And each, as I say, each week is 20 hours. So how we broken that 20 hours down is we've got to 10 hour blocks as it were. This is what we, how we like the academics to think about. We've got the 10 hour block that they work very much with the digital education service to create. And that's around guided tasks. So you're kind of input your lecture type material, completing exercises and activities doing some core reading, watching videos, interactive things like that. There's also the webinars, the synchronous time. There's the synchronous time each week. And there's also the asynchronous time as well. And that's where the bulk of the time spent with digital education service to create resources comes we also have the other 10 hours of learning, which is the more masters level learner guided, they're creating it and putting it in their own context they're doing their own reading, their own research, preparing for assessments. I want to take a moment to focus on the the tutor guided learning that 10 hours. This is where the framework really comes into play. So we've taken that 10 hours and we've broken it down into three kind of core sections. And that's really as much of a framework as it is. We have the first section which is the prepare, which is normally two hours of input. So you might find your lecture type material there. And that goes on to the apply section. This is where I said the scenario based learning comes in. Each unit has a central task or a central activity that students, that students complete. And that has a clear goal at the end students work through that task using the information learning the knowledge that they've acquired in the prepare section. So they move on to the reflection at the end which is typically where they'll find their synchronous time with the academic, and where they can do some reflection and in between that there's time for, thank you, time for their discussion, either before the task or after the task depending what, what makes the most sense. So that's the, that's the framework, we implemented this about a year ago, and I'm going to pass over to Samantha she's going to tell you a bit more about that. So hello everyone implementation and evaluation so we are now using this framework for all of the masters that we're designing but we started off by piloting it on two different master's degree programs. The first was our masters in sustainable business leadership and we piloted it on the business analytics module. And we also piloted it on the international fashion marketing degree on the two models that you can see listed there. I'm going to call these SPL and IFM, these master's degrees because the titles are too long. And then we use them for all the remaining modules on IFM, and then we then used it for all of the modules on our MA in digital design and communication DDC for short, and I'm about to start using it on precision medicine, and my colleagues are using it on a data science degree. But initially, we were just piloting it on these three modules. And so, as part of the pilot, I applied for funding to do research and I got funding from the Leeds Institute of Teaching Excellence or light for short, and they funded me for one year one day a week from September last year. The purpose of the research was to find out is the framework actually working. We know we think it's great, but it might not be achieving what we wanted to achieve. Does it need to be approved improved in any way. And just to get that feedback, basically. I started in September last year so I started with my ethics application, and then data collection was a bit delayed due to module delays. Data analysis started in July so I'm still in coming to the end of data collection, analyzing data and disseminating all at the same time. So the initial findings are not complete but it's what I've found so far from my initial data analysis. I collected data from everybody who is impacted by this basically so I collected data from there stands for the digital education service that's who we work for within the University of Leeds. So I collected data from my colleagues, I collected data from academics, and I also collected data from students. I've been collecting data about from about modules that do use a framework and also that don't use the framework so that I can kind of compare what people's experiences are and see if there's any trends in relation to either framework or non framework modules. I have two focus groups with their staff, I've done one to one interviews with academics so seven so far, and I've done an online anonymous questionnaire with students. Unfortunately, I've only had six responses so far because they are time poor, unfortunately, but at least I do have some insight from them. So I'm going to go through now what my initial findings are and there are positives and negatives from from all sides basically. And then I'll go to Dez staff. You'll see there are some points in bold and those are the consistent themes that appear throughout all the data so there's staff academics and potentially students as well. So that clear structure is clearly a big positive in relation to the framework. And my colleagues have also found that having that framework helps their discussions with academics because it gives them a platform upon which they can start talking and pedagogy because it has there is that framework there, rather than having the kind of blank sheet approach. So that helps in that respect. And it prevents too high a volume of materials being created and that was one of the big reasons that we created the framework in the first place because we were just drowning under the amount of materials we had to deal with. So the application to real life scenarios so learning is more meaningful, and they generally prefer framework to non framework. There's a lot of negatives though. So there's difficulties with with working with academics because often they're looking at the phases to prepare the apply the reflect as three distinct phases that are isolated, rather than culminating in one learning experience. So there are issues about you know how the academics want to approach we're saying they have to do the apply section first because the prepare section can't be designed until you know what the apply sections going to be. But some of them just want to start at the beginning and go through it chronologically which does create challenges. I think note in bold is good task design takes a lot of time and skill which means the framework may not help to speed up the design process because to create a good apply phase, you have to do a lot of thinking a lot of research, a resource finding, etc. There was also criticism of the pair phase being mostly passive learning and not active enough, and that it does not work for all disciplines so that was also a key thing. So academic staff. Again, we've got this clear narrative structure, and that's the consistency is good. There's now a reduced number of discussions in the blank sheet approach that we had before. There would have been multiple discussions in a week. There were issues with student engagement. There were disagreements as to how many discussions there should be. I had one module where the academic wanted a certain number of discussions, the peer reviewer said there should be more, and the program lead said there should be fewer so I had three different views all in one go and I was, I didn't know what to do with it. So, at least with this it kind of removes that issue and it's just that one substantive discussion, which is good. And also that the inquisitorial approach to reading is good so because the prepare phase is only two hours. They may do a lot of their reading in the apply phase, which means that they are reading for a purpose because they're reading in order to complete a specific task or activity, rather than just being told to read something and just read it with no, potentially no purpose. Lots of criticism though, and that's fair enough. Flexibility is key and to be fair the framework should be applied flexibly but it could well have been that when we were piloting it we stuck to it quite rigidly. And that's not what academics want and it doesn't necessarily work for all disciplines. And that brings the next negative point that it doesn't work for all disciplines. For example, in some disciplines, the answer is right or wrong, there is no discussion, there is no reflection, it just is it's just fact. So I think now that we've used the framework a bit more, we can be a bit more confident about ditching certain things if it doesn't work for that particular discipline. A big thing is that it only works if learners are motivated to do the task if they don't do the apply phase they miss out on a huge amount of their learning, and they may skip it. So that means that we need to design in a way that's going to motivate them to do it. Limit an engagement in discussion forums, this could be an issue with workload. And this relates to some of the student feedback. So there are caveats with the student feedback. It's difficult to find trends due to the low response rates and small cohorts. Many of our master's degrees are quite new and it can take a while to build up the size of the cohorts. And also the feedback may be more to do with a specific module so the students didn't know if they've done a framework or non framework module they've just done a module and that was it. And when I collected data from them, I didn't tell them which one it was because I didn't want that to influence the research. So there are a few caveats with the student data. But there were some issues about them feeling there was insufficient talk content on framework modules for equally on a non framework. They felt that exercises didn't relate to the assessment enough high workload. So one person said it was there was a high workload. It was too much for them to do. But again, this could be to do with their familiarity with the topic. And also that there was just too much going on. So tasks, there's discussions, there's feedback. And so they felt that maybe fewer tasks or shorter tasks would be helpful for them so that they can actually go through things properly and participate as they wish. So this may be why, you know, discussion engagement has been quite low because students are not going to prioritize it. They're going to prioritize the learning materials they're going to prioritize the webinar and they may not prioritize that. So there's lots for us to learn from this basically so there are a few improvements we can make. So yeah having the confidence to adjust the framework according to the discipline so I think now that we've used it a bit more maybe we've got more confident we can be a bit more flexible with it and talk to the academics about what works in their discipline and adjust the framework accordingly. Perhaps make the pair phase a little bit more active. This may then help to link it better to the apply phase. Maybe they've done a bit of practice that's guided before they're sent off to do their kind of apply phase more independently and more flexibility in relation to the size of the tasks. So maybe reduce the size or if there's a bigger task to reduce the workload elsewhere and to provide that motivation but also to improve academic onboarding because there were some issues with how academics are approaching the design in relation to the framework. So perhaps we need to improve the onboarding so they understand a bit more about why we're doing it in this way why the apply phase has to be done first etc etc. So that's the initial findings. I think we've learned a lot from the process so far, and hopefully in the future we'll be able to learn more from applying it to more disciplines. So yeah, these are our references. And yeah, any questions. Thank you. Thank you, it's great to see under the hoods of how it's working. Yes, we're sorry. So how do you deal with genics for everything and teaching sink into a module and have you convinced them that maybe this is more than some of this is. Yeah, it's really hard. When some of them say what two hours we can't possibly do everything in two hours. That is something that they have struggled with. But the framework at least give it gives some kind of container first of all, whereas previously with the bank blank page, the students were just drowning and under it so at least this gives some kind of a limit. I think this also comes down more to academic onboarding and helping them to understand the pedagogy because the read a lot of the reading will come in with that apply phase it's just like you're just giving them an overview you're giving them an introduction you're giving them the tools. You're not expected to cover everything in two hours, but you should do enough that when they come to that actual apply phase they feel that okay, I know what they need, but they will need to do further reading they will need to do further research in order to do it properly so I think it's about explaining that that side of things but yeah it's a challenge for sure. I'm not entirely sure. Yeah, what do you what do you what do you think. I don't know. I mean I didn't I didn't think ABC was a was an actual framework to design within it's like a certain number of hours. Okay. That sounds like we use it. We're bringing learning types, not the same types, right. But that goes to how might it relate or how it's different. Yeah, I'll have a look. Yeah, I mean the process isn't necessarily, you know, we're not introducing any groundbreaking, we know that it's more about how we've adapted to online, how we're adapting it to different disciplines and the results of the research into it as to how it's being received but we're not trying to like introduce anything like absolutely new. Yeah. Before I continue. Yeah, just big comment as someone who was involved in the genesis of the ABC. Yeah, it was ABC was never intended to be covered for this fine process, but it's, it works very well as the initial quick file. It's a rough prototyping of module structure or structure, but then you obviously have to get involved in other methods. But you can sort of take the learning tax point and this is if I understand correctly, quite big difference in your overall framework anyway to make learning. So is that something that the learning designers use with the with the academics they're always talking about the type of learning and trying to mix that up and making the academics aware of basically not just acquire lots of other types which is where the apply phase really comes in the the the task is often to produce something. Yes. Exactly. In addition to the design design, they can't involve become ABC to sort of like we are in. Right. They are in my process. I'm impressed by your evaluation and the presentation by a program. We'll be useful to before you do the same, but in your own free framework will be useful to compare what is already available and different from work. I'm interested in this area. I would say that even now, you can take on your, your free work and compare it to others and to the content. Yeah, I mean, I think there's definitely scope for us to research more about other people, how are the people are approaching it and certainly how the students have received it as well. I think it'd be really interesting. Yes. I have a question. Did you find out just because you said you're still analyzing the data that these differences between the disciplines, can you give us a sense of science, not like this. Yeah, so I think it's for so from the pilot modules business analytics it was like we don't have anything to discuss. We don't have anything to reflect on either your code is right or your code is wrong that is it. So, for them, the structure really worked because that's how they would do it face to face so the academic was like yet it was fine I could adapt to it because it worked. You know you give them some prep work to do and then they actually do the coding and they do that kind of thing and then they kind of have a look at it afterwards but he said the reflect really came when you've done your code, and then you tell somebody else about your code, you think okay well how did this work but there's no real reflection really, whereas in more humanities based or design based subjects, where there's lots of, you know there's lots of differing opinions as different cultural influences to fight that there's a lot more to discuss and reflect upon. So those are the examples that we have so far for my precision medicine masters that I'm about to start designing. We're probably not going to bother with any of like the reflection and the discussion because I can tell it's just not part of pedagogy and I think that's really what it's it's whether or not there are grey areas, whether or not there are a moral issues where people can get engaged with those sorts of reflections and discussions and it's those parts of the framework really that are going to vary depending on the discipline. I think. We have a structure. Oh yeah, not a great. Yeah, thanks. Any other questions? I have one. It's a bit more philosophical than the new ones, but were you into the keynote this morning? Yeah, she was alluded to some of the political tensions about online learning that she's experienced in Canada. There's a bit of a narrative going on about the second class nature of online, beyond my learning experience. I'm just going to wonder what you take on that was whether you feel that that narrative has any impact on the subject that you have in your working framework. I think it. It really varies who you're who you're talking to. So, you know, some, I don't know make broad generalises but depends who you talk to some some colleagues that we work with are absolutely on board with this. They can take it to them and they can see the value straight away. Others are more well this is the way I do it. And you just want to take it and just put it online. So, you know, in terms of what Anne Marie is talking about I think, you know, yeah, just it depends on you depends who you talk to some some academics can see that the focus that the university has to do more online is going to be a good thing. There's not so much. And does it affect the students on the master's course and does that come across to all the new research. Not yet. Yeah, it hasn't yet. I think it's, I think for for the students this is the only way they're going to be able to study because they've got so many other commitments I wasn't in the keynote this morning because I was very delayed and my drive down so I'm kind of just guessing. Oh, I'll have a look. Yeah. That might be an effect. But you're saying in some of the subject areas, the scope for reflection discussions. I'm not surprised at that because we need to talk about you as soon as you. There are points that we need to discuss. Surely the point of discussion is an opinion. It's absorption of what you need to learn to see what's her name. So yeah, that's pretty black and white. I mean that's an engineering track. You know, you know, you know, but it helps to say, oh, here's a case study and here's a reflection on this. I'm surprised that you get a business in that area. I think it's more to do with the fact that the after they've done an apply phase and they've done something there's something to discuss right they're talking about what they've done and there is that space for it within the apply phase they do the task and then there's like the sharing of your outcome. So they have to share their outcome with their colleagues and potentially with their tutor so they can get feedback and they can discuss that and then they can reflect on their work. It's more about the specific. Okay, now let's discuss something specific that may help you prepare for the task. It may help to reflect on it. It's more those sort of additional discussions and reflections that we've had kind of resistance to in those sorts of subjects. But when it's to do with the actual apply phase, then they're up for it and there's more engagement. If I did this, this reflection is just helping me understand. Excellent. How can we show appreciation?