 I am thrilled to welcome you all here and to also introduce our speakers for the session, John and Megan, and the session is called Impact and Iteration, Improving Healthcare through Clinical Research MOOC. We invite you to participate in the chat throughout the session and we'll also have a little bit of time for Q&A at the end. So without further ado, over to our speakers, welcome Megan and John. Thank you, John. Hi everybody, it's going to be here. I'll just introduce myself and then I'll pass over to John. So I'm Megan Kahn and I'm Interim Director of the Digital Education Service at the University of Leeds and I'll say a little bit more, a slightly later one in the presentation about our service and how we've partnered with NIHR, but I'll pass over to John to introduce himself. Oh, you're muted, John. Okay, give me control again, thank you. So again, good morning everyone, thank you for joining this sort of session and as Megan sort of says, delighted to have this opportunity to present to you this morning about the work that we've been doing with sort of MOOCs. If you are watching the recording, we have got our email addresses at the end of the slides. So if you aren't able to obviously ask questions live, we'd be very happy to have those sort of questions offline sort of afterwards. So without any further ado, firstly let me introduce myself. My name is John Castletine. I'm Head of Learning in the Clinical Research Network, which is part of the National Institute for Health and Care Research. A little bit of a mouthful and probably needs a little bit of an explanation. So I'll start off by giving a little bit of an explanation about who are the NIHR. Now if I'd asked this question three years ago, I'm sure there would have been blank faces all around the room. Hopefully with the pandemic, the NIHR's role has become much more visible in terms of what we do across England and sort of beyond in terms of enabling and delivering world-leading health research and care research as well. So you do see a number of members from the NIHR appearing in national sort of media certainly over the last three years. The National Institute for Health Research is however a virtual organisation. It's made up by a series of contracts that are issued by the Department of Health and Social Care. And it's one of those contracts that was issued to the University of Leeds along with Guy's and St Thomas's for running the clinical research network, which is why you sort of see myself and Megan come together today to talk about the MOOCs that we've been running, a partnership between the National Institute for Health Research and our host, the University of Leeds. And just before I move on to the next slide, one thing I would really like to acknowledge is that health research is only possible through the efforts of patients, carers, members of the public that co-create the research with us and volunteer to take part in research studies. So a critical part of the research infrastructure is public and patients alongside obviously the researchers that we support. So why MOOCs? Why are we interested in MOOCs? Why have MOOCs been a really useful sort of tool that we've been deploying in the National Institute for Health and Care Research? Well, firstly, our role is to support the NHS and other organizations build and maintain the capacity and capability of the research workforce. And particularly in the clinical research network, we're focused on those that are delivering research in hospitals, in clinics, in GP surgeries, in pharmacies, and sort of elsewhere. So we look to support that network of individuals across the whole of the country. But of course, it's wider than that as well. We really need to engage with the full clinical and healthcare sort of workforce. Your GP may be the person that could alert you to a study that may be suitable for you. Likewise, the pharmacist. Equally, we want to ensure that as many clinicians as possible are aware of careers in research and the role that they can sort of play if you will, balancing a clinical role with their sort of research role. And again, I come back to the critical role of patients and members of the public in partnering with us in any sort of research study. And just to give you a sort of sense of some of the numbers, the research delivery workforce that we directly support is of the region of 10,000 to 15,000 people spread across England. The wider research workforce in England is at least 40,000, if not sort of more clinical professionals. Of course, the wider clinical and healthcare workforce runs into millions. And as you can see on the slide there, through the peak of the pandemic of COVID, 1.4 million participants were enrolled on studies that we were coordinating within the clinical research network. And of course, that was up on the previous year. But if you will, a normal year when we were looking at some research which wasn't around COVID, over 700,000 individuals participating directly in research that we were supporting. So in addition to formal education that we sort of support, in addition to what members of the public may learn through social media, there's a key role for us to engage using the channel of MOOCs in terms of its open access which is fantastic for engaging these large numbers of people, both clinical professionals and members of the public. Bringing greater awareness about what is health research and the different roles that individuals can play. So a little bit of an introduction to the NIHR and why we fit into the system with the University of Leeds. As I say, it's been a really great partnership with colleagues across the University of Leeds to make it live the MOOCs and really bring the advantages of MOOCs into the NIHR. So with that, I'll turn to Megan to talk a little bit about the Digital Education Service. Thanks, John. So just very briefly, colleagues may or may not know, University of Leeds is one of the founding partners of Future Learning. Future Learning is the platform that the MOOCs that we'll be talking about in a little bit more detail run on. And the Digital Education Service at Leeds has developed over 120 courses for the platform since its launch in 2013. The portfolio of courses that we've developed and that we've run are for a number of different purposes. So some of the key things that we have tried to do with the courses that we've developed include courses aimed at pre-university, so 16 to 18-year-old thinking about university study, either to help them understand a little bit more about the subjects that they might be wanting to study but also for skills development to help them with that move up into undergraduate study. Then also courses aimed at our own students and undergraduate students elsewhere for skills development or indeed for them to think about transition to a taught postgraduate study. And then courses for professional learning or continuing professional development in various different sectors. But most relevantly for our partnership with the NIHR, we've also seen a really important role that the platform and the courses we develop can play in public engagement and research dissemination and knowledge transfer. And the partnership that John has spoken about with the NIHR has been one of our really most successful examples of that model. So working with partner organisations to support their public engagement activity using the medium of open online courses to reach a broader audience and to engage them in understanding in a deeper way that might be possible through other mechanisms. So in this case engaging key audiences with understanding what health research is and how they might engage with it and what that might mean. And we have some other partnerships in the areas of digital skills and also in arts and cultural organisations. And then I'll pass back to John who will talk in more detail about the courses and how they've been developed, etc. But we'll just say at the end about how we've then kind of continued to partner with NIHR and others to think about the evaluation of the impact of courses and then use that to further our pass back to John now. Thanks, Megan. And we title this presentation impact in iteration and we really want to sort of bring alive the impact that this MOOC has had for us. But also as you'll sort of see in a moment, we've introduced more recently a second sort of MOOC and that allows us to do a little bit of a compare and contrast between these two courses. And that helps us with the iteration. So if I start with the long standing sort of MOOC, a massive open online course that we've been running, improving healthcare through clinical research, we launched this not long after the University of Leeds, along with Geisner St Thomas's, won the contract to operate the clinical research network's coordinating centre. So this launched back in November 2015. And as Megan sort of says, through the University of Leeds working closely as a partner with FutureLearn, this course has sort of sat on the FutureLearn sort of platform. Since its launch in the very end of 2015, over 46,000 learners have enrolled on the course. And as you can sort of see by some of the sort of stats there, it has been very well regarded in terms of the sort of various sort of ratings, very sort of positive sort of feedback that we've got from the course. It was structured, is structured as a four week course. And if somebody wants to work through all the activities, a typical learner will probably take four hours per week, so 16 hours in total to work through all the sort of content. The activities, the modules, if you will, in the course are very much sort of video based. They're very much sort of focused on our lead educator, Professor Alan Gore, who takes the individuals through this sort of four weeks of journey to bring alive the process of clinical sort of research. And when we started the course, we offered it in very much a cohort model. So the, there was a waiting list of people sort of signed up and then three times a year, we ran the course when it was sort of available to people and it was very actively sort of moderated during that sort of time. And then it closed again, there was a waiting list and we went through that sort of cycle three times a year. More recently, and based on the feedback that we got from sort of participants, we moved to a sort of forever on sort of model where it is available throughout the year. It is sort of actively moderated in sort of waves, but there is sort of some ongoing moderation throughout the year. And we tend to reset the comments on an annual sort of basis. We work on the financial year. So if you do go in to look at this little link after today's presentation, you will find that it is very light on comments at the moment, but that is because we have just literally sort of cleared the deck in terms of the comments and restarted the course again. And this also really helps us for tracking year on year for the statistics and the analysis that we do to the course. On the screen now, you can see some of the feedback that we have received. Obviously, we've got a wealth of sort of feedback from the 46,000 individuals who have engaged with this, of course. What this illustrates here is that we are attracting particularly those people who have got an interest in clinical research, in clinical careers, or maybe people who are already in a clinical role who are looking to sort of understand more about the roles of clinical research. So some fantastic sort of quotes that we've sort of enjoyed receiving from this sort of MOOC. I'm now going to say a little bit about the second MOOC that we launched. We launched this in November 2019. And as you can sort of see there, the title is What is Health Research? And the reason that we decided to launch a second sort of MOOC was that the feedback that we were receiving is that the original MOOC wasn't maybe focused towards meeting the needs of public and participants fully. In terms of really explaining to members of the public what it was like to partner with clinical professionals in the processes of clinical research. We took the opportunity to make this a shorter course. So this is three weeks, and it probably takes about three hours on average for an individual to complete each of the weeks or nine weeks in total. As I say, it's very much designed to walk in the shoes of members of the public, either participating in a research study or partnering with us in the design or dissemination of research findings. Since we launched it at the very end of 2019, over 6,000 learners have gone through and rolled on the course. And again, we are seeing sort of very favorable ratings, 4.7 out of 5, as you can see there. And what we do see with this course is a significantly higher completion rate, which is something that I'll come on to in just a second. Just before we do, again, just a few sort of nice sort of quotes there from individuals who participated in the course. And maybe just one thing to pick up from this, as well as members of the public engaging with the course. What we have sort of seen, maybe we didn't fully anticipate it at the time, is actually we get a fair number of clinical professionals also going through this course, wanting to walk in the shoes of members of the public and really understand what does the research process look like from a participant's point of view? What are the types of questions that members of the public may have about a research study? So I'm really grateful to the work of colleagues, Megan and her colleagues in the Digital Education Service, for starting to really look at the analysis of how these MOOCs were performing. And that has given us great insights for, firstly, the iteration with the what is health research MOOC, but also as we consider moving into the next iteration of the MOOCs, and particularly the improving healthcare for the MOOC in the next generation. What you can see on the screen here is the first piece of analysis that the Digital Education Service did looking at the improving healthcare through clinical research MOOC. And it was to look at how many people were posting your comments and against which of the activities in the MOOCs were, which ones were triggering the greatest response in terms of the number of comments, and you see that in the top sort of graph. And where we were running an activity that then sort of raised the question and invited people to put comments about what does the term respect mean to you. That was one of those sort of activities that did engender lots of your comments being posted into the MOOC. What you can sort of see in the bottom is analysis for how much the comments were, how long the comments were, how many words people were using. Again, there's a bit of an indication as to how people were engaging with individual activities through the MOOC. Now, of course, we don't need individuals learners going through the course, postings of chapter and verse against every activity. But I think what this analysis really helped us think about is actually which activities and the phasing of the activities are likely to trigger that greater engagement. And through that greater engagement, can you then retain people and encourage people to move through the whole of the course, rather than maybe just sort of dipping in and out of components of the course? And this slide really sort of brings that alive as we look at that comparison between the improving health care, which is the first row and the what is health research, which is the sort of second row. And on the screens in terms of the graphic sort of representation of the number of people that we retained within the course, as you can see on the left, the improving health care on the right, what is health research for the MOOCs. We did this analysis over a similar sort of period of time. So that's why you see that there's 27,000 enrollments rather than the full 46,000 enrollments in the improving health care sort of MOOC. So three things that we took from this sort of data. Firstly, by being quite targeted with the what is health research MOOC, targeting that towards members of the public, you see that the number of activations people who really enrolled and started to get involved with the course is significantly higher. And that is obviously a trend that is then retained throughout the MOOC. Secondly, in the purples of text, looking at the comparison of the figures throughout the sort of stages, the number of people that we retained through 25%, 50%, and then 75%, all sort of marginally higher, but obviously adds up to an overall sort of completion rate, which is sort of significantly sort of higher. And that was partly using that analysis about where do we put those prompts around engaging people, asking them to sort of write comments, asking those provocative sort of questions, and putting a lot of those maybe a little bit earlier on in each of the three of the weeks of the presentation. And then finally, one of the things that kind of always sort of surprises a little bit when we looked at the improving health care MOOC was the very low completion, the fully completed stats, you know, the 7.8% that you sort of see there on the screen, but really it is that very last sort of step. And we've probably over complicated the end part of the course in that respect. And so by simplifying the end of the course, it allowed us to give probably a much more representative sort of view as to how many people working through the complete sort of course, all the way through from sort of beginning to end. And you see that with the much higher figure in the what is health research, the second MOOC there. So final bit of analysis that I'll cover and then hand back to some Megan to talk about just this analysis, but also then comparing it with the other MOOCs that are run within the digital education service. And we look to the demographic data. And as you can sort of see on both courses, most of our learners are female. We are attracting a large number of individuals in that sort of age group of 26 to 35, and individuals who are working full time. But when you dig into a little bit more detail, unsurprisingly, the what is health research MOOC, which is attracting members of the public, we see a much larger number of people who are retired or not working. And sort of obviously, likewise, that correlates to a slightly higher age range. So that wasn't a surprise. But again, I think it helps illustrate that we are targeting that MOOC towards a particular demographic. So with that, I'll hand back to Megan. And if there's anything you want to say about these slides, but then certainly please go on and talk about how you did that analysis comparing the MOOCs that were running within the NIHR with that wider portfolio that you run in the digital education service. Thank you. Thanks, John. Yeah, so I mean, John has presented it very clearly and I'll be brief to make sure we have some time for questions. But just to say that it's very much part of the partnership that we've developed and maintaining with the NIHR and with others in terms of obviously supporting the design and development of the courses in the first place, but then having an ongoing focus on the evaluation of the course performance and who the learners are and then how they're engaging and what the impact is. And it's great to work with colleagues such as John who are so engaged with that analysis. In terms of other courses, so there are various different ways that we can compare the NIHR courses either with other courses within our portfolio, but also courses run by other providers on future there. So this slide shows a comparison with other courses on future there, not just these ones, but other ones in the health and medicine category, particularly thinking about courses in the subject area of health research. So first of all, just thinking about the health category, the performance as John's already talked about of both of these courses is good in terms of enrolments and then the engagement and completion through. But as we've seen, the completion is much stronger with what is health research, the higher enrolments on the improving healthcare through clinical research, but then a higher drop-off rate. We then had a look at other courses on future there and specifically in categories of health research and clinical research to identify what, whether there were some kind of key characteristics of those courses, which were in common across the highest performing courses in that category. And there we did see that the kind of hours of learning matched in terms of the 12 to 16 hours of learning across the whole course. There's a definite correlation between the courses which performing well in terms of enrolments and then having CPD accreditation, which is interesting given especially with these courses where really the primary aim is public engagement and knowledge transfer, but there is still clearly that appetite for people to have a badge or something that they can take away with them. And especially for those learners who would be completing and that's where that badge would come into play. And then there did seem to be a trend towards courses that had a general research skills focus being high performing. Again, suggesting that although we're aiming these courses to public engagement, as John's already said, there is also the element of clinicians and others wanting to upskill themselves as well, even through the what is health research course in terms of understanding that research participant perspective. Some slightly speculative thoughts, I suppose, from the team who have been, and I should say actually that this, I'm presenting this activity, but it is very much a team effort. We have a team within the Diseducation Service who have been leading on our Insights activity and this is very much their work. More recently, they've been looking at our courses across our portfolio, but including the NIHR ones and thinking about shifts in future learners, model in terms of how content is presented to learners, especially learners accessing courses for free without either subscribing or paying to upgrade for a particular course. We do see that most learners tend to skip kind of assessment content and introduction content and are heading into the bits of content that they are most valuable to them. And so some of the simplified design that John was talking about for the second course kind of speaks to that. We're also seeing, although this is very early, because it's a recent change, but future colleagues may be aware, future learners have recently shifted to for learners accessing courses for free that content becomes available week by week, so you can't kind of work through a full three week course in one week unless you are subscribing. And that it does look like that might be having a negative impact on those free learners and their drop off, as you might expect. And that's in terms of where we've got courses where we do want learners to subscribe or engage, it's not necessarily a challenge, but in the open education space and in particular with these types of courses where the real purpose is to engage as many people as possible in the health research and in that understanding, it does present a little bit of a challenge that we need to think about. So that's kind of something for us to think about going forward. The other thing referenced on this slide is just John mentioned the difference between the two courses in that the second course is shorter in length and just that that's backed up in literature in terms of that the shorter courses tend in to have a higher completion with higher retention and completion rates. So just an interesting comparison there. Just very quickly, which we have to run out of time. As I say, we're keen to work in partnership with colleagues such as John on continued evaluation, and there's a number of different areas that we want to look at there. I won't go through them all. In terms of our key insights, and we'll pass back to John in case there's anything he wants to add here, but it's really just that point that well, the active moderation John may want to say more about, but the learning preferences towards the kind of shorter courses and accessing the content that is most relevant to them seem particularly important and that by paying attention to the completion rates, but also those measures digging into where the where the engagement and the active participation is happening has really helped us with iterating course design. And this has been a great partnership for us to be able to do that. I'll just pass back to John for the final thoughts. Thanks, Megan. And just really to highlight that first bullet that from day one, we went with a philosophy of actively moderating and respecting everyone who's put comments in. You deserve sort of a bit of a response. So our team of lead educators and moderators trying to be as welcoming as possible, encourage as many sort of questions as possible, ensure that people feel that there are individuals supporting that learner journey at all the points through the year. And I think one of those that's given us a lot of good feedback that that along with everything else that we're talking about is part of the reason we're getting the metrics that we are getting on these two MOOCs. Thanks everybody. This is Mara and again the moderator. Thank you so much. John, Megan, before we finish up the session, are you happy to take a few questions or did you want to just have a few more points? Why are you ready to take some questions? Happy to take questions. Fantastic. Well, we've had really good comments starting to come in through the chat here. So Paul is saying really interesting talk. Thank you so much. We have a question from Anna, Anna Page. So Anna says it sounds like you might have to do bite-sized courses to get round that future learn barrier for those who are not subscribers. Did you want to pick that up? Which one of you wants to come in on that? I'll come in on that. John, do you feel free to chip in as well? I think absolutely. Because of that stage release of content, we really want to enable learners to access for free and be able to access all of the content and making it more bite-sized a week long would be one way to do that. It then becomes interesting learning that starts to, you think to what extent is it a course or is it more a learning resource that people might need to be accessing, which itself could be fine. That might not be a bad thing. But there's some really interesting questions we need to think through there in terms of the design. But yeah, absolutely. That's a really insightful thought. Thanks, Megan. John, did you want to come in on that? I think just to add that we fully understand that future learn have a platform to run. They need to have a freemium model. They do need to think about the financials. Equally, as Megan is saying, we do want to ensure it's a great learner experience and people aren't so frustrated if they do want to move through the full course quickly. The other thing I'd say is that we also educate people that the clock starts ticking once they've subscribed to the course. So not to sign up for the course, leave it a couple of weeks and then get frustrated that you haven't got the time to sort of complete the course. And I think that's really just about learner education. That's a really good point. Thank you. We have time for one more question. This question is from Paul here in the chat. And he's saying, I was wondering if you're finding data that supports the shift to bite size in terms of student engagement or similar. So are you finding data that supports the shift to bite size in terms of student engagement and so on? Yeah, so we do. I mean, to be honest, the majority of the courses that leads ones on future then are fairly bite-sized in terms of kind of number of weeks and content. So the courses I mentioned, which are aimed very much at students or potential students kind of come into university are all two weeks and kind of live content within each week, kind of a couple of hours per week. So they are pretty bite-sized. And that was a deliberate decision based on really wanting to get that engagement, but to kind of not demand too much in terms of the engagement that really it was a taster. And that's the kind of aim of the experience. I guess the challenge is where for the particular audience or the purpose of the course that you're aiming for, you're wanting to take people on a more in-depth journey. It's then how we make sure that we can do that whilst retaining that engagement. In terms of the data, the data mentioned in terms of kind of initial indications based on the more recent shift is really just an initial indication we need to do some kind of proper work on that to make sure that that what seems to be happening actually is. So that's kind of work in progress at the moment. Thanks Megan, that's really interesting. So I think we're nearly at the end of our time about before we put our virtual hands together and give you both a big thank you. Just any final comments from you, John or you Megan? I just firstly thank you for everyone for participating today here in what we have to say and just reiterate what I said at the beginning. If there are questions when people reflect on this or see the recording we'd be more than happy to hear from people and our emails are on the screen at the moment. Thank you. I would just echo those thoughts absolutely. Well thank you so much very much for joining us here at OER 23 and thank you to everybody in the room. It's been a really thought-provoking session this morning and the recording will be made available but I'd also like to encourage all of you to head over to the Discord channel. There's a dedicated chat for today's sessions so if you want to post slides in the chat there to share them please also continue to follow the OER hashtag on social media and share your work more widely this way. From everybody here at OER 23 see you at the next session and thanks again to our presenters for a really wonderful session today. So thank you very much John and thank you very much Megan.