 Good morning. Hi everyone. Thank you for attending this session. We have had two presenters drop out. So the first two talks you see on the program have dropped out. So the good news is that we have more time if they wish with our two other presenters. And the other thing you will notice is a very big room with a small group of people. And that's always difficult to manage. So if I can ask you all to move forward, please, if you don't mind, it will just help the group work and the connection. And the third thing to know that we are on YouTube. So hello, mom. So, but we'll manage it as usual and try to have a conversation with you guys here whilst kind of keeping an eye on our participants from around the world. So without any further due, I'll call over our first speaker, Jessica. Okay. So I'm still going to try and stick to my original timing. I'm here today to talk to you about empowering students to improve with my feedback, moodle, my feedback. And before I get started, I'd be interested to know who here is from an institution that uses moodle. So most of you, great. Okay. So two years ago, I presented at Ultsie about my feedback report that we were developing. And the timeline goes something like this. So in December 2014, we released the first version onto moodle.org. And in September 2015, so last time I presented at Ultsie, we were about to start a pilot with around 750 undergraduate students in the engineering program at UCL. In October 2016, so about a year later, we launched the feedback, my feedback report to all staff and students across UCL. And I'm about to start evaluating how it's being used. So what I'm here to do today is to talk to you about what the report looks like now and where we're hoping to move in future. And for those who do have moodle, it's available for download from moodle.org. So if you want to try it out for yourself, you can download it. What it does is it allows staff and students to easily view grades and feedback from across all of their moodle courses. And I thought I'd just take a quick screenshot of the usage and downloads of moodle in installations across the world. So you can see that there's been a steady increase and it's on roughly 360 moodle installations now worldwide and it's continuing to rise. I'm in talks with a few institutions as well about helping to contribute to the development. And if you'd like to tweet about this during the presentation or afterwards, if you could use the hashtag moodle, my feedback, that would be great. So a bit of background to the report. Quite a few years ago now, the IOE ran an assessment careers project and out of that project was developed a report that ULCC actually developed for them. And because it was a GISC funded project, we were able to get our hands on the SQL for that report and we used that to develop our own report. The big difference between the IOE version and the one we now have at UCL, because this happened before UCL acquired the IOE, is that the UCL version is for both students and staff, whereas the one that was developed at the IOE was only available to staff. So students couldn't actually view their feedback within the report. And we felt it was very important at UCL that students were able to use it to empower their own learning without having to rely on staff for that information. So the project aimed to make feedback more visible. One of the big problems in moodle is that students don't really understand how to view their feedback a lot of the time and it's not always straightforward because they might be studying a number of modules where feedback is given in different ways and it's not always explained to them very well how to view the feedback. So they might be using Turnitin in some modules and moodle assignments in others and quizzes and what the report does is draw all of that together in one interface. So it's in a single view and that allows students to compare their feedback and identify common areas for improvement. We're hoping that this is going to be done in collaboration with the personal tutors. So I know a lot of institutions have personal tutoring programs where students can go for support. But one of the big issues is that personal tutors don't really understand how the students are performing academically. They don't have access to that information. So one of the things that the report does is make that visible to the personal tutors. And we're also hoping that it's going to increase staff adoption of the assessment and the return of feedback using moodle. So that's already happening naturally across the institution and it's really helping to provide feedback quickly as well. At UCL the Union petitioned for four-week turnaround of feedback. So now across the institution in every module students are supposed to be receiving their feedback within four weeks. And in some areas it's actually dropped even lower to a two-week turnaround for feedback. And this has been made possible through tools like online assessment in moodle assignments and turn it in. So what does the report do? It takes feedback from a number of different assessment tools in moodle. So at the moment we've got moodle assignments, turn it in assignments, quizzes, workshops for peer assessments and also manual grade items. So things that are added manually to the grade book or maybe bulk uploaded into the grade book are displayed in the report. And with code adjustments further assessment items can be added into the report. So I'm currently speaking to the developer at RVC to add the coursework module to the report as well. So if anyone's using that here, does anyone have the coursework module that allows for blind anonymous marking in moodle? That should soon be appearing in the report as well. So the benefits of this tool is that as I've said already students can find and compare their digital feedback. And what we've been hearing informally which I'm about to start evaluating is that students are starting to use this tool as a first point to go and view their feedback rather than going into their moodle courses. As you know if you're using moodle courses are very siloed so the information can be difficult to find. Once a student's in a course they need to know where to look for the feedback. So personal tutors as well can now help students to act on their feedback. So they've got a talking point for these meetings where they're supposed to meet with their students a number of times a year and they can view the report and see how the students are going. And I want to show you some screenshots in a moment. And module tutors as well can see how their students are performing overall. They get a dashboard to view that and they can also look at individual students. So is anyone here working in a technical role? Okay so about half of you. We use a plug in to moodle called user role assignment from external database to actually draw in our students from SITs which is our student record system into moodle. So it makes a connection between the personal tutors and their tutors, their students. And that's done using the moodle parent role. So if this is something that you're looking at implementing this is a great way to scale it up. So this is how we're able to give personal tutors access to their student data for everyone across the institution and we're talking over 40,000 students and many, many stuff. So I'm going to show you now what the feedback report looks like from different perspectives and we're going to start by looking at what it looks like to a student. So when the students first come into the report they see this overview page and it shows them all of the modules that they're enrolled on and any assessments which have grades and feedback visible. So if anything isn't yet available we don't show it in this report and that's something that we were very rigorous in our testing to make sure that we're not showing the students anything they can't already see. Which is quite complicated when you look at all the different ways that things can be hidden in moodle at different levels. And students can then click through into the assessments to have a closer look and we provide a full feedback link as well. So students can then click through into say review their last quiz attempt for example because a lot of the feedback especially on quizzes is going to be embedded into the quiz questions for example. And then if it's a numeric grade they get a bar graph which gives them a really easy way to see how they're performing. Actually one thing to note that was very important as well I think the original report we got from the IOE didn't have the range showing and that made it difficult to know if your A was out of A or A star so we had to make sure we put that in there as well. Everything can be exported to Excel which students quite like to do because a lot of them are trying to see how well they have to perform to make sure they get there first. And you can also modify how many columns are showing. This is a responsive design so if you're using it on a mobile device it automatically drops down to just show limited information and then you can click to open up and see more information about each assessment. And it remembers across logins as well what you've decided to show and hide. So that is persistent across logging in and out of Moodle for each user. The most important interface really and what I would like to do is have this as the default appearing when you first come into the report is this feedback comments tab. And this is where you can see general feedback from Moodle assignments and quizzes and other assessments appearing on the page so that you can start to compare and see whether there is repetitive feedback coming from different modules and across different assessments so that personal tutors and students can start to think about how to improve for the next assessment. And one feature that we were keen to put in here which is available is an area for self-reflective notes so students can come in and say okay so I've had a few comments here about a certain area that I need to work on maybe structure of my assignments and I can put a note into myself so that I can make it clear this is what I can work on to improve in my next assessment. When we spoke to students in our focus groups they said that they wouldn't be using this feature. So we will be working with personal tutors to see if that can be incorporated into some of their their modules so that they see a purpose for actually coming in and using their self-reflective notes. And you'll notice a little help icon alongside the title there. This makes it clear to students if they click on that that this area is only visible to themselves and their personal tutors. When we spoke to the students they made it clear to us that they didn't want their module tutors so the people that are marking them to be able to see everything in the report because they were worried that that would then influence the marks that they receive on future assessments. So again this differs a lot from the IOE report that was originally developed where module tutors could see everything and could use it to check that students were actually acting on their feedback. Okay so that's what it looks like for a student. From a staff perspective the staff get a few additional areas so they get this my students list and if they have personal tutors these will appear automatically. We used to have all of the students appearing after this that the staff member was teaching but obviously this took a lot of time to load the page so we've actually cut that back and if you want to view a specific student from here you'd need to search for them in the search field but the personal tutors display automatically and you also get this dashboard personal tutor dashboard which gives you an overview of how the students are performing. A few things to note about this though is that some of these numbers might not be accurate depending on how the Moodle course has been set up. So for example with non-submissions if there are optional submissions available for that course it might be a quiz that's not compulsory or an assessment that might be allowing students to submit a draft and get feedback if they choose to that will show as a non-submission so it doesn't necessarily if this number is higher doesn't necessarily mean the student's not been engaging and that's something that staff have to understand when they're looking at this information. The same goes for late submissions if staff aren't setting up the assessments with extensions for example so a student might have a legitimate reason for submitting late and have been given an extension and that if that isn't put into Moodle or if perhaps they're using a turn it in assignment whether it's not possible to add an extension that's going to show as a late submission. So really this is a conversation starter for the personal tutor that to then ask the student if are they actually submitting late or have they received extensions and the graded assessments as well is useful. The low graded obviously will only work for assessments that have a numeric grade so that scale grades won't show up in here and what you can do here is break down into for each student into specific courses so if you click the little toggle and look at the course breakdown you can see course by course how the student is going. One thing to note is when you click on a student's name and you're taken into their report as a personal tutor you probably won't have access to their Moodle course so this report doesn't actually change any of the permissions that are already set up in your Moodle which means that if you click through try to click through onto the review attempt you're not going to be able to access that and you're going to get a permission error and one thing that we might do if we do get further funding for further development of this is think about maybe graying that out if somebody doesn't have permission because it has been confusing some of our staff when they click on that and all of a sudden they can't access that assessment because it's in a course where they don't have permission to view that. Some departments have worked around this by allowing their staff read only accessible their courses but there are some ethical considerations about whether students are comfortable then with other staff who aren't actually teaching on the module coming in and seeing what they might consider to be private discussions so that's not really very widespread at the moment. There have been discussions about that though. One very useful piece of information on this report is the viewed date. So has a student actually come in and viewed their feedback? This is one of the things that I think might slow down the generation of this report actually because we're having to for those technical people there view the log file in Moodle which if you know the Moodle database it's a huge table and the viewed date will show whether or not the student has actually come in and viewed that piece of work. So if you see across there the person who knows that that student hasn't actually seen their feedback which can be quite disheartening when you see how much effort is put in by the tutors a lot of the time to provide that feedback. So that's another conversation point for these personal tutor meetings. You'll also notice at the top here that there's an option to choose the current year, the academic year and what we had planned when we first envisaged this was to have a way to view previous year's work. Now what we do at UCL is we actually look back, we take a snapshot every year of Moodle and we make it available as an archive. So students could using this feature look across different years. We did encounter some problems when we were testing it though. So I said before that we made sure that nothing is shown in this report that students can't already see and we ran into some problems with older versions of Moodle. So we've actually had to turn off this feature. So only the current academic year is visible at the moment. It's still possible to enable this but we don't recommend it for production systems. So what is possible is the students can log into the old version so the snapshot from the previous year and they can review the report from that year from last year onwards. But that's an individual process. They have to go to that other Moodle separately. They can't easily get to it from here. So we would at least like to link them up a bit more easily so that you can easily jump from this current year to other years and ideally view all of the years so that they can be compared. You'll also see that as we're viewing this as the students report we get this button at the top here that says view our own dashboard which is how you can get back to looking at all the other students that you might be supporting as well. Now as a module tutor we get a slightly different interface. We can see for each assessment how many students are actually achieving certain grade brackets. And when we ran this project we had a business analyst go out and speak to a lot of the academics and find out what made most sense across the institution both undergraduate and postgraduate programs. So we have an overall grade chart that you can see here a bar chart and you can see how many students are actually achieving grades within each bracket. And again you get similar information with a similar caveats about non submissions late submissions that I've already mentioned. Underneath each assessment as well you can go in and have a look at a student breakdown so you can see how students are performing or you can switch to a student breakdown view and then it goes student by student and you can break down into assessments underneath each student. So look at it that way. And most of the time people will be looking at this for an individual course but they can also select multiple courses to analyze in the top right corner there so that they can compare across different courses. Now those of you who are using Moodle do you know if you have category level enrollments? Yeah so a few people are nodding. Okay so there's a couple of you here that would be able to enable departmental admin view then. This relies on the category enrollments plugin which we use at UCL and it allows you to have a broader view of how students are performing across the whole department. In some cases we have categories for undergraduate and postgraduate so it's split so you could then look at those different levels as well but this is intended for those who are in charge of looking after entire departments. So the feedback obviously has to be in Moodle for it to be of any use for the system to be of any use to students and that's our next step really is working with staff to make sure that they're putting feedback into Moodle in places where it can be drawn into the report. One thing to note here is turn it in assignments don't automatically come through into the report. The feedback isn't available in an API or anything and I don't think they have any plans to release an API that would allow that to be drawn in automatically. So what we have done is we've given students the ability to copy and paste paternity and assignments but we're not expecting many students to actually want to have to do that. So one thing that this is actually leading to is a move towards Moodle assignments more, being used more than turn it in assignments and one thing that's going to aid that move across as well is that we're in the process of getting turn it in assignments integrated into Moodle assignments. I know most institutions have that already. We still have a separate turn it in direct plug-in separate from the Moodle assignment so it makes the whole setting up assignments and students viewing their grades a bit more complicated because there's different options and different decisions, more decisions that staff have to make when they first start setting up these assignments but hopefully we'll move to a simplified version in the future. One thing as well with quiz questions is that students might not know that their feedback is attached to a quiz question which is where it makes most sense. They've got a specific question wrong in a quiz that want to know why. So one of the things we recommend is in the general feedback for a quiz just guiding students to tell them to look at the feedback on the quiz questions which they might not know to look for. And again making sure staff are granting extensions in Moodle assignments and quizzes so that information isn't coming through and showing late submissions incorrectly in that in that column. Okay so a few things to note here I might take away points. We've been running this project for quite a while now and communicating is existence to staff and students is very difficult. We have noticed students starting to use this just because they've noticed it when they log into Moodle but staff we really need to work with them to make sure that they're putting feedback in the right place and they know to tell their students about this and that personal tutors know that they can use this as a talking point in their personal tutor meetings where they may otherwise struggle to really start the conversation to support the students. And personal tutors as I mentioned can't access the actual assessment unless they're enrolled on the course so that's something that institutions have to deal with by looking at their permission systems and the way that that's been set up in Moodle and make a decision about whether they want to allow read-only access across a department or not. And as I've mentioned as well the turn it in feedback not being displayed in the report it's still very easy for students to view their turn it in feedback because all they need to do is click on a link and it opens the turn it in assignment. So it's still a big improvement on them not being able to find their feedback because they're not quite sure where that turn it in assignment might be buried in a course somewhere. In terms of technical tips you need the category enrollments plug-in for departmental admin access and permissions need to be set at a site level so one of the things that came up in the forum posts on Moodle.org is that some people were setting permissions at course level and this report actually operates at site level so that wasn't actually very useful. So the permissions you need to basically set up new roles at site level to enable students and staff to view the report. And I've already mentioned that the archive feature is not recommended for production installations. We are hoping to develop this further at some point. At the moment we don't have a developer working on this but hopefully in the future the archive feature will be able to be used again. And when we did talk to staff and students about what they would like to see in the report students were saying they wanted the archive feature the previous year's feedback. They also wanted to see performance against class average and I know a number of institutions do this in various ways but we decided not to implement this. There wasn't consensus amongst students that everyone wanted to see how they were performing against the class average and we were worried about what it might mean to students who might not be performing that well. So we have backed away from implementing this at this stage but it's still something that might be considered in future. Assessment weightings as well is something that students asked for and we don't have a reliable way of really drawing out of that information at the moment. Same with formative and summative assessments. There's no way to set an assessment in Moodle at the moment to be formative or summative but that would be very useful information for the students because then they could filter to just the assessments that count towards their final mark. And we were considering how to do this and I think one of the ideas we came up with was maybe using different categories within the gradebook and having a summative folder where we could then put assessments into that folder and they would appear in the reporter's summative. What would be nice though is if Moodle had a way to tick an assessment as being a summative assessment across all the different types. And the other thing that students asked for was for notification when feedback is given. I've forgotten the name of the plug-in but there was a plug-in in Moodle that I was hoping would be able to do this for us so we didn't have to build it into the report. And I think it turns out that only staff can easily set these notifications up. So it relies on the teachers of the courses, the tutors, to allow access for feedback notifications to be given which is not really what we're looking for. So we're hoping that the alert system that's in Moodle will be developed further so that it can be used by students independent of the tutors. And in terms of staff feedback as well, they've asked for improved page loading times. As I mentioned, the log file is huge in Moodle and it does slow down the page loading on a number of pages. We'd like to enhance the personal tutor dashboard as well so that students you can see which courses students are studying and make some improvements so that the mobile interface is clearer because not everybody realises that you can expand the information. And I've already mentioned resolving the archive problems. We'd also like to link directly to annotated Moodle assignment PDFs. So at the moment you have to click through into the Moodle assignment and then click on the assignment PDF. So we'd like to make that as easy as it is with Turnitin assignments at the moment. So there are a few things we have planned. If you do use this plugin, we have a Moodle tracker that you can raise these suggestions on further suggestions and report any bugs. But basically we need the community to help develop it, which is the main reason I'm here today, to tell you about this and that if you are interested in something like this rather than developing something independently, this is available on GitHub and as a community we could develop it together. There are a number of institutions I'm already talking to about this. But basically here are some links for you if you'd like to find out more about it, see what it does and suggest any improvements to it as well. There's the tracker and the GitHub links as well and documentation online. So basically if you have any questions I'd be happy to hear them. Thank you, Jessica. Any questions from the floor from you guys? I have a question around anonymity. So when I was at the University of Cambridge there was a big discussion about anonymous submissions. So how does the system deal with anonymous submissions? So the staff member can't actually see it doesn't come through into the report until it's been made visible to students, which is at that point no longer anonymous. So the report doesn't actually show assessments that are still requiring marking or that haven't been released to students. So we get around it that way. Any questions from the floor? You mentioned that you were reluctant to implement the revealing students grades against the class average. I wondered if you could say a bit more about this. I know there's research that shows that some students find that really motivating, but for the bottom 10% it tends to make them basically leave university. So I was just wondering what your institutional thinking was around that issue? Basically we're not brave enough to try that until we know more about what it would do to the students who are not performing that well. And we even had students in our focus groups who were performing well who didn't feel like they wanted to see that information. So there would be a question around whether or not we make it an option so students can choose to turn it on. Although even then I think my students would want to see. So it's something we might address again in the future, but at this stage we're just leaving it off just to be safe. So thank you very much. And round of applause. Just passed over to the second speaker. Thank you. Hello. This is quite a large room. So forgive me if I bring out something that I call teacher voice. But it's quite privileged to be in such a large room with the best members of all that are stuck around after the keynote. And those that are live on YouTube, hello wherever you are. I think I meant to say hello mom. So hi mom. Cool. So it's quite a privilege actually to be following Jess who I used to work with at UCL. But I've now moved into a new role at FutureLearn and I'll give you a bit of background around that. But my talk is around this concept and I really appreciate we're going to be doing something interactive. So warning, you're going to be using your computers and your brains to be doing things. But around this concept of building what I call a pattern library for designing for learning. And I'll explain what that means in a moment. So to give you some context, I've spent 13 years in e-learning and counting. I started with doing stuff with children. I'd had a little dabble in further education. I spent a long time at UCL doing higher education. And now I'm at FutureLearn, which explains some of our pink branding along the top. But if I was at a party, my quick way of explaining what I do is I'm a teacher. But because I think I can go into a bit more detail with this audience, my role at FutureLearn is learning developer. And I try to spend half my time helping the partners that we have. So we have about 120 global partners. Many are in the UK because of the excellent higher education that the UK has. And I promote this idea of a development process that encourages learning design, encourages thinking through your ideas and developing hopefully great courses as a result. And I also work with our development teams who have specific different focuses. But my aim is to try and enhance the educational offering that we have via our platform. But I do tend to say FutureLearn is more than the platform. It tends to be a combination of our partnerships and the way we do things and our philosophy and pedagogy and everything else. However, as I'm sure you know, if you work with educators that you can only go so far as an individual. And often you get all these fantastic ideas but often that they they tend to come with a very limited amount of time that you can give these people before they have to go back into their world and carry on with the things that they want to do. And my analogy there is it's a bit like a black hole in that once they're gone, they're truly disappeared again until you get to see them come back out again, if at all. So what you do with an educator when you have that moment of interaction, you have to think about something that works very well and allows them to remain autonomous and have a sense of agency, which means that they can go back off and carry on doing the things that they want to do. But hopefully slightly better having interacted with you. In my role, I focus on two specific areas where I feel that this can have a really maximized effect, which is looking at learning design and then looking at course evaluation. So broadly, if you design a course well, hopefully when we come out and looking at it from having it, from it running, from the course having completed, that we can see what works well, what needs to be improved, and a thorough evaluation can help draw out great ideas that if you look at what we call the kind of course lifecycle, it can go back into the next run of a course or the next idea, the next round of planning that you have. And these are the areas that I focus on for future learning and the areas that are most interesting to me in terms of making significant change. And hopefully, as you know, but not everyone knows that good design hopefully leads to not building crappy products. And this is true in education, this is true in car manufacturing, this is true in dietary requirements, whatever. Like if you design something and then go and do it, it's probably better than just racing ahead and doing what you think was a great idea. Design generally leads to better outputs. And as again, as you know, if you're working with people and you're thinking institutionally about problems or challenges or new barriers that you want to reduce and try to improve things, that as one person there's only so much you can give an individual. And so we have to consider interventions like workshops or like touch oversight or the way of looking at a sample and saying if you've done this and it's great, please just go away and do more. And also looking at research and contributing to research or contributing to conferences like this to try and share the things that are working well with everyone else so that you can be there to support someone but ideally let them continue on their own path. So in terms of learning design, I'm drawing quite heavily on work from Dana Laurelard where she has defined these six learning types. And their aim is to try and define and there are very loose definitions, define something that the learner will be doing. And these six learning types yet to be thoroughly debated and thrown into the bin are a way of categorizing many other things that you can build into a course that the learner will actually do. So if you pick up on the word produce, that could be that they could write an assignment, it could be that they make a video, it could be that they're standing on a stage like I am right now, or it could be something very minuscule like a forum post could be a form of production. It's very much completely designed to be open to interpretation. This is for a learning design to give an indication of what might be happening. And it's a way of getting an educator to consider more than the top one, which is if I lecture to you, you'll be fine, which is obviously as we know not the case you need to get learners to do things, which the other five tend to be more activity focused. But if you take those learning types and add them to, and I think this is again hard to argue with, the Bloom's taxonomy, which is effectively a level of complexity and comprehension and cognitive effort, the bottom level stuff actually tends to align quite closely with watching, reading and listening. But as you get up the triangle, it gets harder. So you're asking your learners to do more complicated things. But the idea there is obviously you're designing something that encourages critical thinking and evaluation and reflection synthesis of ideas. And that's harder. So therefore that should be a good thing for learners who are trying to overcome new challenges and make new cognitive connections in their brain. And a way that I see any way of achieving that is to consider learning types and thinking about actionable, measurable verbs, which describe what the learner will actually do, which effectively there's a whole lot of words up there. But the idea is they can be categorized through Bloom's. So it's a bit of a trio of ideas here. But the concept is if you have a learner to critique something or list something or describe something, then that's an action that you can define as a part of a learning design, but something that will hopefully lead to some sort of higher order thinking. So that's the sum of what I'm trying to say here. Now one method of trying to do this, going back to the idea that you're only one person and you have an army of educators that want to go off and do their own thing, is this idea of a learning design workshop. So a critical moment of intervention with a course team who want to go off and run away and do all their ideas and start producing videos and you know, just hold back for a second. The workshop is a moment to spend time with these people in a developing a course. And it's your opportunity to say, right, in this hour or two, we're going to have a moment together where we're going to get all your ideas out. We're going to then put them down broadly on paper, then you can go off and carry on with producing things. But the learning design workshop actually has an approach and it's quite structured. But the idea is it's a collaborative and engaging way of getting educators to bring all their ideas together to think about the learner and to ultimately come out of the workshop with something that looks like a high-level course outline or broadly a list of things that might go into the course in some sort of structure or order that will lead the course team to go off and then get on with producing things. I'll go back to acknowledgements later because this is borrowed from certain parts of UCL do a very similar thing. And but what I really want to talk about is the kind of educational tools that we use within the workshop, which are Diana's six learning types which are printed out onto physical cards and you arrange these cards to describe a learning activity. And then the critical bit is you flip the card over and you go into a lot more detail about what will actually be taking place in that part of the course. So if you flip over collaborate, it's not very clear, sorry, you've got lots of verbs that help explain that can be quite easy linked to learning outcomes or learning objectives. But the idea is just to get the educators to agree what might be happening at this part of the course and that's a significant achievement I think if you get educators to agree with one another. You can stand back at that point. But the workshop itself effectively creates a storyboard. So you put these cards down, you move things around, you just you discuss your ideas and your view on how it can be taught and how the learners will interact with your ideas. And then you turn the cards over and you go into a lot more detail. And that this is a kind of live cast that we ran in one of our workshops. The future learn one of our partners was demonstrating this kind of five step thing that they had created where learners are going to be watching something or discussing something and then investigating something. And the idea here is the educator is just having using a tool. These cards are just a tool for them to explain what they want the learner to do. And if you're interested in this workshop approach to learning design, there's actually going to be one tomorrow at 1 p.m. My ex-colleagues Clive and Natasha from UCL will be running a 60 minute version of the workshop. So if you want to go and really investigate this and interrogate its principles and how it works, go ahead, it's tomorrow. But what I really wanted to draw on is just the kind of the background of where all this has come from. And the final output of the workshop is very much a kind of high level view on an element of a course, a bit of a course. If you spend enough time, you can actually flesh out something like a whole master's program and several hours of educators say two or three hours you can get to the point where you have a high level overview of an entire master's program, which I think is quite an achievement given that that could be several months in the making if you did it asynchronously. So drawing back to the topic of my talk, the outcomes from a workshop like that can be very beneficial because if you look at this here, this is a set of steps within a course. There's a discussion, then they're going to watch something, then they're going to go and collaborate on something. This is a series of progressive activities that a learner is being asked to do. It's a bit structured for them to work through. But what I see is if I look at this, say at Future Learn where we have hundreds of courses that run lots and lots of times, so we can start to see patterns emerging where we think, well, actually, that's funny. If you do an assessment, learners really like to talk about it afterwards. Or if you have five assessments in a row, we tend to lose lots of learners because they don't like that. After a while, they kind of get a bit pissed off of being assessed. They want to do something else. And so these patterns start to have naturally emerged, which is kind of signal something to me to say, maybe there's more we could do here. Maybe these patterns are actually indicative of successful patterns in learning as a whole. And so if we take away maybe the topic and just look at the actual verbs being used and the learning type being used, then maybe we can take something that might look like a chunk of learning and start to say, well, actually, perhaps that can be a reusable object that someone else could adapt or just straight up use again as a part of a learning design. And I know that we need to be very wary of saying that there are formulaic approaches to education. And I'm showing this screenshot from Ken Robinson's RSA animated TED Talk around changing educational paradigms, because, of course, what we don't want to do is say there is a template for successful learning and here it is. And if everyone just follows this mechanized approach, we'll all be fine. That, of course, is not the objective here. And that's the reason for saying if you take a chunk, say, 30 minutes or a few hours of learning, there may be a pattern within that that we can reuse, not that we should be recycling the same idea over and over. So this is where I'm going to ask you to do something slightly more interactive. I was talking earlier about verbs and kind of blooms, taxonomy, organized verbs. What I'd quite like to do is there's a padlet wall that I've created that has, effectively, a couple of very basic learning designs or spaces to add some very basic learning designs. And if you've got a laptop, it'd be great if you can contribute to this. And if you're watching on YouTube even better, we'll get some remote participation as well, is I want to create a pattern library or a learning design for making a cake, which not too complicated, hopefully. But I think that what's interesting, and I'll load up Padlet in a second. So I want you to think of three verbs. And we've got a verb list here. But if you can't quite see it, there's a link in the Padlet wall as well. And they're just verbs. So go ahead and think of your own. But think of what would alternative cake be? What would cake theory look like? What would cake mechanics be? Or cake physics? Or what would a meta cake look like? Or if you said to your students, you may have never made a cake in your life. What would they need to do in order to make a cake based on these verbs? So describing the process of making or improving or criticising or debating cake theory have you? And I've got an example, but I'm going to give you five minutes. So if you've got a web browser, which I need to find now on this computer, I want you to load up the Padlet wall. And I'll load it up with you. So padlet.com. Ford slash Matt. Jenna. Oh, no. So I think it's underscore. Ford slash alt C. There's nothing here. It's because I can't spell my own name. How about that? Look at that. We've got one coming in already. So this live Padlet wall, live tech demos, never do this in front of a live audience, et cetera. There's the cake example if you want it, which is why I just kind of knocked one up last night just to make sure that you've got something to work with. And if you feel that you're not completely sure what I've asked you to do, have a look at this example on the screen in Padlet and just change something. Just change one of the verbs or change the order or just change anything you like. Quite frankly, it doesn't quite matter at this stage. The intention is just to get a couple of different perspectives on how you might do a similar task. So hopefully if I go back to Padlet, things are happening, which is always reassuring. Thank you for those that are doing things. But broadly, the idea here is again, if you're stuck, there's a link in the description for verbs is to be, we've got the kind of Bloom's taxonomy where in this example, it goes from the bottom of the triangle to the top, then a whole load of active verbs that you can use. And I'm sure you're all experts in this, but educators tend to write pretty awful learning outcomes, which is like, understand what cake is as a learning outcome. We all know that's not an outcome. That's barely even a good sentence. I think if you can say, you know, rearrange the ingredients and make it cake disaster and then try to understand why that might be, why that might have happened, that starts to give some validity because at least they've done something and they can break down the experiment slightly. Okay, awesome. This is the point where I just feel awkward. I'm going to just shut up and watch this happen. So we'll just spend a few minutes on this. Put up my live interactive timer. Remember, try to get three verbs in your Padlet contribution. Very quiet in here. I do hear keyboards, which is reassuring. Okay, let's go for one more minute just to try and get there. Nothing like a deadline, huh? All right, I'm going to have to draw it to a close just in the interest of those that are maybe getting bored. But there's something about using Padlet that's great because you just can see the live kind of hive mind of people contributing to things. And there's a lot of really great contributions here. I can see that, I said three verbs. I can see some people have definitely taken liberal application of the number three and gone really gone for it. That's very reassuring. I like that very much. I think it's probably not, I think it's obvious to say that there are so many different ways to take a very simple activity, which is to make a cake. And depending on the subject, depending on the outcome that you would like your students to achieve, depending on the approach that you want to take as an educator, depending on the kind of evidence that you would like to be presented back to other students so that they can see different perspectives and different approaches that we then start to see different, what I'm going to call patterns, emerging because the critical thing here is if you just focus on the verbs, I can see from my interests that you can see a pattern of, in my example, make, define, research, argue, and compare. That's something that could work ignoring the cake idea, just that structure of the verbs. And if I go into Padlet, let's pick on one that's easy to point to. I've got list of the ingredients, researched the costs, make the cake, taste the cake, and compare with others. Thank you for that contribution, fantastic. That's a very clear thing that I could do, but I reckon I could remix that. I hate doing live things. But I could be list the problems, research other opportunities for the problems, go and make one of those problems disappear. Okay, that's quite an achievement. And then taste, never mind. It's falling apart. But the idea is that the specific verbs listed under blooms are quite generalizable. Phrase or verbs that can be used, if I had to go back to the list, they're quite simple verbs that can be used that can be repurposed for lots of other applications. Okay, I couldn't live reuse taste, forgive me. But what we've got here is, in this example, we had design a cake experiment by maybe putting out some attributes of how the design might play out once you make the cake and describe the kind of the flavor that you might be creating if you really remix the idea of maybe taking out egg and swapping it for some other protein. Construct, so go and make that cake. And then finally, based on how it turned out, maybe revise your idea and maybe have another go. But the order there could be used for designing something, experimentation, describing what might happen, constructing it, making something, and then revising it for another iteration. The verbs can generally be reused either in that order or with a slight remix. Again, I'm gonna be really loosely fitted here. I don't think any of that really matters. The idea here is that we have an order that you could then share with someone else and they can interpret it and reuse it however they feel like. And you could do it in a completely different way. So you could assess an existing cake, recognize how you might improve it based on your own preferences, illustrate how you might do that in a diagram or a design for a new iteration, go off and do it, and then compare your outcomes with what you started with, and then propose that that might change the future of all cakes for the rest of humankind, who knows. And you may or may not do this, I think in the interest of time I won't, but the second idea was to take it to a slightly more complicated level of designing an engine for mechanical engineering where you might list all the parts in the engine and describe how they fit together, and then you might say, do you know what, we've merely missed something here, we should really experiment and try a whole new approach, and I'm gonna design that approach before we go and build it, and then I'm gonna compare theoretically about how I think that might improve the internal combustion engine, good luck by the way, and how it might perform based on my changes I made previously. But again, if you just hit the verbs of listing something, describing it, experimenting and designing and comparing, that could easily be reused in another context. Fast forward five minutes in time. It's really easy as well to stick down on the knowledge domain at the bottom of Bloom's taxonomy, which is why Bloom is a very simple mechanism to always kind of think back to. It's quite easy to ask students to kind of list and record and name and repeat what you've just told them. And it's a lot harder to do the kind of evaluation, synthesis, critical thinking, hard stuff. But equally, you can't just say, well, because this stuff is better, we'll just do more of that. You have to have a full mix of verbs that work well together, and again, there's a little part of me that says maybe computationally, there could be an algorithm that starts to say, hold up, week one of a very complicated course, maybe don't be throwing your students into something incredibly complicated, you need to ramp it up slightly, and that could be something that you could programmatically start to identify, even if it's just a kind of warning signal of learning design that starts to look like it might be overly complicated too early, or at least the educator can go, I'm cool with that, I wanna challenge my students. It's okay. So, as I was mentioning earlier, we already have known elements of a course, and particularly within the learning design stage, we have things that if we start to see warning signs early that we want to flag them up as early as possible so that no one goes off and makes a course that intentionally has a bad design, which is quite possible, and so I mentioned earlier that if a learner does an exam, one of the first things they want to do after some sort of assessment is maybe go and talk to someone, or talk to a tutor, or talk to one another, or have a moment of reflection. The last thing you want to do after an exam is go and do another exam without having any opportunity, and for a fully online course, if you have a kind of assessment window where there's just lots of assessments going on, you need to be mindful of what a learner might need in order to be prepared and mentally cleared from the previous exam and being prepared for the next exam, and these are patterns that have already emerged both from my time at UCL and also at Feature Learn in other areas as well. So, just to recap, what have we done? Well, we had a learning opportunity. We had a thing that we needed to design, and then we listed measurable verbs for learning activity, and there's several benefits to that. First of all, it gives a high-level overview of what the course might look like. Second of all, it can help write great learning outcomes or learning objectives, and that's what we've done already so far in this session. But what would we do next? Well, next we would take those very simple descriptions and we would expand them into something that looks like a design for learning. They could look like an overview of a course, and then we can go off and build the course. This isn't meant to be revolutionary, this is just to say, think about what you do before you do it. That's all I'm trying to say here. But what could happen next in terms of taking those learning designs and making something useful out of them? Well, the idea is that there might be repeatable patterns in there that actually could be reusable, and it might be that during evaluation, you could identify through data analysis, through learning analytics, through anything else, that actually there's some good in there that could be reused for another purpose, or a very similar, say, a module within a program, that module might have a pattern that could be repeated in other modules so that students have some sort of consistency within a program. But it might be that if we look at it from particular perspectives, you might see that actually a combination of learning designs, one increases retention and one decreases retention. So obviously we want to try to mitigate students just disappearing because of bad design of courses, and this might help us to achieve that. The same could be said for the level of interaction. We might be able to measure students' contributions and say, well, actually this kind of design has helped increase interaction between learners. That's a good thing, let's try to do more of that. Satisfaction, assessment scores, the list probably goes on, but these are just measurable attributes or metrics that we could use to start to evaluate against. Second of all, I love the idea that we have to share designs. I know that Lambs was, past tense, a way of doing this. They haven't updated anything since 2014, which is slightly indicative of it maybe not having a great future. And the learningdesigner.org website is still going very strong where you can produce learning designs, full learning designs, and share them publicly. But I'm trying to atomize it slightly here where I think actually you can take segments of learning designs and share those instead, which might have a slightly wider applicable usage. And lastly, the idea that you can start to inform best practice or inform template design or inform perhaps or reduce bad patterns that ever merge, that we try to say, well, actually there's research now to say that we should stay away from having lots and lots and lots of video in a row because quite clearly that's too passive and that we need to have something interactive in the middle to keep learners engaged and get them to do something before you share them a whole other bunch of media. And I think lastly, we probably don't want to get to the point where we're building teacher robots off the back of this stuff. I just put it in there because some do worry that we're just going to automate the whole thing. That black slide is indicative that I'm finished, which is a bit of an odd ending. But I'd happily take any questions or criticisms or opinions about what I've been talking about. Thank you very much. Any questions for Matt? If you just take the mic for the viewers on YouTube. Thanks for that really interesting presentation. Do very similar work in my institution. I just wanted to find out, do you just do one workshop with people? Because one of the things that we find is we get quite good engagement from staff when we do learning sign workshops. Do very similar process. Well, what we find is it's quite difficult then to go back to the teams until it's at the review stage. And sometimes it might be a bit late at that point. So I was just wondering in terms of future learning because it's obviously a bit more structured than on campus development. Do you have times where you go back to that module team or that program team and review what they've done before you see the final product? I think like any good professional, I reuse what's worked previously. And at UCL and at FutureLearn, we have a very similar approach, which is the learning design workshop is a great opportunity to have that high level interaction. And then I guess I'm stricter at FutureLearn because we can kind of slightly more enforce a process, which is that we really strongly encourage a high level course outline. And you can do it in any format you like, but we have an Excel template. And I hate Excel for that kind of work, but it's what we've got, where they just list week by week, everything that goes into the course at a very high level. And that's something that again, we can very quickly review and give them feedback on before they go off and make things. I find the biggest challenge is just holding them back and saying, don't go and produce things before you've thought about the design. And just encouraging that even the notion of design before development is one challenge and one hurdle to get over. But by looking at high level course outlines, by looking at sample content, and that could easily be a video at that stage or a perhaps a week or what we call in FutureLearn an activity. So a series of bits of learning. We can say if you do more like that, then that's a good thing. But if we start to see things that we should definitely be flagging up, we try to pull back and offer them more support. But as always, it's a question of scale. How much time can you as an individual really give to all of these different groups of people wanting to make courses? And so in a way it's about protecting your time and also maximizing the interactions that you have. But I have personally found that a high level course outline, just a descriptor of what will go into the course is a really beneficial thing that I then sell as a kind of project management tool. So I say if you make this outline and then start to put people's names against it, you can use it yourselves as a way of describing who's gonna do what during the production of the course. Which gives it a little bit of longevity rather than just saying, we have a form you've got to fill out because that can often be received negatively to say that there's a new process where the submission of the form allows you to make a course, which we're quite keen to avoid really. Yeah, no, I understand that. And I think that's what we're doing maybe I just didn't articulate it properly. It's just, I was just wondering after that now, so after you've got that high level design, do you then go back in before like the module has been or the program has been produced? Do you have formal times when you go back in or is that just developed kind of as and when with this program? So I presume you have some kind of production, you're probably maybe stricter about timelines and things and maybe sometimes. Yeah, it's true. At Futureland, we're definitely a bit stricter and we have a quality assurance window where actually we do fully review the courses before they go live and that's 30 days before they start effectively. At a university level, I think that would be harder to do, we did try to do that at UCL with our more publicly facing shorter online courses. But the window turned into more of a scrutiny of the quality and it went a bit the wrong way. So yeah, it's a really tough one to do. No, thank you, that's very useful, thank you. Thank you for that. Any other questions? So just a comment that I spoke about a similar issue yesterday in my workshop. I work for Wiley Education Services and we provide a similar service to develop courses and programs, so curriculum design workshops. And one thing I've mentioned is that there are benefits to work with third party higher education institutions, especially for someone like myself and Matt who are homegrown, who have come from higher education institutions and we care about the student journey. And these workshops are one example of the kind of things that higher education institutions can learn and apply to the face to face courses because as Matt said, the design is a special period in itself and it gives the course a real holistic perspective and the student journey can be really planned out well. It's also completely technology agnostic. This is nothing to do with future learn what I've just been talking about. It works anywhere, that's kind of the point. We have another question with the mic. Thank you. Just while we're on the topic of slightly shameless plugs, there is also a learning design network across institutional group of people, a special interest group, which is we meet I think three times a year. I just tweeted the link to Matt so it's got the obviously hashtag if anyone else is interested, you're welcome to join us. Definitely. Good plug. Good plug, yeah. Just one more check for any questions, comments. If not, we're all good. Cool. Thank you very much Matt. Thanks everyone. So we're finishing off a bit ahead of schedule as two of the first talks were canceled unfortunately. So enjoy lunch. I've been told this at the exhibition hall where we signed up initially. So see you at the next session. Thank you.