 Okay, moving on to our second speaker who is certainly no stranger to Alt. I'm delighted to introduce Diana Lore specialist. Those of you who went to Richard Noss' presentation yesterday would have seen her on video but today we're going one better thinking her in the flesh. Dana is going to talk about evaluating learning designs through formal representation of learning patterns. Thank you. Thank you very much Steve, NVP, if I can I can actually remember how to operate a PC these days, I guess that one, and it believes it's there. It's not there, is it? I'm guessing the techies will figure out how to get it up there as well in due course. The title of this is Evaluating Learning Designs through the formal representation of learning patterns. I'm speaking on this in collaboration with all my colleagues on the Learning Design Support Environment project, which is one of those which Richard Noss was talking about yesterday. The main point of this project is to try to develop something for helping teachers get to grips with learning design and indeed with the technologies which they're all very anxious to use. The problem I think for teachers is that they have not really been taken care of in terms of finding the kinds of tools, digital online tools which would help them with the process of getting to grips with what the technology now offers and what the places like Hefke and the wider world and their own institution are now demanding of them in terms of how they make use of the technology. The acronym LDSE, you could perhaps think of it in terms of let's develop something for educators for once. We've had a lot of focus on students and what students need, but actually teachers are the people who mediate between what students want and how they actually get what they want out of higher education. So they're an incredibly important part of the process of making sure that we enable students to get the best out of the technology. They've got to be engaged with it too, so in a sense I suppose what LDSE is trying to do is to figure out what it is that will make it as easy as possible for teachers to be able to discover how best to use the technology. We can't claim that we know how best to use it because it's changing and we're constantly trying to understand how best to do it. So it's much more the kind of environment which is trying to enable teachers to act rather like the research community does. Where you can build on other people's work, you can discover new things for yourself, you can share back into the community, you can collaborate, you can treat it as a problem. You're kind of problematising teaching, seeing teaching as a design process. So if we could build the kind of environment that would make that possible, that's really what we're after. It is of course a research project so that the amount of actual development we can achieve on the way towards that ambition may well be modest. But I want to focus here on the particular problem we have of enabling them to engage with what counts as a good piece of learning design. Where's the pedagogy that really makes the difference? So we can offer planning and advice and guidance and exemplars and with all these digital open educational resources available say here's something which you could make use of. But can we do a bit more than that? Can we give them something back in terms of giving them the tool which would make that design process easier? So could it also for example evaluate a design which they've made? To do that we'd need to be building on the design principles in the literature so I'm going to talk a bit about that. Making use of the learning design pattern projects which we're around is terribly important for us because if we've got those kinds of projects who can say here are some good patterns already which you could adapt into a different context or into a different disciplinary field then we're well on the way to being able to give them something which they can use as a starting point and then give back to the community. So the learning design patterns idea is very important. But if we're going to use it as an evaluative process then we've got to have something to evaluate it against. So representing all those learning theories as what we're really after here and how does your current learning design measure up to that is what I want to talk about then in terms of an evaluative framework. So if we can evaluate the pedagogy in a learning design then we've made something useful for teachers. That's the expectation, that's the ambition. And then I want to talk about the extent to which the framework which I'm suggesting actually does achieve the kind of evaluative capability that we're after. Design principles in literature there are many of them. Here's one example, very nice example recently of a paper by David Nicollum colleagues from University of Glasgow on good feedback practice. You have to help clarify what counts as good performance by specifying goals and criteria. You have to facilitate development of self-assessment, deliver high quality information and so on. They're all very good principles, unanswerable principles but they're actually quite difficult for most of us as teachers to interpret in terms of what we're doing. One thing we've got to try and do is capture what those principles are about in the kind of advice that the system offers. But it's very generic and it's true of all the design principles throughout the literature whether it's in educational psychology or it's instructional psychology or instructional design. They're developed by going to good examples of pedagogy, understanding what's there and then abstracting from it and you kind of abstract all the good and self-assessment when you've gone too far back into the generic. We end up with these very generic principles which are difficult to work with. Chickering and Gamsun's most recent version of their seven design principles quoted a lot and they're very good, developed reciprocity and co-operation among students. Use active learning techniques, give prompt feedback. Of course they're all very important but for the teacher who needs help with what I've done this with my students and I'm planning to do this. And I think I've got all that but what exactly are active learning techniques and what are the good ones and what's right for me here and now and what kind of feedback do I offer, what counts of good feedback. So those principles don't really go far enough into aiding the teacher who's really serious about trying to get some good teaching done. And our question then is can pedagogical patterns assist with designing and evaluating the quality of a learning design. Can we get a bit more crunch into that process of saying just how good is this in terms of what the theory tells us we should be doing. So learning design patterns in the way that we're thinking of using them within the RDSC is to evaluate the pedagogy in a learning design in terms of theory. So we want to take the learning design and say how good is that given what we know the theory tells us, learning theory tells us we should be trying to do it. So we're looking for pattern templates that will help with that process of evaluating pedagogy in a design. And these are somewhat interchangeable, whether we're talking about learning patterns or pedagogy patterns or pedagogy templates or learning templates. So there's a lot of this terminology around and I don't think we're completely clear about the difference between all of them, quite how they're differentiated. But I'm taking it for the moment that a learning design pattern is a learning activity sequence designed to lead to a specific learning outcome. Now sometimes we come across things which are called learning patterns which are not specifically oriented towards a learning outcome. They're oriented towards a rather more general solution in the same way as patterns we use in architecture and things like that. That way you're trying to solve a particular kind of problem. So sometimes a pattern might be used to solve a problem like how do you get everybody listening to you all at once? Or how do you get people to engage in discussion? Or how do you deal with disruptive behaviour in a classroom or there are problems that you have? And a pattern can be a solution to that problem and those are certainly in some sense they're learning patterns. But the particular kind of pattern we're interested in is those directed towards the particular kind of problem which is a particular learning outcome. So the learning design patterns we're tending to look for are those which help us with how do you get students to understand the complex system? How do you get students to understand the importance of good research skills and so on? So they're those higher level cognitive activities. So I can't go through obviously all the different kinds of patterns projects but I want to give a flavour of the kind of analysis that we're trying to do by looking at just three of them. The ICOPE, the Planet and the TELL projects and looking at the kinds of headings they use for characterising a pattern. And if you're looking for the pedagogy, those bits which are highlighted are really where the pedagogy resides. It's in the rationale to some extent, it's in the learning outcomes sometimes called problem. It's in the sequence of activities and roles that students undertake and the type of assessment and that's what in certain pattern projects they simply call a solution. The group size, the duration, the learning characteristics are all important parts of that but the pedagogic idea, the way you get students to think the right way about a particular concept or skill, high level skill they're trying to get through is not particularly dependent on the duration. You have to cut your cloth of course but the idea, the pedagogic idea, the clever way you get students to think the right way is characterised in terms between the learning outcomes and the roles and sequence of activities. So what we're looking for is where is that good pedagogy captured in this descriptor? And just looking, it's not enough to just have those kinds of headings. And what we're looking for is categories which will discriminate a bit more the form of pedagogy that's going to make the critical difference to learners. So it's really unpacking that bit which in many of the patterns project is called the solution bit or the sequence of activities and roles type of assessment. That's unpacked a bit more than the ICOPA project but that's what needs further analysis and we need a further degree of categorisation. It is there if you unpack some of those projects, this is the sort of thing which is sitting under a pattern in the ICOPA project where you've got quite a lot of information about designing the kind of task, how you group students into pairs and small teams, what you get them to do, what they have to generate, they have to tick photos of each other and upload photos and videos onto a shared site and then they have to work on those and provide annotations. So there is a lot of information there. And that's the kind of information that needs to be captured in a set of forms, in a set of sequences, in a set of types of activity which would enable us to say okay now we recognise that pattern as one that will do this kind of learning outcome. Or within the TEL project, pattern for helping the whole class benefit from individual experiences. You exercise the students and think about a particular concept from a number of different perspectives, you get them to present findings, they go through an activity which exercises a concept from a number of different perspectives and so on. So somebody has thought about how to do this, they've got a number of quite detailed activities and that's what we need to unpack and get more detail on. So I think what we need as a research field is categories within that solution or within the sequence of activities which will help the teacher to do all those things that Tricking and Gamson and Nicole and colleagues were saying, develop reciprocity and co-operation among students, how do we do that? Well here are several different patterns and different ways of achieving that kind of outcome. Use active learning techniques, well what are they? And I've just shown you several different examples of active learning techniques. Well let's foreground those so that we know more about what they are and we can pass them on. So representing learning patterns then is going to be something which is going to remain a hot topic I think. I mean there's been a lot in the literature about it already. And we've seen a number of structured text based accounts over a number of years now. The ICOPER project and the original learning designs project from Australia. We've seen the different kind of representation in the box scenario diagrams for sequences of activities which Lambs has produced, the ICOPER project again, ISIS, ECTIL has done that. That's an example of the kind of template which comes out of the ICOPER project which has all the right headings. It's got rationale, it's got the sequence of activities. But the interesting bit is in that little paragraph right at the bottom there where it says the lecturer explains the method and then it spits up the students to do this for a minute and then they do that for another couple of minutes and then they do this. That's the interesting bit of the pattern. So these are the headings of rationale and group side and all the rest of it. Well fine but how do we capture that bit? And the learning design project from many from people in Sydney captured this kind of central, well they divide things into resources, tools and supports and the central idea is that you explain, describe and apply. And that's a pattern which is then replicated in a number of different ways but the really interesting bit is the pages and pages of text which sit under each of those in a number of different contexts. Again it's predominantly a kind of text based analysis. The diagram based approach which is what Lambs operates in their authoring tool allows the drag and drop process where you can drag a particular activity onto the screen there and then link them up. So you're linking a number of computationally interpretable activities in a sequence of activities which the system then runs. So this is a very straightforward representation of a learning design which when run with students will first of all take them to a notice board and then take them to a group activity and then take them to a forum where they can chat and write things. So it's a very straightforward description of a sequence of activities but it's essentially a box and arrow diagram or this kind of representation of a pattern for the jigsaw method which is the way in which the ICOPA project represents the jigsaw method. Well you have groups which are made up of people with red skills, yellow skills and blue skills and then you reorganize them so that each group has got one of each kind of skill and basically what the jigsaw method is. So you've got a very efficient representation of it which could be, it doesn't happen to me at the moment but it could potentially be interpretable again. So with computationally interpretable activities of that kind then potentially a learning design support environment could say okay we can look at what you're planning, we can interpret that and we can say this matches that kind of pattern or expectation or learning theory and doesn't match that one, potentially. So those patterns projects which are giving us activities which are computationally interpretable, computationally defined are going to be very useful. The text we can't really deal with, we can offer it as here's something which somebody has done and you might like to read through it and think about how you'll do it for yourself, fine. But it doesn't go quite far enough for the kind of support that I think teachers struggling with this would most appreciate and under such pressure they don't have time to read yards and yards of material, it's really got something which can help them do it reasonably quickly. So what we want to do then is to find those kinds of projects which can help us by saying here's a way of describing a learning design in a computationally interpretable way. Then potentially we can evaluate that against the theory, but how do we do that? Well within the RDSC project we're using the conversational framework because since I developed it some time ago from an analysis of what's in the literature about what students need. This is a good place for us to start because it's something that is rich enough to start grappling with. But essentially this could be any kind of learning theory framework which could be put into the system. The idea of it was to create a representation of what it takes to learn in education. So taking everything that everybody's ever said about what you need to provide for your students and trying to say what do we know, how minimally can we describe that. So it's based on the main design principles for learning and pedagogy which is drawn from the literature. So it ought to be able to act as a test for a learning design in terms of which aspects of the framework it covers. So what is it? Well a quick way of see a transition problem here from one system to another, never mind it's not pretty but it's still said what it is. So this is representing a learner having a conversation with themselves at the level of concept, learner concepts at the top and learner practice at the bottom. So you have an iterative process of dialoguing with yourself about your thinking and equally your planning of adapting your actions on the basis of your thinking and reflecting on what happens in order to change your concept. So that's a sort of an internal learning cycle if you like. But the learner is helped in doing not just that kind of thinking but is also helped by what the teacher and the teacher's concepts can supply through reading and listening, lectures, books, websites, whatever it happens to be, digital resources of various kinds. But they also act by asking questions and they will get guidance as another way of being supported and they will also produce something, essays or diagrams or models or whatever it happens to be. They will also get help from other learners so others concepts will come in through peer group exercises where they're exchanging ideas and it's not just listening to other people's ideas but also the active process of articulating your own ideas and then going around that loop as well. But there will also be other people's productions, what they've done on a particular kind of exercise so you can learn by looking at what someone else is doing, learning through imitation in a sense. And not only that but by preparing your own outputs to exchange and then see what other people make of that, how they compare so that's another kind of way in which the learner can learn from others. And then the teacher can also set up a practice environment and this of course is very common in all forms, you set up a laboratory process or you send people on a field trip or you give them a set of problem based exercises or you give them an online simulation or model and the learner is working to a goal, acting to achieve that goal, getting feedback from the environment and then adapting and reflecting and revising again. So you've got a number of iterative cycles which represent a variety of different ways in which the learner is learning. Traditionally within education we call them things like learning through tuition, through discussion, through imitation, through practice. But they can also represent these kinds of principles that we've seen from, these are the Chickering and Gamson principles, encourage context between students and faculty, yes that's certainly there. Developing reciprocity and cooperation among students, yes that's certainly there. But they're there with a bit more specificity than they are in the general principles but at least it certainly is compatible. It's also, since it's derived from what's in the literature, it represents social constructivism for example by showing that the highlighted things here are what social constructivism offers. Did that actually, yes you can see that there's some highlighted and some grade out. So the teacher's role in lecturing and giving stuff and answering questions and so on is less prominent. Social constructivists of course will expect students to be probably going to lecturers and reading books but what the focus is is about how you're interacting with other people and how you're sharing ideas and outputs with them. There is also instructivism where the focus is much more on what the teacher does and how they set things up for the student. The student isn't left alone to reflect and adapt their actually given guidance so you're emphasising a different aspect of the framework that way. Another way of seeing it is representing situated learning where the focus here is much more on that practice environment and making sure that the teachers are setting up the kind of environment in which the student can play and reflect and share with others again. And in technology terms of course we've got all kinds of technologies which operate in those different kinds of ways in either acting like a teacher or enabling students to debate with each other or enabling students to represent their own content and ideas and share those or providing a kind of natural authentic immersive environment in which they play and discover. So the technology is representable in all those different ways as well. So if we want to evaluate a learning design what we've got to be able to do is to link each of the LANs activities I was talking about earlier or some similar computational representation of a learning design. We've got to be able to link those into some part of the framework and if we can do that then the framework can say okay you've got those activities that's this part of the framework so you're pretty good on the social constructivism end and not so good on the situated learning end or something like that. So if we've got this kind of representation of a particular learning design let's say to understand the process of the system and you're taking students through a number of different things. Let's expand those activities so this is the way within LANs at the moment we can represent a series of student activities and we can map those onto the different parts of the conversational framework so that we can know the extent to which if we've got all those activities embedded in our learning design then we've got a fair amount of coverage of the conversational framework. The bits that we don't have are the bottom left hand corner and I'll come back to that so let's take that particular set of activities and there are things like Q&A and boating and chat and scribe and those are expressed on the conversational framework like that so it's in that sense that potentially we could computationally evaluate a learning design against a theoretical framework so it's pretty good on the top part of it you know you've got your discussion learning going very well but there's not much opportunity for students to share their ideas with each other so the social constructivism isn't going to work in the same way as it would if you had students sharing their outputs with each other. It's that kind of analysis it would be able to do. Now of course it can't direct you the very best it could do is give you some sort of feedback of that kind. So the system could potentially interpret your design as scoring a certain number out of the possible activities. It could check the number of iterations you've got because lands will actually record whether this activity is repeated and students come back to it or not. It would record time spent on each task and it could tell you that the sequence is good for awareness but it's less good for understanding because you've got all that awareness opportunity on the top but the understanding really comes from the active learning which is more at the bottom level. So for evaluating the quality of the design we could do something to say maybe you'd like to look at this bit again and therefore improve it by proposing that you could add in design patterns that complete other parts of the cycle. So the system should potentially be able to know there's another bit which fits in here and checking that the sequence follows motivating cycles of iteration. So I'm hoping Steve that's all right for me to take two more minutes. Is that okay? All right. So testing the framework we can differentiate between sequences in terms of coverage and amount of iteration and effective ordering activities but to what extent will that kind of analysis recognize the relative effectiveness of the kinds of learning designs which have been identified in the literature. So now I'm going back to the literature to look for a challenging test of this kind of approach. And a nice piece of work by Schwartz and Bramford some time ago now was looking at two different patterns for helping students get to the point in the teaching of the psychology of memory to the point where they could predict accurately the outcomes for a hypothetical experiment. And their patterns I've been able to shoehorn into these kinds of learning activities which are represented on the conversational framework. One pattern gave students some contrasting cases of data from memory experiments. Got them to analyze the data and then produce graphs then they went along to a lecture then they had to go and do a test where they were given multiple choice questions on the factual understanding and they had to predict the outcome of a hypothetical experiment. And the outcomes of that were that they did very well on both of those tests. An alternative pattern done with a control group was to give them the same theory and same examples in a text which they read and then had to make a summary of the text. They spent the same amount of time on both. They also went to the same lecture, they also did the same test. They again got very good factual understanding through the multiple choice question, much lower transfer to the prediction. And the number of that pedagogy, the essence of it, is not on the left hand side in the particular kinds of activities. It's in the way in which that contrasting cases text was enabling them to analyze the data to think about producing the graphs and to go through the kind of processing that made them ready to go. They were ready for the lecture in a way that the people who just summarized the text were not. Now that's a nice clear pedagogical analysis of the kind that we ought to be able to say, this one's better than that. And looking at those internal relations with the activities is what we need to do. So I've highlighted there how the conversational framework can represent that and it's not quite good enough because we can't differentiate between what it takes to do summarizing and investigating at that level. I think we need more than that. So we've got to be able to help the system to differentiate between, for example, producing graphs and producing a mere summary of a text. Between providing the text in the practice environment and providing data for them to work with. And that practice environment is where those open education resources are sitting. All those digital libraries, how is that being used? How does that make the kind of practice environment which is actually helpful to students learning? I think what we've got to do is to try to find what's going to count as representation of good pedagogy. And one of our problems is that in the lamb's environment, that bottom left hand bit is not represented, is not meant to be represented. I mean it can't do it, it's not that kind of system. But that's where all of those learning technologists who are creating specialized environments for people to learn particular types of things are going to be important. All those reusable learning objects which tackle a particular kind of learning problem. So to summarize, the design principles in the literature I think are really written as being too general to be helpful. We've got to crunch down further than that. The learning design patterns are missing the categorization of the critical bits of the pedagogy. Representing learning theories as a framework for design, I think that's the sort of thing that makes it possible for us to theorize about the computational activities, the computationally represented activities that we're designing. We need to be able to map from whatever framework you've got to lamb's and I've shown how the conversational framework can do that. But I think at the level of evaluative capability we've still got some way to go because we haven't yet captured all of the key differences in effectiveness. So finally, finally it's a message for I think learning technologists. There's been a nice debate in the York community recently about what a learning technologist is. Well I think they're the people who create these things. We need those user editable patterns for activity sequences and that's the kind of thing which lamb's has made possible and which other systems like it make possible so that teachers then can be the kinds of people who can work like researchers. But we also need these customizable programmed patterns for activities. It's the kind of thing which the Glotall is developing with the RLO settle at London Met which is where Tom Boyle and his team are part of what we're doing. NetLogo would be another example of that. It's the practice environment which enables students to get a different kind of pedagogy than we've been able to describe in the other systems. Thank you Steve, sorry. Gosh Diana, thank you very much for that. Before I test you on the content of any quick questions, we've got one for maybe one or two. Josie. Microphone coming. Thanks Diana, fascinating as ever. But I've got, mine's a quick question so it's a little query because I got a little bit lost when you got to your last example doing that analysis because if you flip back, can you flip back your slides to where you had the first analysis you did. So you're looking here, next forward. Okay, so if you'd asked me to do this and I was using your diagram, I wouldn't have put the summarizing activity down at that bottom level. I would have put it at the top because it's not transformational. Is that the point you were making simply or is there something there that I'm missing? Well, I think it most logically goes at the top level because it's merely a kind of re-representation of what's there in the text. So yes, I agree with you, it is that. But that's so often what people think of as being the task which we're setting students to do. So I'm not too sure exactly how it gets in there. But when you think about inquiry based learning for example, which is working with texts and you're doing things with text and doing some kind of analysis of it, then that's an activity which is meant to be transformational. And it's not very dissimilar from summarizing sometimes. So it's a fair question. One more quick question of man of blue shift that is coming. Thank you very much, I really enjoyed your presentation. That's my first time at Alton, it's a real pleasure to hear your ideas. I've been influenced quite a bit by the work of Rob Foushey at Playtoe Learning. It's the Playtoe that came out in the 1960s and is still making computer based instruction. Rob Foushey emphasizes a lot of the work of Gagnier and cognitive psychology in different, again categories as you described, with declarative knowledge, conceptual knowledge, procedural knowledge, and then working your way up to ill structured problems and structured problems. As I try to apply that model and models like yours, a challenge I frequently face is a strong pattern for people to want to wander toward problem solving instead of spending adequate time on the lower forms of knowledge, the boring forms of knowledge actually. And I'm wondering how your model addresses this or your personal thoughts on that. Thank you. I think it's, much as anything it could be a curriculum question you're asking because this is not about curriculum I think you're asking about how do you progress people through from simpler ways of tackling the topic or representing the topic to more complex ones. So I think it could be a curriculum question and that's not a curriculum based diagram but we must talk further. Okay at that point I must ask you to thank both our speakers for a great session. Thank you very much Steve, thank you.