 Thank you. So thank you everybody for joining me today. As Igor said, I'm Leanne Perriman. I'm a senior lecturer at the Institute of Education and Technology at the OU in the UK. Now we're living through times when we're bombarded with false information, information that can influence people's behaviors in ways that cost lives at the moment. And critically evaluating that information is really difficult as the flow of defection is relentless. But there are other areas in which we can use evaluation to help us cope with these difficult circumstances. For example, evaluating what works in education is more important now than ever, especially for people who've had to take their teaching online at great speed. Now we're 11 months into the pandemic, and it's a good time to take stock, to find out what's working and what's not working, in which context and for whom. As the basis for planning, teaching and learning strategies are adaptable, resilient, effective and sustainable. And this in turn requires robust but flexible evaluation approaches that can accommodate the diversity of context in which our students live and study. Now last year I evaluated the long-term impact of the Commonwealth of Learning and Athabasca University co-produced course Telmuc, which is intended to provide an accessible learning opportunity to teachers, particularly in developing countries, to expand upon their knowledge and skills regarding the use of technology in teaching and learning. My evaluation explored the short-term, intermediate and long-term impact of the course on participants who'd studied the course over the previous three years and also on other stakeholders. Now I needed an evaluation framework that could do several things, that could identify the diverse needs of multiple stakeholders, that could allow for consideration of a wide range of contextual factors that may enable or limit impact. It would allow for analysis of complex mechanisms of change, a framework that would allow for consideration of multiple interpretations of causality, framework that would allow for both quantitative and qualitative evidence to be used as support for the evaluation, and also a framework that would allow the flexibility for iterative refinement in light of emergent finding and also form multiple stakeholders in very diverse settings. And drawing on the existing use of the theory of change approach in other disciplines, I adapted the approach to be suitable for a course impact evaluation and to allow these questions to be answered. Now you don't need to worry about the detail on these slides because they'll be available to download and look at closely. The first stage in the process is to develop a logic model and that details the inputs, so that's the resources required to produce the course, the activities, the various components of a course. Now for Telmuc that comprised three things, the facilitated course, the course was openly licensed resources, and the course is a massive network. Then the outputs and the outcomes, and also the longer term impact on participants and on other stakeholders, and you'll be able to see the details of this as I say in my downloadable slides. Now the logic model was the basis for a more comprehensive impact-based theory of change, which is shown here. The outcomes are in the lower level, and the intermediate and long-term impact is at the upper level, the top of the slide. And a key element of the theory of change are the impact pathways before here. The impact pathways are showing where, how impact is achieved in the short-term, medium-term, and long-term. They're structured around hypotheses, grounded in relevant theory about how people learn and relevant literature, including other evaluation studies. And the impact pathways are produced both speculatively in draft form before the evaluation is conducted, and this is a basis for developing the evaluation criteria for identifying the variables to be explored for selecting appropriate data collection methods. And then they are refined on the basis of the collected data, once the data has been collected. And I'll take through a couple of the impact pathways in detail as an example of how they're built up. Now, here's the hypothesis for the first impact pathway. Telmuc participants make changes in their own practice as a direct result of their study of the course, and any contributory factors, leading to longer-term impact on learners and on society more generally. Now, here's the mechanisms of change for that impact pathway, going from the activities to the outcomes, the impact on learners' attitudes, knowledge and behavior, and through to the phases of impact. Now, if we unpick this a bit, you'll see that only of the two activities, right at the bottom of the screen, feature for this impact pathway. That's Telmuc as a facilitated course and as a network. And the outcomes layer identifies a range of knowledge, skills, attitude changes, and behavior changes that are hypothesized to take place for participants, including more experimentation with Telm, more use of where we are, and increased reflective practice. The theory of change shows how the knowledge and skills, attitude changes, and behavior changes elements are related to each other, and ultimately how they may lead to long-term impact. Here, that's improved outcomes for the students taught by the Telmuc participants, social impact, and increased tel-implementation capacity. Going on to the second impact pathway, the hypothesis here was that Telmuc participants share knowledge, skills, and resources with their colleagues, who are also influenced by participants change in practice, and this leads to practice changes for those colleagues, and then to subsequent longer-term impact on learners and society. Now, the second impact pathway has more going on in the intermediate midterm impact layer, whereby course participants are sharing their knowledge and skills with their colleagues in the short term, and also sharing openly licensed course resources with their colleagues. Their colleagues then learn from participants' practice and experiment with the implementation of Telmuc themselves, and then this in turn is hypothesized to lead to long-term impact on learners, to wider social impact, and to improve capacity for implementing Telmuc. Now, I'll briefly cover the other two impact pathways, and you can look at them in much more detail in the slides and the supporting documents. So, here's the hypothesis of third impact pathway, short and sweet, Telmuc participants' influence institution leaders leading to institution-wide policy and strategy change and long-term impact on learners and society. Now, looking at the mechanisms of change impact pathway three, it features all three activities at the bottom, and includes institutional policy and strategy change at the intermediate midterm impact stage. Moving on to the fourth impact pathway, which is quite a long hypothesis, that Telmuc participants' learning is enhanced by being part of a massive cohort of mid-learners functioning as a community of practice. They gain networking experience and skills and make connections that last beyond their study of the course and a source of peer support as they experiment with the application of their newly gained skills and knowledge to their own practice. Now, that hypothesis is informed by existing theory on communities of practice and also networking theory. The impact pathway is here in detail and you'll see in the midterm impact layer there's a stage where distributed peers outside the participants' home institutions are influenced by them and their practices change or they study Telmuc and the long-term impact remains the same. So, going back to the main theory of change, on the right-hand side of the big theory change I showed you earlier are the assumptions. Now, these are things that are assumed to be true for the hypothesized impact to be realized and which are a risk to achieving impact where an assumption is not true and they'll be produced in draft form before an evaluation informed by existing literature and theory and by the logic model and then the assumptions will be finalized on the basis of the collected data and there are assumptions for each level of impact. So, for the Telmuc evaluation these assumptions include the things like the availability of technology and internet connectivity, infrastructure issues, institutional culture and support, educated skills and capacity and also external factors such as online safety, culture-related constraints on women's online participation which emerge from the evaluation and you can see all of these assumptions on the full theory of change that I'll upload with my slides. Now, on the left-hand side of the theory of change there are three sets of contributory factors and they're hypothesized to account for some of the impact identified in the evaluation and it draws on the principles of contribution analysis which look not only at the hypothesized impact from a particular interventional initiative but acknowledges that other factors will be contributing so it doesn't see impact as an end in itself but all the other sorts of things that could contribute to them being achieved and for massive open online courses where learners are in very diverse contexts that could be a multiplicity of factors, it could be so many different things because the context will be so diverse and that's why this sort of framework allows us to really dig deep into what's going on in specific contexts. So these contributory factors are hypothesized in the initial theory of change of the basis for designing the data collecting strategy and then they're revised on the basis of the collected data. So contributory factors for impact for Telmo included other professional developments that course participants have done, availability of resources, institutional support and for long-term impact contributory factors could be things like school attendance, family influences on children at schools or by the whether participants are teaching and also community support they can also contribute to the impact being achieved. So here's the overall theory of change again and it will be done loadable from my session page. A theory of change approach can be used to evaluate any type of educational intervention where you want to find out about different stages of impact this could be just short-term impact it could be impact that that's actually an outcome happening while a course is underway and also where you want to find out about the contribution of other factors in achieving or preventing that impact and especially how a course or intervention works or doesn't work in different contexts and this is especially useful useful because depending on the data collected a separate theory of change can be produced for individual countries for individual sectors even for individual case study participants to allow for comparisons to be made between series of change to look at how the mechanisms change for it for individuals the sectors and for context and a theory of change can also be combined with case studies of individual learners experiences and I use this approach in the Telmooc evaluation which you can see in the final impact report and I'll include a link to that on my session page and I realize there's sort of a lot to cover here but I wanted to give a sense of how a theory of change can be used to not only a structure and evaluation but to make sure all of the factors involved in teaching and learning and in achieving impact get considered and especially to allow comparison over the first context because then not only the methods of an evaluation but also the the findings can be of use to people in other diverse contexts and they can compare their own their own contexts with those featuring in an evaluation as a basis for making decisions about you know how to further develop practice how to implement new initiatives and I think it's going to be continually important in the sorts of months we don't know how far ahead as we still navigate how to manage delivering education through a global pandemic so I hope you found this presentation to be useful do look at the supporting resources and thank you for listening thank you very much Leanne I'm much appreciated for sharing this this rather comprehensive theory of change framework that you've developed I think that participants here also will need a little bit more time to to help them the digestion of the the complexity of the framework itself but it's it's it's really useful I think to think about it and as you say like it could be used to evaluate the impact of sort of different educational initiatives right so not specifically on just remotes and there is there is a question from Andy Lane here in the chat window is it it is you is it usual for a project to set out the theory of change before they start against which progress can be evaluated but here you are developing one retrospectively how is it informed or validated by those who developed and and the project okay thank you Andy thank you yeah so there was an initial theory of change that had been used for for the course itself and I picked up on that and used that as the basis for then building a a course related theory of change and to expand on that to put to produce impact so yeah there was an initial theory of change to start with and I built on that as the basis for developing an impact related to theory of change that would have formed the data collection so I mean the thing about theory of change is that it can be a multi-stage process if it's best to start it as early as possible and I personally wasn't involved in the production of the Telmuc but fortunately had had something to go on as the basis for identifying the intended outcomes and also there had been several of evaluation reports covering the running of the course so what was happening within the course and learners experiences and also looking at the content and the facilitation so they also informed my subsequent impact evaluation related theory of change. Thank you for the clarification Ian I'm not sure if there are still additional questions if there are we still have another mini.2 left for questions and in the meantime well and Paula's comment here very interesting and complex framework so in terms of its practical practical application Ian has this been already been used to evaluate this framework itself has been used to evaluate specific initiatives as well? I'm not sure if the framework has then been used to evaluate subsequent initiatives we started using it I started using it within the open university to evaluate some of our own MOOCs although COVID-19 and our need to support multiple sectors in getting their teaching online rather sort of stopped our evaluation activities and we were just focused on other things I think that's common across everywhere really that the best plans were pretty disruptive so we're starting to use it but that's something we'll pick up once we get a bit more capacity again. Okay thank you very much and is this something that you would like to encourage the participants in this session or more broadly like to experiment with and give you feedback on? Definitely absolutely I would love it if people were to use this to experiment with it to look at different ways of of working with it to share findings I mean these are this is only useful evaluations are only really useful if the findings are shared beyond an institution but beyond the setting that's being evaluated because there's so much that can be learned especially from the I think the barriers to impact that are realised by diverse people in diverse contexts and we can only find out how about those barriers and how they're being managed by sharing by sharing findings so yet use it share the share the outputs keep the discussion going about this. Great thank you very much so I think that concludes our time for this for this specific session thank you very much Leanne again for sharing. I would like to encourage you to put the link to this to your slides or any additional supporting materials as far as the framework is concerned directly under your session description and I'll just post the link there.