 Ko Elizabeth Pokoa, Pokoa ingoa te te noho oki te whananui a tara. I have the great privilege of introducing Jamie for our following. Jamie has recently joined Kanta Public to head up their evaluation practice in New Zealand. He has over 25 years experience in evaluation and impact assessment and specialising in international development projects for the British government. Today Jamie will reflect on his evaluation of the Newton Fund and how a theory of change can be, say, an advantage or also a disadvantage. But I'll let you talk about the Newton Fund and what reflections or theories of change. So to you, thank you, Jamie. Thanks Elizabeth. I think 25 years experience just means I'm old, right? Well good morning Australia, good afternoon New Zealand. Thank you very much for having me. Delighted to talk to you today. Have you got the slides? Absolutely. Hopefully you'll see those shortly. And we... There we go. So yes, I was just going to... Oh yeah. Perfect, thank you. So yes, as Elizabeth said, I was going to talk today mostly around the theory of change and largely based on my experiences on a project called the Newton Fund which was a largely UK funded programme that I worked on for a number of years before I made the move down to New Zealand, which was only a few months ago. So I'm hoping... Well it's a UK project, I think some of the things I'm going to talk about today are transferable in terms of some of the learning. It's less about the project itself, it's about how we use a theory of change and how others use a theory of change. So yeah, if you just click to the next slide. Just in general over here what I'm going to talk about today. I'm sorry, here we are. Ah, there we go, thank you. So yeah, I was just going to talk a little bit, just a bit of introduction about theories of change and why we as evaluators tend to love theories of change. But then I'll also start thinking about well, hang on, what's the broader purpose of these? And sometimes the fundamental thing I want to talk about today is how an evaluation focus can make a theory of change less useful perhaps. And I'm sort of building the whole discussion around basically a case study of an evaluation I worked on in the UK called the Newton Fund and I'll explain a bit more about what that fund was and what the evaluation focused on. So yeah, and this was, as I say, I'm new to New Zealand. I've spent the last 20 odd years doing evaluation in the UK and the last eight years doing international development evaluations. And this was one of these that was funded through overseas development assistance money, which had some additional challenges. But again, I'm not going to get into that. And I'm not really going to talk about the evaluation findings themselves. I have some questions at the end around that. But I'm really just talking around this notion around theory of change is a bit of a think piece as to how useful or not they might be. And I don't know, evaluation humour, I love this cartoon. It's one of my favourites. How do we think about things and how do sometimes our clients think about things in terms of how change happens? So yeah, if you click to the next slide, please. Oh, that's backwards. There we go. There we go. So yeah, this was just a little bit of history. Some of you may well be familiar with this. And theory of change does have its roots in evaluation, really. And it came about through some of these aspen roundtables back in the mid-90s. And it was Carol Weiss who gave the first definition, really, of what a theory of change was. I mean, I won't read this out. But again, essentially, her hypothesis was that if all your stakeholders are unclear about the process of change, about what you're trying to achieve through a programme, then it becomes very hard to then evaluate and understand what it has achieved. And that was the notion of these sort of mini steps that lead to a long-term goal. And this is a transform then into the theory of change that sets out these logical steps, the causal pathway, call it what you like in terms of how the various activities might ultimately lead to what we would understand now as outcomes and impacts. So that's kind of where it started. And I guess now we're all very familiar with theories of change and use them in our work. And if you click to the next slide, this was my kind of very loose summary of why evaluators now love theories of change. And fundamentally, if we don't know what it is that the funders and implementers of these projects or programmes are trying to achieve, or how they're trying to achieve that, then we're a bit lost. Are we going to evaluate the right things? And how are we going to do that? And we need them to understand the rationale behind the focus. So then to evaluate whether it makes a difference or not. And I originally worried that was what worked or not and there's a whole debate around that. But let's not get into that. But we need to understand that so that we can focus the evaluation really because sometimes you can't evaluate everything. You need to focus on what's important. And the theory of change helps us do that. And that's my final point there really because there is this theory to begin with. We can explore around very loose terminology but to try and understand why a theory that looks great on paper maybe doesn't work in practice or maybe there are some nuances to it that weren't considered in the theory or various underlying assumptions that whether articulated or not had a real impact on whether a programme was successful or not. So, a theory of change can be really useful. It can help guide and focus on evaluation. It helps us to decide what do we need to collect data on, where do we need to focus our efforts in terms of implementing an evaluation and what sort of tools we might use. And it can also help understand the context. And I think that's an important part as well. What is the wider context in which an intervention is taking place and what are some of the maybe the external factors that might influence or impact on whether a programme is ultimately successful or not. So, we love them. Sorry to click to the next slide. The question for today is, well, okay, we as evaluators love these, but how useful are they for the funders of these programmes, the implementers of the programmes so people are actually on the ground delivering them. And in theory, I guess they should be useful. Particularly in cases where funders haven't thought through this logic and that was very much the case in the example I'm going to talk about today. So, it did help sort of unpack that a bit and understand that process of change, those causal pathways. But there's also sometimes misunderstood and I've seen oftentimes theories of change or something that's presented as a theory of change. I wouldn't consider it to be that. It's a results framework, a strategic framework, a log frame, I don't know. We have all these terminology for different things, but to me they're not often theories of change even though my client, the funder, might think that it is and might not mean that to me as an evaluator. And the fundamental question for you is, do they use these theory of change? I think they're most often produced by evaluators or in the process of designing an evaluation. The funders might do that themselves. But do they use it? Do they reflect on it? Do they update it? Because my views, the use of change shouldn't be static things. You don't have a theory of change at the start of a programme and then not update it and refine it as time goes on because inevitably things change, projects change, the context changes. It should always be living documents, in my view, rather than fixed, rigid constructs that don't change from the beginning to an end of a programme. I guess what I'm going to talk about today is how do we make sure they're useful and I'm basing that around this example of the Newton Funds, which is a programme in the UK. Sorry, I can't see the chat, but I noticed Penny Hawkins popped up on the attendee list. Penny was actually part of the... What do we call it now? It was the Evaluation Steering Group or something. So it was a cross-governmental group in the UK that actually came together to advise on the design of this evaluation. So it's nice to connect to Penny, a small world. So you'll be familiar with some of this, having been involved in those early stages of the programme. So if you click the next slide, I'm just going to get some brief background on the Newton Fund because it's important context to where we ended up with this theory of change. So I led the design and implementation of this. Obviously, working with a team, I worked with a company called Tetra Tech at the time in the UK. It was a longitudinal evaluation. It was originally planned over five years. Interestingly, it was commissioned one year after implementation had already started. So lots of things were already underway before evaluation was commissioned. We ended up adopting a theory-based approach. I'm not going to go into why we did that or what the evaluation design was. I really just want to focus on the theory of change aspect for today, but there were some various reasons why we landed on a theory-based approach for the evaluation. And as ever, I guess as consultants, we've probably always accused of this. Ambitious scope and limited budget always a challenge for evaluators. But I think maybe you'll see, I guess complexity was one of the main challenges with this and I'll demonstrate that a bit further in the next couple of slides just as a bit of context. So fundamentally, what was the Newton Fund all about? Well, ultimately, it had this goal of supporting economic development and social welfare. And it was all about tackling global challenges. And this was all in the context of overseas development assistance. So it was very much that aid budget that was funding this programme and that placed certain limitations on how that money could be spent and also where to some extent. So the UK, we ran around £700 million to fund this programme. And it was intended that that would be co-fundate with various partner countries that they were going to work with to do this. And it was all about building research and innovation partnerships across the globe to tackle some of these development issues and to develop skills and talent in research and innovation. And it was structured around these three pillars of activity. People, research and translation. The people pillar was well, kind of, as his name suggests it was all about capacity building. It was focused on individual researchers of providing them with capacity building funding research projects. What they called mobility schemes so sort of enabling sort of transversal people so researchers from one country would come to the UK. UK researchers would go to another country in terms of build their skills, build their knowledge through international working. Research was the middle pillar I think it was the pillar that received the most funding and this was about international research collaborations and it was intended to deliver truly collaborative research between countries again focused around tackling these global development challenges. And then there was this third pillar which they called translation which was kind of okay well what happens next after the research how do you translate that into policy? So can we use this research to translate an influence policy around some of these development challenges or indeed it could be other things too about commercialisation of products or of ideas so it was that kind of think about it as kind of a pathway between okay you can develop good research skills you can develop the good research then and that research will hopefully lead to something rather than just being good research for the sake of research was kind of the theory behind it and if you just click to the next one it was complex put it that way so the fund actually in the end operated in 18 countries when it started there were 15 some were added later on and some dropped out for various reasons there were 7 UK delivery partners so all the UK research councils and a number of others so the academies and the British Council and the Met Office were all involved as delivery partners all with their own funding envelop to deliver activities under all 3 pillars some of them worked under all pillars some of them worked just under some and similarly some of them worked in some countries and some of them not so we had this really complex program of all these countries with many different things happening in each of them by different partners and well if you can another maps quite small but very different context in which all this work was happening so you can see there are some sort of big hitters there I suppose in terms of China, South Africa, Brazil where research infrastructures were quite well developed but then you had others where they were much less well developed so depending on the what would you call it the innovation ecosystem and how developed or advanced that was the activities varied to respond to that local context and in some places where it was far less developed, Kazakhstan was one that springs to mind that was in at the start where it didn't have that same research and innovation infrastructure as China would and that necessarily reflected the way funding was being channeled into activities so some focus much more on the translation pillar others focus much more on the people pillar reflecting where perhaps the most need was in terms of bringing on those innovation structures within each of those countries and yeah the best on the side again I'm just trying to overemphasise the complexity of all this for the number of projects, the number of organisations involved in terms of actually delivering a meaningful evaluation because we were contracted to deliver a fund level evaluation so we had to kind of capture the complexity of all of this in that evaluation work and the next slide please and I guess some additional challenges for us, the objectives were not very clearly defined at the outset so I've said there on paper it was about advancing research and innovation both in the UK and all of these partner countries involved there were no targets there were no country plans so there weren't specific plans of activity for each country and there was no monitoring system or at least no standardised monitoring system to capture all these activities so we had all the partners who had slightly different systems but no common understanding of metrics that we might use for an evaluation so that was a big challenges for us at the start and I guess the other thing that became clear quite early on was that actually there was sort of an unwritten objective around this which was about advancing UK foreign policy interests and sort of using this fund in a way to open doors to other things for the UK government in some of these countries which was again kind of an interesting one and it actually kind of explains you know some of the very early work kind of question will hang up why you'd be chosen some of these countries as partners because if you were looking at ones with you know the most advanced you know innovation system they're the ones which might offer the greatest leaps forward in terms of addressing development challenges they might not be the ones you would pick but there was some of this little driver behind some of those decisions to include some countries so again that was interesting as well I mean that wasn't something we were particularly explicitly trying to evaluate whether we ended up with some useful findings around the extent to which the fund did that but because I think because it was largely politically driven programme where we started was that this logic the theory of change didn't exist it wasn't fully thought through so they had this notion about learning science research and innovation across all these countries but the idea of an intervention logic was you know it wasn't there and that's where we started so if you click to the next slide very early task for us when we commenced this work was to actually develop the theory of change and we did as you can imagine was something this complex with this many partners we tried to make that as participatory as possible we got lucky at the start because there was a sort of a global gathering of all of the in-country teams in each of these countries that they assembled in London and we were able to get on the agenda for that and actually get those teams working together to think through what is this logic and we facilitated discussions to try and unpack what is it they were doing and how does this ultimately leads to change and on top of that obviously we consulted with our client base business energy and industrial strategy as it was at the time and the management team in the UK we also did some familiarisation visits to certain countries not solely for the purpose of the theory of change but to just get our head around this whole programme and what it was doing how it was structured but also that allowed us to capture the views of because it was a partnership programme we wanted to make sure we captured the views of the local funding partners the partners in other countries where we reflected that in this design of the theory of change so yes we had various workshops and activities to try and develop this theory of change to capture this incredibly complex programme for us to then be able to come up with a plan to evaluate it and hold your breath everyone if we click to the next slide this is where we landed as our first iteration of the theory of change which as you can see is super complex and I'm not going to talk you through this so you probably can't even read it but don't worry but as you can see what it did enable us to do to the bottom you've got the three pillars and we tried to capture the vast variety of different activities that were happening there and then follow that process through to decide what are the outputs of that and the outcomes the dark boxes in the middle we were able to agree these three core outcomes that everyone agreed that the programme was working towards and then I guess perhaps one of the most enlightening parts was that going from there to the ultimate objective right at the top that impact statement around the science and innovation partnerships being strengthened there was a huge leap between these outcomes and that impact so those sort of four layers of intermediate outcomes intermediate impacts calling what you want was quite a long road so that was quite enlightening in itself in terms of what does this theory of change mean for us and then it was useless what happened shown here actually were all the assumptions so underpinning all this there were like 27 odd assumptions about that were being made through all of these pathways about what needed to be in place for this to happen and we're able to articulate all those as well which I think was of some use to the client and understanding we'll actually hear some issues we need to get a handle on and things we need to be monitoring and we tried, given this was so complex and a bit of a jumble we did try to unpack it a bit and if you clip to the next slide just in terms of the visual representation we unpacked it by each pillar and this reverses it we've got activities on the top on this one working this way down to those fund level results to try and unpack it a bit and understand what we're doing we mapped all of those onto the diagram so that was quite very useful for us to work out what are all these pathways what is supposed to lead to what and when and some of these hadn't really been articulated before so that was quite a useful exercise in itself but I think our big issue and I think if you click to the next slide was that given all of this the evaluation team I've said loved it that was maybe a bit of an exaggeration but it was certainly a really useful evaluation tool so yes, it was super complex but it did really unpack and gave a representation of these clear logical steps all of the things that needed to happen to ultimately achieve the things that the fund wanted to achieve gave us a framework to then identify how can we try to measure and track these results pathways and therefore design so appropriate evaluation approaches and tools to be able to do that we identified those assumptions some of which were untested and we tried to then build that into the evaluation design and then we ultimately used it to create this detailed evaluation framework of how are we going to evaluate this program over this five-year period so in that sense it was super useful but it was very much designed for an evaluation purpose and as you can see I mean there was a narrative that went with all this of course it's not just about the diagram but even that it was really complex for delivery partners and the client together head around it what does all this mean, what do I do with this am I supposed to do anything with this or is it just something that the evaluators do and that's kind of where we ended up oh, that's an evaluation thing so we don't need to engage with that too much so they couldn't really see the benefit of it beyond evaluation which I think and that's kind of what led us to the next stage of work which was sort of recognising that but then thinking well how can we make this more useful and so if you click to the next slide as Simon on we sort of recognised this so we had this challenge of the funders in different countries trying to work out over an extended time frame you would have new people coming on board to the program and others dropping off because it was over this five-year time period so being able to engage with this and understanding what this programme was all about was quite tricky so we sort of collectively agreed ourselves and the client we called it a refresh of the theory of change to partly to reflect some changes that happened over that period but also with a much stronger focus on the usefulness of the theory of change beyond beyond our evaluation purposes so yeah to make it more useful you know as I say that first attempt was really an evaluation tool we didn't think enough about other audiences and how they might use it particularly those partners in other countries but also the delivery partners here in the UK who you know I think the programme is this type they tend to focus on what we need to deliver the outputs right and are not necessarily thinking day to day about well what's the what's going to say what's the point of those outputs but you know what I mean in terms of how do those outputs to this broader concept of change that we're trying to achieve so we went through a very similar process to update it again it was very participatory we tried to engage as many people in that process as we could and I think if you click to the next slide we should see where we ended up in terms of this new version and visually it's a lot cleaner and again I'll come onto that in a minute it's not all about fancy diagram but actually in terms of simplifying some of the content of that previous version I mean it's fundamentally not different in terms of the story it's telling the activities the pillars haven't changed the long term outcomes haven't really changed but we just represent them and sort of group them together in a much simpler way that try to emphasise how all of this comes together as a cohesive programme rather than just a set of individual boxes and again I'm not expecting you to read all this in detail reports were all published if you google around you'll be able to find those and you'll also see at the top we mapped all our evaluation questions against different elements of this that's the little circles at the top if you can see them we mapped our evaluation questions across all of this to demonstrate what are we going to answer what can we say about the achievements of this fund overall and this and again with the narrative that went with it provided a much clearer statement I think on the rationale of the programme I can't see it by the box at the end that currently my face is covering showed the ultimate impact of the programme but it was much clearer and what we found was that our partners found this a much more useful tool they could sort of engage with this in a much clearer way they could see where they fitted into this whole network of what was happening how all those things were ultimately supposed to lead to these different impacts so as a visual tool it was certainly much more useful to do that and it was used quite extensively and the feedback we had was really positive from these partners I can finally see why I'm doing this or how it's going to lead to these things that ultimately the fund was trying to achieve they might not be the things the partners were doing directly but they were all supposed to lead towards these shared outcomes these shared vision of success so yeah so what did we learn from all this sorry if you click to the next slide just to sort of sum up so yeah I guess we're started for me the use of change are of limited value unless they're actually used by the funders and implementers of these programmes in this case as I said none of this had really been thought through in much detail when we started so this was a really useful exercise in defining these realistic long-term goals and outcomes for this programme it was also useful in agreeing what are the necessary preconditions for success you know what needs to be in place testing those assumptions articulating those assumptions in a way that perhaps hadn't been done before which was useful for then the funders and members think about well actually we need to do something about this because you know if we don't do that you know if there isn't the funding in place if there's no diffusion of knowledge that happens after someone's been on an exchange programme none of these things happen so enable them to start thinking about that and perhaps tweaking some of their programmes to try and ensure that those things took place so I think that was a positive positive outcome and then if you click to the next one I guess perhaps the biggest one for me but actually it's the process of developing the series of change and this collaborative participatory way and then anything else more important than the fancy diagram I mean it looks great but you know so what but actually engaging those partners in that process and the interaction between the different partners when you've got a programme as complex of this you've got people doing very different things in very different countries but actually getting them together to share their experiences and their challenges but then finally to be able to come together to deliver the shared outcome was really really useful and not only you know but understanding where there were some shared experiences so another big challenge with this programme is with all these things happening in different places how do you actually capture the knowledge the learning that come through all of this so sort of through this process there was an opportunity to share that learning to engage more directly and you know overcome some of these shared challenges that previously they'd been sort of thinking about in isolation so it was useful it helped us as evaluators to challenge our thinking you know how do we go about better linking the work of these programmes to the outcomes and to help people to see these routes to impact and how long it might take I mean this fund it was always envisaged that these impacts would be further down the line you know ten years down the line even though the evaluation work would have stopped long before then but this was able to sort of set that out much more clearly and understanding what are the things that need to happen after this fund was ended for any of these impacts to be visible and fundamentally for us as evaluators we've been able to explain clearly to our client, to our funders, to the partners you know what are the benefits for the UK and for the partner countries of all of these different activities so I think I've sort of rattled through some of that quick to the next slide which is my final slide I just wanted to acknowledge that obviously this wasn't just me it was a huge team effort from the team at Tetra Tech and I particularly wanted to thank Ifa Murray she really led all this work on this redesign of the theory of change but more than that she was the one sort of driving this need for the theory of change to be more useful so a really big thanks to Ifa Murray for all of that work so yes I've rattled through that I hope it was interesting it was just some reflection from that work I'd done over the past few years on that project I'm very happy to take any questions or comments from anyone I can't see the chat I'm going to see if I can pull that up now Thank you so much Jamie for this extremely illuminating presentation on the complexity of a multi-program like multi-budget global fund since we don't have any questions in the chat please everyone feel free to pop your questions in the chat or turn your hand up I'm going to ask you the first question Jamie just an indication of how difficult was it to streamline the programs of the various countries and what challenges did you face when it came to agreeing on the outcomes as well as the methodology of demonstrating value for money or cost-effectiveness since it is a budget support evaluation I'm just wondering in the different countries in the different country context how difficult was it to get agreement on the methodology Gosh there are a few questions in there so agreeing the outcomes actually on reflection I don't think that was so difficult I think the client was pretty clear in their heads anyway if they hadn't written it down if you break it down to the most simple level if we were funding research what should the outcome of that be I think the process of agreeing those core outcomes wasn't so difficult what was more difficult was understanding on the pathways of how you actually get there and then as I hinted beyond outcomes how do we get to impacts that was a much more challenging part around how that process would work given that once you're at that level it's much more outside your control there are all these other external factors and influences that impact your ability to do that sorry I'm just going to question the chat which is related as well the challenge of agreeing clear and correct outcomes it was a challenge but it was interesting actually getting everyone in the room because in a way the funder was dictating some of this so this is what we want this to achieve so getting all these country partners on board was interesting it was a challenge in some respects because of the co-funding aspect and this being this overseas development assistance money there are only certain things you're allowed to spend that money on and some of the challenges to face were some of the funding partners wanted to do other things they weren't eligible for this UK government spend and that created some challenges because they wanted to do things that just weren't eligible for funding and that created some challenges with those partners in terms of trying to agree what a success looked like for them because they wanted things that were slightly different to what the UK was able to support put it that way so again the collaborative aspect of that was critical because I think if we hadn't been able to get all these people in the rooms that we did this sort of various exercises in I'm not sure we would have got there particularly for that final version it was really important there was a shared vision of success for all of this and I think we did get there in the end but it did require spending that time that energy to actually get people in the room and talk through this together and actually understand it and ask their questions they wanted about these pathways so that was Sabine's question I've sort of segued into that but the question about value for money that was really challenging and actually towards the end of it it was the client that wanted to implement a new rubric for how they wanted to measure the value for the money of this programme and it just took such a long time and also as I mentioned the lack of monitoring data was a real problem for us we didn't have any of that data even some basic concepts like cost per output or whatever there was no standard monitoring right from the get go we're saying please, please, please can we have some standard monitoring and actually collect this data so we couldn't do some of that so the value for money was a real challenge in the end and I think our evaluation report doesn't talk much to that because it kind of went off and became its own thing this fund sort of became not merged but there was another similar fund called the Global Challenges Research Fund that they kind of then put together on the same management and wanted to introduce a new value for money framework for both of them and our evaluation didn't ultimately get into a detailed value for money assessment because it was so complicated and because we're lacking a lot of information to do that so just to answer all your questions Thank you, Jamie Absolutely, there's heaps more in the chat Todd, Todd Would you like to unmute yourself Todd and ask a question Yes, can you hear me Yes I found the mute button I appreciate the mammoth effort that you've unpacked there and getting so many different perspectives and experiences to a shared understanding and I'm just interested I suppose in terms of how to move from a simple linear model to something that it kind of reflects the reality on the ground and sometimes programs do work nice and simply and as we expect but I suppose my experience is oftentimes I don't so how do we then reflect that with the stakeholders of the reality of moving on from a simple linear model I wonder if you've got any reflections on that Yeah, I think a theory of change, the fancy diagram is always going to be an oversimplification of any program but they can be quite useful in starting that discussion and I think that's where the question is because we started with the complexity and then came back to the simple better so the lesson we can switch that around can we at least get this basic logic a basic understanding of this before we go into the complexity and then having that collaborative process to unpack that that's where those sort of exercise doing that collaborative of these workshops these sessions, some shared learning but all of that would be useful the other thing that was sort of missing here I think I mentioned at the start was the lack of a clear because the model was different in each country and at one point we were suggesting shouldn't you really have a nasty theory of change for each country because the pathways were very different in different countries and that would have been super useful as an exercise even developing that strategy for a country would have helped those working in that country to understand what are we trying to do here it would have simplified some of that funding level complexity into what are we doing in Mexico to be able to focus a bit more on what affects me rather than because some of these fund level concept for abstract at times for some people engaged in it so kind of boiling it down to that level even a very simplified version of what success looks like what we're trying to achieve in each country would have been a real benefit to this program and I think that would have helped because then if you take it down to those manageable chunks it becomes much easier then to get into that messy stuff and to see issues here rather than trying to do it at fund level I'm not sure if that really answers your question but it's just my reflections on the newton fund and how that might have gone better in this case Thank you Jamie I think you have already answered Hadil's question as well I was wondering what you think is more appropriate and useful for a theory of change of detail or a high level one Oh that's a good question Yes, more useful for who right? That was going to where it started this whole thing I think maybe the sort of simplified one can be really useful for maybe someone who's not in the evaluation world but is just trying to get their head around what's this all about, what are we trying to achieve and how are they thinking about getting there I think that simple representation can be really useful for that I think as evaluators we probably do need the more in depth the more complicated one Maybe it doesn't need to be one that's shared widely with partners and others but as evaluators we need to understand that and the funders need to understand that so I think it's horses for courses I don't think there's ever going to be a definitive answer to that one Rula is asking about navigating the unwritten objectives Rula, feel free to unmute yourself and elaborate if you wish I think it was probably an intense process to navigate that and also because it's a lot of different context from the different partners that James worked with so I'm wondering the unwritten is currently written on the screen for us but how did you capture that and how was that helpful, did it benefit the project to sit that on the table define it, measure it see what can be done next if it's useful in another project or stage, I'm just fascinated because it's already complex with written objectives let go other unwritten objectives so just if you can speak to us a little bit about that please it was a tricky one because it was never really written all of this was being published and the UK government didn't want to be advertising the fact that there were some wider foreign policy goals behind this programme so saying to partners this is what we're trying to do and our evaluation plan was not the right thing to do so they were never really written down but in the way we designed some of our consultation, some of the case study work we did in the different countries we did try to tease out some of this about what were the wider benefits of this in terms of these opening doors and leveraging other funds and I think I'm in front of me right now but the theory of change hints at some of those things about leading to these broader collaborations so they were kind of framed and wording that was a lot more vague than I've put it on the screen today that allowed us to unpack a little bit in our case study work consultations with some of the high level government officials in these countries to unpack about what else did this lead to so we were able to unpack some of that but it wasn't done in a super structured way it was, well I'm not going to say anecdotal but there was a lot of evidence we tried to gather about to what extent did it actually lead to benefits beyond anything, the Newton Fund was on paper supposed to be achieving so they never really became written although we did talk to them in more vague terms in the Evaluation Report about how this had led to other collaborations other developments between the UK and these countries Victoria asks how often did you upgrade your theory of change given that things changed in five years Yeah, we only really did it twice in this case because it was quite a long time period and it was the nature of evaluation we were in, we did a lot of baseline work and then there was a period where we didn't do anything at all right, it was kind of the nature of evaluation where there were intense periods interim and final evaluation stage so in this case we only really updated it twice so this was the one we did at the beginning the first one I showed and the one we did well before we started the the end line evaluation I think it may have been useful to think about that a bit more earlier it would have been great if again I'm sort of doing my 2020 hindsight but if we'd had that much more simplified theory of change earlier I would have loved to have seen the client take more ownership of that and have them reflect an update on it and we could have worked with them to do that but for them to take more ownership over that to reflect the way the project was evolving and emerging and expanding but yeah for us if I would we only related that twice but I think again on reflection it would be useful to do that more often but as I say it's kind of feel the ownership of some of that needs to rest with the funders the clients of some of these programs thank you that is absolutely fascinating can I ask is the fund still going say that again sorry is the fund still active is it still being implemented to the same countries with the same partners oh gosh well they were about to I'm not sure where it's at now actually because we finished our work about a year or two back and there were various discussions at that point around what happens next and it was running alongside this global challenges research fund so I think it may still be going but it might be in a slightly different form now sorry I'm not haven't kept up to speed I've been too focused on moving to New Zealand lately to get up to speed on work I was just thinking thinking of the long term outcomes it'd be wonderful to do it before and after did we get it right as I was leaving the UK it was one of the things that was being talked about Bay's the client really wanted to do this sort of post implementation impact study but I'm not sure where they got to with that but there were all sorts of funding challenges in the UK so they were talking about that before I left whether they actually managed to secure the funding for that I don't know but there were certainly an interest in doing that and having that kind of okay a few years out where are we looking now because that you know our evaluation ended we weren't really to say say anything at all about impacts and even some of those sort of higher level outcomes we were only really able to talk about progress towards those outcomes so yeah the clients are the benefit in doing something further but it was sort of outside the scope of our contract at the time we have about 10 minutes feel free to unmute yourself and ask them questions we can have an open conversation we only have the people right now it's dropping in feel free to ask more questions from Jamie and make the most of 10 minutes we have from it's a vast amount of experience and knowledge as we are working Jamie I was just wondering did you have a huge evaluation team on your side how many people were you know in terms of partners So for us delivering the evaluation you mean? Yeah well huge I wouldn't say huge so yeah Tech Check in the UK has a reasonably big evaluation team there were about 20 of us at the time they were all working on this project we also partnered with some other firms to deliver some of the specialist work on some of the innovation work that we didn't feel particularly specialised then we brought them on board but then we then had to we ended up with kind of a case study design to various these countries to inform the evaluation and doing case studies of different things that were happening and we engaged there with teams of researchers in each of those countries to help gather that information and navigate those landscapes so yeah I suppose when I think about it it was quite a big team it didn't necessarily feel like it when I was sat there in London with two or three people around me but actually the whole thing bringing it all together was quite a big team so how does link people in each country more upskilled in terms of change and evaluation? Oh gosh that's a good question I need to go back and ask them I need to learn another evaluation I'd like to think so I think yeah certainly some of them could see what I was saying about making this more useful beyond just evaluation because I didn't understand why evaluation was useful to what they were doing and it was going to make that link where to actually being something more fundamental to the way this program was being delivered and monitored so yeah I like to think so I think that's probably the only way I can answer that question We have any more questions for Damie Patricia do you want to unmute yourself and elaborate? Yes okay hi so much that's a fascinating project so much hard work I actually work with a fund but it's much smaller scale than this and I have so many questions but one that you touched on that I think it's extremely important is the monitoring aspect of that and I was just wondering if you had attached to the outputs and outcomes indicators and if you did what was really the problem with monitoring I mean I think I can guess what you're going to say but given the multitude of people the resource what were the issues that and what do you think you would have in hindsight now pressed a bit more for them to do because obviously some of those metrics are what really we can rely on to check if we are achieving outcomes thanks so much James Yeah I mean that was something very early on when we discovered the resource because I was used to in the UK a lot of programmes using this ODA money are funded through what was at the time the Department for Industrial Development and they are very strict on you must monitor this very strong on the outcome frameworks and evaluation and it was interesting this programme was being influenced through a different department who didn't necessarily have that experience of implementing this ODA money so yeah we have very early on kind of mapped out partly building on that that you change okay well what are the indicators that you want to measure for each of these outputs and each of these outcomes then we made some recommendations early on on that I guess the challenge in doing that was because they had so many different delivery partners who were locked into contracts which were very much focused on the delivery and not the whole administration side of it which they would consider monitoring to be administration and there was a how do I describe it there was a an exchange of views on the extent to which you know they should be collecting the depth the extent of the monitoring data that we felt be useful for evaluation purposes but not just for evaluation for me there was a big accountability issue for the department here about you know how are you justifying the spend what are you actually achieving from it even simplest things like you know how many people are being supported or how many projects there wasn't you know a consistent way of doing that and you know we were trying to edge in towards can you have a centralized system or at least a common set of indicators that each of the partners would report on and they never really got there certainly where I was involved in it we ended up having to do a whole exercise of collating all the individual monitoring data from different people and then trying to bring all together and it ended up being really patchy you know we had some data from all partners on some things some data from a few partners on others so it became a really a real kind of patchwork so it helps in a sense but it certainly wasn't comprehensive so yeah it was it was super difficult and it was partly a kind of a contractual issue about the extent to which the partners were going to be expected to collect all this data Pini Hawkins Pini Hawkins I can see your message do you want to unmute I can do I just thought it was worth saying that and it particularly I guess points to the value for money and so value for who so at that point the UK Government and still now actually was primarily concerned about value to the UK and that really changed the direction of official development assistance so i chi for those of you who don't know is an independent commission for aid impact which is a kind of independent assessment organisation that looks at spending on UK aid money so there were lots of noises about the Newton Fund around the time and so an assessment was commissioned and Amber Red is not doesn't come up very often so I don't know if it's directly related but I know that Difford don't stop funding it unsurprisingly if you look at some of the findings in that report but I don't know whether did the evaluation come up I honestly can't remember it's too long ago but was that pointed to in the evaluation some of those issues were very much pointed to and there were a number beyond the issue of funding in the UK what are we evaluating here but there were also issues of how much it was really related to ODA eligibility particularly under that people strand there were funding lots of work well to develop the research skills which is perfectly eligible for ODA funding but some of the research topics that were funding them to do were very much not related to these global development challenges so there was a huge debate around that as well in terms of how this person's skills have been developed but they're actually not going to do with the global development challenges so that was a big issue from Aikai too and we highlighted that in the evaluation so yes and we did touch on some of these issues about where the funds were being sent most of all what we were asked to do was evaluate the funders at home and what is it achieving so it was kind of hinted at but we were trying to stay out of the politics of that and say here's what your fund achieved but there were certainly some very real issues around how this programme was actually aligned with ODA funding requirements put it that way probably worth saying that's become an even bigger discussion recently enough said on that I think yeah we have five minutes there's a question from Rula on ensuring cultural competency across teams yes so the way we did that was to work with essentially local evaluators in each country that we visited so often we'd kind of double hat we might have someone from the UK to go but also have a local evaluator on the ground who would kind of lead those conversations and provide that cultural understanding and be able to input and lead those discussions on the ground and so you know Tetra was a global company so we were able to sort of leverage the networks of contacts that we had so that each case study visit had a local evaluator at its core and that helped us to navigate some of those cultural activities and issues thank you so much Jamie for your time if there's no other questions I don't see any hands raising we might Yes Jamie thank you so much for sharing your knowledge and insights on this extremely complex newton fund project we have plenty of people applauding and thanking you I will do some googling myself and I will read some of those reports there's much coming to them I will try to navigate the complexities of international development on a massive scale thank you so much for your time