 Hello and welcome to the first episode of a short series of podcasts entitled Sustainable Development Goals, evaluating progress for a brighter future. My name is Stefano Derrico and I am head of monitoring, evaluation and learning at the International Institute for Environment and Development, IIED. In this series we'll explore how the national policies and interventions are going to achieve sustainable development because sustainable development is every country's goal, but how we get there isn't always straightforward. Too often projects and programs work in silos and ignore the interactions within, between and around them and these can hamper progress. That's where evaluation can play a part. Evaluation of progress against a particular sustainable development goal can identify the interdependencies between the social, economic and environmental dimensions of policies and interventions. And now we have obviously come out of a key climate moment with COP26 and climate change, which are the most striking example of how action towards one goal can dramatically affect programs addressing the other sustainable development goals. So under new and very different climate conditions the existing adaptation practices and processes will no longer be viable and progress against other goals will be seriously undermined. Thus it's vital that climate risks are identified and integrated into evaluations of different thematic areas, education, health, clean water and sanitation and all the other things that are covered by the SDGs. So in this first episode we are discussing how evaluators can do that. We'll hear what Kenya is doing, we'll think about the challenges and also the best way to address those challenges by using a systematic approach. So without further ado I will ask my guests to introduce themselves. My name is Samson Machuka and currently serving as a director of monitoring and evolution, directorate within the national treasury and planning of the Republic of Kenya. And the evolution has been one of our core main activities as far as monitoring and evolution programs. And I'm Emily Beauchamp, I'm a senior researcher at the IIED in the strategy and learning group. My research focuses mostly on the nexus between environment, climate and evaluations and that means my work involves designing evaluative and research frameworks to assess progress in climate and development policies but also working with national and local governments on evaluation and monitoring evaluation and learning systems. Hi I'm Nick Brooks, I'm a researcher and consultant and my work focuses on climate change adaptation, particularly on climate change risks and how we can integrate considerations of these risks into development, including sustainable development. I worked extensively on monitoring evaluation and learning issues around climate adaptation in both an academic and a practitioner context in space of international development. And I'm the director of Garama 3C, a small consulting firm based in the UK and I'm a visiting research fellow at the University of East Anglia. So Nick, I've just talked about climate risk, perhaps to start us off you could tell us what we mean by that. Sure Stefano, yeah so there are lots of different types of climate risks and typically in the field of international development, sustainable development, we're thinking about risks associated with particular interventions, development projects, programs, sustainable development initiatives and so on. So when we think about projects and programs, particularly from a monitoring and evaluation perspective, we often think in terms of outputs and outcomes. So on the one hand we can think of risks to a project's outputs and outcomes and ultimately to the impacts that we want to see, but we can also think about risks from a project, from an intervention. So we've got these sort of risks facing in two directions, to the activity we're talking about and from the activity. So I'll say a little bit about those in turn. So we might have risks to a project or an intervention's outputs, these obviously are things like goods and services we're trying to deliver in the short term. So this might be as simple as climate disasters such as floods or storms, hurricanes, simply destroying infrastructure that's been put in place by a project or preventing access to project sites, for example. We might also have risks to these project or program outcomes. We might be looking at an intervention to enhance agricultural productivity, this could be climate smart agriculture, sustainable agriculture, or the objective of an intervention might be, for example, to improve the health of a particular ecosystem or a particular set of natural resources. So this might all be very well under current conditions, but as a result of climate change, we might find that we have things like water shortages, additional pressures on ecosystems, and so on. And this might make it more difficult to achieve the outcomes. We might increase sustainable agricultural productivity for a while, but then find that as water resources become more scarce, that isn't sustained and perhaps can't be sustained in the future. We also have the risk of a project or an intervention being redundant. For example, we might be looking at an intervention to try and sustain resources, natural resources, biological resources, for example, that simply won't be viable under climate change and that they'll just disappear. So we have to ask whether trying to sustain something that's ultimately not going to be there in the future is worthwhile. And these are really hard calls to make. So I'll say something briefly now about risks from an intervention. We might have an intervention that increases the vulnerability of a particular group of people. A classic example would be something like agricultural expansion that displaces pastoralists or makes it more difficult for them to access pasture during dry seasons, making them more vulnerable to climate change and climate change impacts. An intervention might also increase environmental vulnerability. Again, if we're looking at coastal developments, then these might create a sort of hard infrastructure that prevents coastal ecosystems migrating in land. Now if we're looking at sustainable development interventions, hopefully we wouldn't be in that sort of situation, but that's a sort of a pertinent example there. And finally, if we fail to account for and understand how climate change might change what is viable in the future, we might create entire systems that are based on false premises. We might make an entire economy or society or region dependent on resources that are not going to be there in the future because of climate change. And why are climate risk important when it comes to evaluating progress against the sustainable development goals? Yeah, so I mean, this is the critical question really. I think the bottom line here is that something that is sustainable under current conditions might not be sustainable under future conditions because of climate change. So if we are ignoring climate change and we're evaluating a sustainable development intervention and we're saying, yes, this looks great. It looks like it's going to be economically sustainable, technically financially sustainable. It looks like it's going to deliver the outcomes that are intended. And we're not thinking about how climate change might alter the landscape, the context in which this intervention plays out. Then we might just be getting things completely wrong. So I think the task for sustainable development evaluation is to ensure that sustainable development initiatives, interventions, activities have taken these risks on board because if the evaluation doesn't ask this, then it's pretty pointless, particularly in high-risk contexts. So what we are saying really is that it's vital to include considerations of climate risk in sustainable development evaluation. I mean, as sustainable development goal evaluations. And Samson, I know that Kenya is doing this. Can you tell us more about it? Yeah, thank you, Stephen. I think the question you're asking is very pertinent, particularly to us as a country. And you're asking that question at that time. We have had just a big conference in Kenya involving our governors. You know, we have governors who are heading out for seven counties and the theme of their discussion was mainly climate change and how it poses major risks in terms of agricultural development. What I need to mention at this take is that Kenya's technique climate change issues very, very, very seriously to the extent that in our Kenya Fission 2030, it is one of the key pillars, key goals, that is being focused on in terms of our planning processes. And therefore climate change issues have been captured at national and our account levels in their main five years manifestos and development plans. So what we are doing is that they have those programs and we have come up with critical indicators that have been put in place to ensure that they are being tracked and therefore to make sure that the climate change issues are addressed effectively in terms of making sure that the green emissions are actually reduced drastically over the five years we have planned. So it is one of our key goals in our development agenda and I believe that we are headed somewhere because indicators are really behaving positively in terms of making sure that there is a lot of responses to the results of the climate change policies that are being implemented. And it's great to hear that Kenya is really considering climate risk not only in evaluation but also in national policies. So I really wonder whether it's been always so straightforward. What I'm wondering is what kind of challenges you have encountered. Yeah the kind of challenges we are facing here definitely is that climate change is not a one-time or an instant thing that can come to to be addressed. It's a long term so the kind of challenges we are facing is that obviously resources have been a big problem and they also make sure that the policies or the wish of the government to the people still takes time to be any culture. You know to any culture the people to start addressing climate change issues is not a very easy issue but nevertheless it is going on and then I can also say that the lack of adequate resources particularly in terms of making sure that sensitization programs take place on time and also making sure that the evaluation issues are strongly mainstreamed in our laws and so forth. I think it says it's where we have a slow uptake but nevertheless we are moving in and we hope that that will take shape. Lastly maybe something else I can say the other critical challenge which I can also say is that coming up with the very very very smart indicators that can give us evidence on the basis of how we are doing at climate policies effectively and how they are impacting the whole process is where we have a slight challenge because it's not easy to get the actual data and it's also not very easy to to slow down to that empirical information that can now tell you for sure that this change is purely coming from the implementation of our climate change aspect. So those are the small challenges that we are facing but I think with time we'll be able to overcome it takes a little while but that's where we are and we are getting their support from international community. We are also allocating a submission before our policy, mandate policy is already attracting one percent of the budget and once we have those resources available with us then we can now be able to have competent people with us who can work the ground to make sure that they are getting credible source of information that can be used to inform policy in terms of how we are moving in changing the climate aspects so that we have amicable progress in policy. So credibility of evidence is where we have that kind of challenge particularly emanating from the climate change so that we are able to report substantively that this is where we need to change or this is what we need to change and and the and the and the influence policy and so forth. So I think it's a bit of a challenge but we are we are hopeful that we are moving we are moving anyways. Thank you. Great thanks Samson and I I mean it is it is really interesting to hear you know how you are addressing these challenges and I was wondering how these challenges in light of the challenges and of the climate act that has been approved. How do you think evaluation is going to change in Kenya? How do you think evaluation is going to address these issues? That's a good question we have we have prevailed since monitoring and evaluation policy has already been approved. The most important thing then is that now evaluation results will be required to inform policy because in the past policies have not been informed by anything because we policies have been made on on on arbitrary basis but now with evaluation evaluation reports coming out very well out of where research findings of the evaluators, then it's unsuspected it's actually required that those policies now or the the the reports must be able to influence and inform policy of what is required to be done and therefore the evaluations are going to be made to and be used by the implementers. But I think for climate change, climate change requires elaborate evaluation so that that kind of evaluation can then be able to influence government budgeting system and policies so that at least then we put in a implementation mechanism to ensure that the results or the findings of those evaluations are implemented with a letter and I think that way we can then be able to say that we are doing something meaningful in terms of climate change development. Thank you Samson and Emily, we heard from Samson that Kenya has already got that far, it's already doing a lot for the rest of the climate. Policies and evaluation, is that the same in all countries? Well unfortunately not really, there is a spectrum of different experiences and levels of integration of climate in evaluative work. Some countries have started to integrate climate issues in the evaluative frameworks of the country and some countries are considering climate risk in symptomatic evaluations when they're assessing specific development progress. What I mean by that is for example countries like Costa Rica, Colombia and Mexico have started to monitor and evaluate climate indicators across development while other countries have successfully done cross-dematic evaluations that look at several areas of sustainable development including climate as one of the themes. So climate is increasingly a theme of evaluations and a focus of monitoring and evaluation or or M&E systems. Although there is still an overwhelming focus on mitigation or reducing emissions and being climate smart if you want rather than also looking at adaptation or how we adapt to ongoing and to future climate risks. Mostly integrating climate risks into evaluations and recommendation is not happening very often systematically. So you have cases here and there but you don't have a constant practice of doing it especially not across all countries and that really involves both countries from the global south and the global north. So in general one of the problems is that and echoing what Simpson has mentioned right before me is that there's a lack of resources both in terms of human resources within government evaluation teams but also lack of resources in terms of financing to target the themes of climate in evaluation and the integration of climate risk. There's also a general lack of awareness and commitment at higher political levels that's that's lost things down really. For example in most countries just like Kenya governments decide which are the priority programs and policies they want to evaluate on a yearly basis. So evaluation tends to be short-lived and in that way you have limited information that can be useful to predict future impacts but also that can lead to having smaller political impact. I'm saying this this is within governments but it's also true at the international level for example from large multilateral development institutions and climate and development funds that are often funders of evaluations and so in that sense evaluations are often used for accountability reasons to justify how resources have been used rather than truly as a tool for adaptive management and learning. So how can we address the fact that there is different progress across the board? I mean it sounds like we need to be more systematic don't we? What would be the first steps towards that maybe? Yeah you're right Stefano there is a need to be more systematic and actually the good news is that just like some other aspects of responding to the climate crisis we do already have the necessary data sources of knowledge and methods to understand climate risks and shocks. For example national meteorological agencies now have good climate data across a lot of countries most countries and enough projections and the quality of earth observations or satellites geospatial data is increasing every year. Down to the community level more and more people are aware and understand the impacts of climates on their communities and ecosystems. So we do have data and information but the blockage is changing the traditional evaluation approaches from looking primarily at retrospectively at impacts towards a more forward-looking outset. In fact evaluations are primarily looking back at what effects and intervention have had in order to recommend improvements in the future but that premise doesn't really hold because of the uncertainty of climate changes and shocks as Nick mentioned earlier. So the value of information of current evaluative approaches is limited especially if climate risks are not considered. So the really first step for the new generation of evaluation is the need to focus on capturing how climate risks have affected progress or lack of progress in development and then looking forward when doing the evaluation. So in a nutshell evaluators need to change their thinking from what worked to what will work. Nick can you tell us more about forward-looking methods? Sure I'll have a go. Yeah I think you raised a really good point there only about the use when we have a lot of data that we can call on now and we need to start thinking about how we can use that data that information about you climate risks, about potential future changes and impacts to inform evaluations but I think maybe more importantly to inform the activities that are being evaluated and then the job of the evaluation is to ask how well those activities have actually incorporated that knowledge and that information and those data about climate change and climate change risks. Now so that means that this can be pretty challenging for you know development sustainable development undertakings to actually develop a good understanding of climate change risks and to develop good ways of addressing them. Historically we've seen probably quite simplistic deterministic approaches where people look at climate projections narrow range of variables and say okay so we expect it's going to be on average about three degrees warmer and rainfall is going to decline by 10 to 20 by this state. Now of course climate change is sort of much more uncertain than that we're very confident about changes in temperature I think it's only the global level and at the regional level and maybe more locally leaving when it comes to things like rainfall and more complex hazards it's a lot more difficult to sort of look forward. So we have these emerging techniques things like robust decision-making where you use climate projections to inform your thinking about risk but you don't let them drive it you think more about the conditions under which certain activities processes behavior systems fail and under which they succeed and then you think about how likely these might be to be met in the future under climate projections but also considering unknown unknowns outside of the range of projections. So as you can probably tell this can get very complicated very quickly so while I think there's a is a central place for using climate information climate data in the programming of development actions and also to consider in evaluations the extent to which this has been done. There are also some other sort of more general things that we can look at as well if we're trying to do more forward-looking evaluations. There have been a number of sets of principles developed for guiding adaptation and for assessing adaptation so we might look for example to the iid principles for locally led adaptation we also have something that I worked on again with iid colleagues and this is the so-called camels framework and camels stands for climate adaptation monitoring evaluation and learning systems. Yeah so so I think again just to summarize that rather convoluted explanation yes we can do forward looking evaluation by either using or assessing the extent to which climate change data projections have been used in planning and project design but we can also do this forward looking practice by asking whether the thing that's being evaluated has adhered to certain key principles around for example use of science and traditional knowledge use of appropriate information around transparency participation equity and so on. So that is a very good and useful framework for practice Nick and I was wondering what happened next then Samson what what are the key next steps for Kenya? The next key steps for Kenya is to ensure that the evaluation culture is strongly entrenched in our system and in fact we already have a very strong organization in Kenya called the Evaluation Society of Kenya which is really working closely with our directorate so that we also give evaluators autonomy you know when you want to do evaluation of government programs you cannot do it from internal platform so you need to give us some independence of people outside government to to execute an evaluation because then they are done and parceled so we are we already have we have the Evaluation Society of Kenya which has come out very strongly and it's really the body that will be required to coordinate of course in in in collaboration with us the selection of competent evaluators wherever they are who can come and do evaluation in specific sectors of the economy particularly the climate change issues that are now affecting us so that's what we are doing and we believe that we will be able to mobilize resources in view that we already have our policy in place of one percent so ministries will have to put one percent of their development budget to support evaluation programs so that when the evolution is done then nobody can reject it so we are moving towards the direction and therefore evolution is going to be a key thing and we are even proposing that the revolution to take a top notch position in government we are elevating the department of monitoring and evaluation which is within the national treasury to become semi-autonomous and you know when you are semi-autonomous then you can now be able to do you can now work independently because we know the subject matter is not a popular subject matter you know when you talk of evolutions not everybody's amused about the evolution so we want to have some independence so that's that's actually we are moving to and I believe we are soon getting there yeah thanks Samsung and Emily in terms of supporting evaluators more broadly what that means in terms of next steps yes well as Nick mentioned there are a lot of resources available and data but a lot of information is still spread across different sectors and sources so it can be difficult for evaluators to start making sense of applying a new approach or rather a new outlook that is why IAD will be publishing a guide to support a more systematic approach of how to integrate climate risk into evaluations the guide will include an overview of key climate issues to take into account across sustainable development sectors or sustainable development goals if you want and but it will also include a higher level approach of how to use principles from the 2030 agenda for sustainable development and filter in climate issues and climate risks so this will help evaluators use a new lens in their evaluation but there will also be a step-by-step approach to help guide evaluators there will be suggestions of questions to inform the evaluation for example we suggest moving from just asking for example if there is an evaluation about poverty and food security has the program improved the livelihoods and food security of the most marginalized group so additionally asking have there been climate shocks and changes that have affected how livelihoods and food security improved or did not improve over the course of the intervention but also we need to ask how will a similar program need to be adapted to continue being relevant for the future marginalized group in light of future climate risks so and this report is really based on years of shared experience between the authors new empirical research and collaborations with government evaluators like samson we're expecting the report to be launched shortly in 2022 so with that guide as a support it sounds as if the process could be made more straightforward but then what needs to happen to make this systemic change more training more money perhaps you don't like to tell us what you would prioritize and why samson first and then emily and finally nick yeah yeah I think resources are important you mentioned about resources sources still remain core if evolutions have to be very very effective and I also need to mention that one embedment that perhaps I needed to have mentioned earlier is the fact that evolution you know must be evidence based and evidence evidence means data getting data that can help you combine an evolutionary part but the efficacy the reliability and the timeliness of the data we are talking about may may not be very easy to come by and even the people who want to give you the sources of that information still requires to be it's an area that requires to be addressed because sources of innovation of information the type of information you are getting might not be a very easy thing for you to be able to execute a very credible evolution process so to me that's an area that perhaps will also require to to be looked at because the source of information is very very important thank you thank Samsung Emily do you want to say something yes and I would very much agree with that Samsung and it's also good to remember that evaluation needs to be done in collaborations to start creating effecting changes just like we need more than one source of data and knowledge to do cross-sectoral evaluations and direct climate issues what I'd like to see happening more is collaboration and connections between institutions between government departments and key decision makers information but also policy making making is really siloed at the moment and that prevents cross-learning and that prevents really addressing climate risks so collaborating between department is the only way to understand the implication of the synergies and the trade-offs between different climate risks and different policy impacts and also collaboration can help with sharing limited resources Nick yeah so building on what Emily and particularly Samton have said I think this idea of like mandating evaluation is really important and I think Kenya has some good precedent here I think the other thing to you know just reiterate is and again Emily's touched on this as well evaluation process need to be inclusive and they need to include diverse voices particularly the voices of the people who are going to be affected by the interventions we're evaluating and again this goes back to addressing some of the risks and trying to ensure that that these risks are considered and hopefully avoid it for example risks of increasing the vulnerability of certain groups which are related to risks of displacement exclusion human rights abuses and so on so you know a sort of do no harm principle demands that we take an inclusive approach but I think more importantly or at least as importantly if we have inclusive and diverse voices then we get a better picture of how well things are working on the ground and that that sounds sort of obvious and triped but I think too often that's not done and again in the name of sustainable development we have activities around things like conservation carbon offsetting and rewilding and there will be risks there that are analogous to these risks of increased vulnerability we hear a lot about people being excluded from areas on which they depend in the name of these these sorts of outcomes so I think again inclusion is really important there and I think finally I think certainly training is critical we need people who are designing interventions but also the people who are evaluating those interventions to have a better understanding of climate risks and how consideration of climate risks can be integrated better into sustainable development planning programming and activities I think too much sustainable development and wider development activity fails to grapple with some of the really profound climate risks we're going to be facing and is quite often limited to quite tokenistic sort of incremental adaptation generalized resilience building generalized conservation and what we need to be looking at is how climate change alters the dynamics of sustainable development and how we can ensure that development is sustainable in the face of climate change I think just a sort of final word on this for me would be that if development isn't adaptive if it doesn't incorporate adaptation to climate change then it's not sustainable and that's the bottom line that's what I think evaluation of sustainable development with a climate change lens needs to grapple with okay that's great Nick a bold statement and an excellent way to end our conversation so thanks to all our guests Samson Machoka thank you Emily Busham thanks Stefano Nick Brooks thank you thanks for listening to this episode of sustainable development goals evaluating progress for a brighter future on the next episode we discuss the process of defining the scope and focus of an evaluation something considered by many to be a major stumbling block to measure in progress against the sustainable development goals to find out more about this podcast our guests and their organizations visit the IIED website IIED.org see you next time