 Hello, welcome to episode two of the podcast we're calling Sustainable Development Goals, Evaluating Progress for a Brighter Future. I'm Doug Hoffman from DEVOL, the German Institute for Development Evaluation, and I'm your host. In episode one of the series, we heard how important it was to consider the effect of climate risks in the evaluation of strategies and programs designed to make progress towards the Sustainable Development Goals. That's certainly critical. And how are these evaluations done? What are the best methods, and what are the potential stumbling blocks and challenges? Why have so few SDG evaluations been done up to now? That's what we'll consider in this episode, because we know these evaluations are complex. Only some countries have done them, and the process has learned lessons that can be shared with others. For starters, we'll think about how to define the scope and the focus of the evaluation. Sometimes a major challenge. We'll hear about the first ever national evaluation of SDGs carried out by Finland in 2018 and 2019, where Finland took the bold decision to look at the whole set of all 17 SDGs together. And we're also here from Costa Rica that delivered an evaluation report at the end of 2020 that had been two years in the making. And I'm glad to say that the lessons it learned have been captured in a parallel process that was carried out along with the evaluation itself. We'll hear more about that shortly. So let's get started. And that means I'm first going to ask my three guests to introduce themselves. Good day, everyone. My name is Karolina Suniga, and I work for the Ministry of Planning of Costa Rica, which is the steering institution regarding evaluation for the public administration. Specifically, I work in the evaluation unit. And well, it's my pleasure to be here and be able to share our work. Hello, my name is Satu Lahtenoja, and I'm a senior expert on sustainable development in an independent think tank called Demos Helsinki in Finland. And I was leading the first national SDG policy evaluation in Finland. So in my work and research, I am looking for different ways to accelerate sustainability transformations. And I see evaluation as one of the ways to do that. Hello, how are you? My name is Gonzalo Hernandez-Licona. I'm from Mexico, and I'm the director of the multidimensional poverty network. And I'm also working with the Global Evolution Initiative, UNICEF, IDB, and others in terms of evaluation. I've done evaluation for the past 25 years. So I'm very excited to be here. Thank you very much. Well, thank you for those introductions. Let's start with the basics, definition of SDG evaluation. Perhaps we should be clear what we mean when we say an evaluation of the sustainable development goals, or when we talk about country-led evaluations. Gonzalo, would you like to start us off? OK, thank you. Thank you, Dick. I believe that the first thing we need to remember is that is what the SDGs are about. They are not only about a collection of goals to advance on them separately. The SDGs have principles, for instance, integration and coherence, or leave no one behind. So, for instance, in integration, what we mean on the SDGs is we have to advance on the objectives and the targets together, right? So the SDGs emphasize the inter-linguages between goals and targets, sometimes the trade-offs between the dimensions. So it is important that an evaluation of SDGs should follow those principles, and, for instance, to address objectives at the same time. So give an example, even in a family, we need to move together with education, health, and nutrition. Otherwise, a child cannot go to school if those three elements are not addressed at the same time. It means that we need to evaluate, in that case, that the family is moving on those objectives together. The same is happening in the evolution of SDGs. We need to move forward by taking into account the advances in all of them at the same time. Okay. Now, how many SDGs does it take to make foreign SDG evaluation? Do we have to look at all 17 SDGs? Or is it enough to just look at a single SDG? What's the experience of Costa Rica? Well, that's a tough question, especially for us, because I don't specifically think that I have an answer for that because I don't think there's a magic number. We have this conversation in our institution, especially in our evaluation team, when we were talking about the next national evaluation agenda, and we came to the conclusion that it's necessary for us to identify what is specifically the information that we need. And, well, in that case, it's not about how many SDGs we're going to evaluate, it's more about what else we need to know regarding the process of the sustainable development. And in our case, we haven't performed like a specific SDG evaluation, like Nigeria did or like Finland did, but we do perform country-led evaluation with an SDG component. And we include this component in every evaluation since 2019. And for us, it's very important to identify what is specifically that we need to know and then evaluating that specific component. In this case, we're talking about the biodiversity and climate change evaluation, which is one of the evaluation inside our national evaluation agenda with a broad analysis of the SDGs. So I don't think there's any specific number, and I completely agree with what Gonzalo just said. It's more important to identify how we're going forward in the sustainable development. Thinking about what you just said, how does evaluation relate to the SDG policy cycle? Gonzalo, what would be your take? So let me go back a bit of what Karina said rightly. I mean, there's no magic number about how many goals should we take into account to name it an SDG evaluation. I think as long as it reflects that we are evaluating systems and as long as we include in this the policy cycle, right, which means we have to understand that our evaluations should improve what we do. So and as long as we include some of the principles, like integration, leave no one behind, resilience, universality, right? So for instance, if we want to evaluate the no hunger SDG number two objective, so we need to assess the various elements that take us to have zero hunger. We need to assess the food production, waste, consumption, partners, nutrition, markets, green solutions to the food processes. So we have to evaluate the policy of zero hunger instead of one single program. I'm taking into account that we need to put it in the cycle of the policy in the government, meaning that many actors should improve according to the evaluation that we are producing. That sounds very complex, taking into account the policy and all the actors involved. If we just think about one SDG, Finland has opted to look at all 17 SDGs at the same time. How did you tackle this challenge of having 17 SDGs to look at so many sectors to look at and all the different stakeholders? Yes, in Finland we really did look at all 17 SDGs because we wanted to have a comprehensive evaluation. So having an overview of where the country is and then it didn't make sense to leave something out. However, we didn't look at all 17 SDGs with the same depth. I think one of the key challenges for our evaluation in Finland was how to cover the key substance issues in a reasonable depth with restricted resources. I think this is one of the general concerns in sustainability evaluations when the topics are often very broad. In our case, the solution was to find the top five most burning sustainability concerns by indicator analysis, surveys, expert interviews. We then found that competence and societal stability were one of the strengths of the country and then there were many systemic needs for improvement, including energy system use of forests and the global footprint of our consumption as examples. Okay, so you clearly went for a prioritization of the SDGs and looked at which ones are most relevant to the country. We've just heard Gonzalo talk about the SDG Agenda 2030 principles that they are important, like he mentioned coherence, equity, participation, leaving no one behind. How did Finland cater to the principles underlying the 2030 agenda? Or did you basically look at the targets and the interactions within the 17 SDGs? I think it's very reasonable to evaluate the policy against its principles and priorities. And I think the 2030 agenda principles can also work well as an evaluation criteria. In the case of Finland, we used the focus areas and policy principles of Finland as the criteria as Finland had defined, sort of own sustainable development policy principles and those have of course a lot common with the 2030 agenda principles. So they include like long term action and transformation, then policy coherence and global partnership and totally commitment and participation. So in our evaluation, we used those as criteria asking basically are the policies coherence is our sustainable development policy creating long term transformation. And so of course, super, super big questions in one evaluates. It is always difficult to tackle SDG evaluation because it's such a broad topic. It seems so broad. Let me move into the next set of questions looking at what is the value of doing an SDG evaluation and already talked a little bit about it. Within the UN Economic and Social Council, the ECOSOP reporting mechanism has been set up for the SDGs. Know that each country is asked to report on the state of implementation of the 2030 agenda in what is called the National Voluntary Reviews of VNRs for short. Gonzalo, about the importance of doing SDG evaluations, what can you tell us in relation to the VNRs? I mean, how is evaluation coming into the exercise of doing voluntary national reviews? Okay, so we just mentioned a little bit of that before that it is important to evaluate the forest and not only the tree or the leaf, we have to see the whole picture. I mean, development is complex, Dirk. It was, if development was only about cash transfer programs, then development would have been easy. But development is about incentives to grow and to innovate about balance of power between agents. It's about addressing various dimensions. And therefore, what we, I believe, we need to do when we write and we produce VNRs, Voluntary National Reports, is that, of course, we make accountable of what we're doing in the country. We describe what is happening in the country in terms of SDGs. But we also have to be critical about what we're doing. Right? It may be common that VNRs look very, very nice, because sometimes the country's put there that everything is wonderful. And therefore, we, in a true VNR, we should put our advances, the full picture of development that we are facing in the country, as well as the challenges. So we all could learn and we all could move forward with advances and challenges. Let me ask the same question to you, Carolina, for the evaluation that you did in Costa Rica. Why do we need evaluation when we have already Voluntary National Reports? And Costa Rica has filed Voluntary National Reports already. What about the relation of the VNR with the evaluation of SDGs from your point of view? Well, in our case, we see it more like a process, because for sure, evaluation is an input for the VNR. And the VNRs are not only for accountability, but to ask ourselves if are we on the right path? What else is needed to achieve our commitments? And for example, in our case, we have been identifying as part of the several evaluation findings that programs and projects, strategies, they all need to design considering the SDGs, because otherwise it will be hard in the future to achieve them. And overall, we see an opportunity because the VNR becomes another input also to identify priorities in evaluation. In our case, we are at the door of a new government. This means a new evaluation agenda. And we have to ponder what needs to be evaluated. Is it a specific SDG? Is it a complete, as Satu said before, like a complete policy regarding a specific SDG? We don't know, but we need to identify what is most important for us. And that is information that the VNR can definitely give us. So we see it as a cycle, evaluation is an input, but also is the voluntary national reports. That seems to be very interesting that the movement goes both ways. Evaluation, ideally informing the VNRs and the VNRs informing SDG evaluation. Would you agree on that? Or how does that work in practice, Satu? Yes, I very much agree with the others. And that's how it goes also in Finland. And what inspires me is that with the help of an evaluation, we can really have an overview of where we are in terms of SDG implementation and then understand where the challenges are. And then based on that, and that is often not done in a VNR, is to develop then recommendations on how do we then come closer to reaching the goals. And to get there, in my opinion, the evaluation should be at the same time of participatory, developmental and future oriented, or at least all of those were needed in the Finnish case. And there, of course, VNR is one important tool in there. Dear, can I add something else? Sure, go ahead. No, I think taking the example of Finland and Costa Rica for the VNRs is very important, because they include evaluation processes in the VNRs. They include challenges in the VNRs. And I think that's very important because as I said before, where you put a VNR, where everything is really beautiful in the country, it is not the way we have to do it. So example of these countries are very important. Okay, I think that is an excellent point. Thanks for signing this to us. We've heard a lot about the benefits of doing an STG evaluation. So what strikes me a little bit is the discrepancy with very few evaluations on STGs that have actually been done. We're now already seven years into the 2030 agenda. So what seems to be the major difficulty? Is it finding the right start? That is like the major hurdle. Would you agree on that, Gonzalo? No, I believe because as I said, developing is complex. The agenda is comprehensive and therefore complex. We were used or used to evaluate single projects or single programs. So from going to evaluating a single program to evaluate a whole full agenda, that's tough. But I believe that that countries we should carry on in trying to do this with simple questions. For instance, how did the concept of development change in the country as a result of the agenda? What happened in the country when the STG arrived? What was the change in terms of development strategy? What is the sustainable development theory of change in your country? Does the country have a solid data generation process on institutions? And a very important question that I believe it's, I mean, it's a tough one, but we have to address it is how are the national priorities made compatible with the 2030 agenda? And I think that's a key element. As a very key element, how the country is combining the priorities that already the country had with the new STG agenda. I think addressing that question with clarity is a good start, for instance. So aligning national priorities to the STG agenda, you tell us that is the most important thing we have to focus on. Carolina, taking your experience with the evaluation in Costa Rica, does that ring true? I mean, you've actually done an evaluation. How did you get started? Yeah, definitely. For us, it was exactly like that, because we were the first country in the world right to reaffirm a high level collective commitment to achieve the STGs. And for us, this meant put the 2030 agenda as a country priority. And however, we include this specific topic into evaluations until 2019. Of course, there was a lot of management to do between the identifying of how to measure the indicators, where to get the information, setting governments, governance, I'm sorry, but for a moment, I think that we forgot about evaluation and the STGs. And I think it's just like Gonzalo just said, it's how we combine the priorities of the country with the international commitments. And we decide the evaluation with the biggest component regarding the STGs was going to be probably the most complex evaluation inside our national evaluation agenda. So it's not easy, for sure it's not easy, but it is about make it happen. It is about what is important. Yeah, this is to me a very important point. You have to get started. You can't wait for everything to fall in place and then move ahead. You have to identify national priorities. You have to take decisions. And as Satu earlier explained by expert interviews, etc., involved in stakeholders, and then moving ahead. I would like to ask in the concrete case of Finland, because we have the country experience. Satu, can you give us as an example, how did you go about defining scope and focus of the evaluation? In our Finnish case, the focus and scope of the evaluation was quite broad and it was defined in the terms of reference. And that already then included also the evaluation questions. However, we then shaped those together with the evaluation team and with the steering committee when we started to work. And if I very briefly reflect on the choices we did, I think this concentrating on these certain sustainability issues instead of trying to cover all in equal depth was a good choice. But then, however, in addition to detecting on problems, I think we could have given more attention also to strengths and in particular teams and questions in which a clear sustainability contribution could be made also globally. So by asking what is the global sustainability handprint of your country and what it could be that your country contributes to the global sustainability. I think that is an aspect that is maybe not the case with each and every STG evaluation, putting the country in a global context. Usually from what I've seen, it works the other way around that the global context is the STGs and that is put into a national context. So that's interesting your perspective that you're giving us. Something else that we've been talking about a little bit already, mentioning here and there, but I think deserves a little bit more attention is the participatory process that an STG evaluation might be or should be. How do we go about designing a participatory process? How far was the evaluation participatory in Finland? Yes, as we all know, these participatory processes, they tend to take a lot of time, but I would say it is really crucial. So in the Finnish case, we had several groups we wanted to involve into evaluation. We had a steering group with representatives from the ministries, and then we had a support group with like 20 experts from different fields, sustainability, evaluation experts, civil society. Then we had international experts, and this was very, very relevant and useful. And then we had like a large and open sustainable development stakeholder community. And then of course the Finnish parliament, especially there, the committee of the future. So what we did was actually quite a lot of interviews, workshops, different discussion events. And one thing that worked really well was that we co-designed the recommendations together with about 40 stakeholders. And this was a way to really engage and also empower them. So when the evaluation report was published, there were actually no surprises for this group of people anymore. And they were then also highly engaged to disseminate and work further with the recommendations. Okay, this is a very valuable point, which probably holds true for evaluations in general involving stakeholders in the whole process, in the design, and as you just said in the formulation of the recommendations. Gonzalo, from a more theoretical point of view, would you see any specific challenges for participation when designing evaluation of the STGs as compared to a regular evaluation? No, I believe that because we are trying to deal with not only with a single program or a single project evaluation, but we are dealing trying to address the full agenda. Of course, we need more people and we need more stakeholders and we need them to be on board on the evaluation and to be frank and to know that they will benefit as well for the evaluation. And that is not that that easy, right? So I'm really impressed with what Nigeria, Finland, Costa Rica have been doing in terms of evaluating the STGs because that implies a lot of participatory process. Okay, excellent. There is another topic often forgotten when we talk about the design phase of an evaluation and that is communication. Should communication already be considered in the design phase? What is your opinion? Or is it better to first see the results, the recommendations, and on that basis define a communication strategy? What are your experiences in Finland and Costa Rica? Carolina, maybe you go first. Sure. I think that you can have a specific strategy, depending of the evaluation, sure. But in our case, this is kind of a standardized already because we evaluate public interventions and it is our job to share the final results with anyone who's interested. So, yes, in our case, we pretty much do at least three things for every evaluation. We first upload the final report in our web page. And of course, we encourage the institutions involved to do the same. We also organize at least three final presentation of the results of the evaluation, considering different target population. And at least one of those presentations should be like an open one to be able to invite different stakeholders, especially citizenship, because it's very important for us for people to know what we're doing and to use also the evaluation results. And we also develop different materials. This is different for every evaluation, but for example, we do different videos, documents, infographics. This kind of depends on the evaluation, but it's usually some kind of material that can be distributed easily. And we participate in workshops. We also attend to a specific request to share the results. We participate also in conferences, because it is important for people to know that this kind of evaluation are being performed. In Finland, did you consider communication as important in the moment when you designed the STJ evaluation? Or did that come in later? Yes, I think it's very important to think about it in the design phase already. So first, I think timing is important. So in our case, the timing of the evaluation was good, as the results were published right under the parliamentary elections. So there was interest towards the results directly. Secondly, what Caroline also mentioned are these visualizations and key numbers are very important in communication. So it's good to require them and also leave resources for that. And then thirdly, I think it's very important to leave time and resources for communication in general after the publication of the results. In Finland, we then as an evaluation team went to discuss the results to the parliament. I counted in total eight times. So we had five different separate discussions with different political groups and three different committee meetings, even several committees together, which doesn't happen that often. And we were then also invited to present our findings in the government negotiations. And I guess our work had at least a small contribution to the fact that actually the government program of the current government in Finland is based on sustainable development. So we are at the moment quite lucky. Congratulations to Finland. I think this is really a great example for others to follow along. We're almost coming to the end of this episode and maybe wrapping up a little bit, looking at the different topics. And obviously, we could have delved into much, much further. And this is just starting off discussion on those topics. But maybe instead of me giving overall conclusions, which would be very, very difficult, I would like to do a final round briefly giving the microphone to each one of you for one main message to formulate a recommendation or one central thought for our audience. If you would start, Karolina. I would like to bring a topic that we talked about at the beginning and throughout the whole episode. And it's about the principles of the 2030 agenda. I think that Gonzalo talked a lot about this. But we present times with Corona showed us that we need to have them present at all times, not only for evaluation, but for planification, for planning, any kind of intervention also. We have to have them present if we want to achieve common goals as a country. During the whole policy cycles, and of course, in our work as evaluators, we need to know how we're going forward. If we're going in the right way, we need to share experiences, resources, knowledge, making sure that we're going in the same direction, working together. I think it's important to have them present. Moving over to Finland, Satu, what would be your main message, main thought? Quite difficult to choose only one main message, but perhaps for this audience, I would highly encourage you to use external experts, meaning also your experts and contacts from other countries. It helped us a lot and I think raised the level also. So use your networks to get more perspective. Thank you very much. Last but not least, we're moving over to Mexico, Gonzalo. Thank you, Dirk. I believe that having evaluations beyond single programs and single projects is a key element for the SDG. You have to evaluate policies. We have to address objectives at the same time when possible. We need to do it in a participatory way. We need to look beyond the obvious, I believe. I mean, for instance, it's very simple. It's very common that countries just link what they were doing before with SDGs and they say, hey, we are addressing the SDGs because everything that we used to do is linked. We have to address that and challenge a little bit the country and challenge a little bit the government saying, hey, we need something else in terms of a sustainable agenda if we really would like to put it forward. And evaluation can help in understanding that. So thank you very much, Dirk, Satu and Karolina, I'm glad that we're here together. Well, thank you very much from my side to our three guests, to Karolina Zuniga from the Ministry of National Planning and Economic Policy in Costa Rica, to you, Gonzalo and Anders Likona, Senior Research Fellow at the International Initiative for Impact Evaluation III, IEE, and also to you, Sartu Lechtenoje, Senior Expert on Sustainable Development of Deimos Helsinki, Independent Think Tank from Finland. Then I would like to finish with a reference to the website, the IIED website, where you can pick up additional information linked to the organization's website reference material to delve deeper into the topics we've just been discussing. And as a last point, finish with a look ahead at Episode III, the final episode of this podcast series. In this episode, we will be covering the aspects of implementation and use of STG evaluations in order to accelerate progress for reaching the STGs by 2030. I hope you'll join us. Thank you very much and goodbye.