 Welcome to the third of the series of the podcast on Sustainable Development Goals Evaluating Progress for a Brighter Future. This is Kasim El-Sadiq. I am the co-chair of Eval SDGs Network, the global network of evaluation practitioners around the globe. We've discussed so far adaptive evaluation when considering global climate risks and in the second episode we covered how to get started in designing, scoping and defining the SDGs evaluation. In today's session we will be exploring with our guests the use of the country-led evaluation of the SDGs. And with that I'd like to introduce and to welcome three of my guests for today, Dorothy Lux, Ana de Oliveira and Pethi Osekela. Dorothy? Yes, hello to you all. My name is Dorothy Lux. I'm the executive director of SDF Global that stands for Sustainable Development Facilitation. I'm a credentialed evaluator and we operate a social enterprise that works not only in evaluation but also in a whole range of other sustainable development support processes. And good to mention that Dorothy was the former co-chair and one of the founders of Eval SDGs Network. Welcome, Dorothy. My second guest is Ana de Oliveira. Ana? Hello everyone. So Ana de Oliveira, I am the policy and research officer at Cooperation Canada which is a Canadian network focusing on international cooperation. So I lead the files around the 2030 agenda and the SDGs within the organization and for two years in partnership with 12 other global organizations, I coordinated an independent civil society-led report on the progress of national SDGs implementation. Welcome, Ana. My third guest is Pethi Osekela. Pethi? Oh, hello. My name is Pethi Osekela. I'm a managing director of Friskit and Joy and a research company doing evaluations but I also have a position at the University of Vasa. I'm a research director in a research platform called the complexity research and that is a systems approach to more complicated and complex societal issues. But I've just started an evaluation of the national SDG goals in Finland and that is national evaluation so I'm willing to share also some of the ideas that we have in mind during the national evaluation. Thank you. Thank you Pethi and Ana and Dorothy and I think Pethi you've laid the ground for probably the first set of questions and kicking the dialogue and basically I'd like to start with you so we're clear and we have some clarity on what is a country-led evaluation since you're just embarking into it. What is country-led evaluation? Why is it conducted and how is it done? That is an important question since Finland has been very active in evaluating the national implementation of the SDG goals. We carried out the national evaluation in 2019 and that was an excellent evaluation. It gave a very good state-of-the-art analysis where we were at that time and also I was very happy to see that quite a few of those recommendations in that evaluations you could see in our government the forthcoming government program and its implementation guidelines so it's really fed to the political process as well. And then Finland has also carried out so-called the voluntary national reviews and the latest one was in 2020 and that's also an excellent report and there was a Switzerland and Mozambique that gave the peer review comments that report and it's an excellent piece of work also. But why Finland is doing another evaluation at this time? I'm not fully aware of all the motivations behind it but I think one of the reasons is that SDG goals have been very high in our government agenda and also quite a few of our ministries want to see the continuation since we have a government election coming in next spring and that is a way to give the state-of-the-art analysis and results prior to government program preparation. So I think that's one of the key reasons. And the main emphasis in our evaluation is the implementation, the governance and the mainstreaming issues from the governance perspective. That's one of the main topics. Thank you, Petri. So mainstreaming issues, governance issues that seem to be carried over from one evaluation to the other within the context of the agenda. Dorothy, what if it's not a full-fledged evaluation of the SDGs? What if it's only contribution to the evaluation of the SDGs? And from your perspective, what can it bring to the picture and what role different stakeholders would be contributing to? Yes, thank you, Kasem. There's no doubt that Finland has been a world leader with regard to evaluation of the SDGs and Petri, we watch Finland with great interest on how a more systematic approach across the SDGs is actually carried out. Most countries find that difficult to achieve, that very often the issues that they're facing are very broad and vast and therefore the resources to lead a full-scale evaluation of the SDGs at the national level is beyond their reach at this stage. But what has been happening is where there's a lot of focus on the partnership evaluations. So they may not be fully country-initiated and led, but there is strong partnership on particular SDGs, if we take the example of Nigeria, where they went through a process of prioritizing which SDGs they would focus on. Or in other countries where there's been a focus on SDG-2, SDG-1 that particular organizations have worked with national agencies and other stakeholders to focus on particular SDGs and accelerate and deepen understanding of progress around those. And a contribution is still progress and better than nothing at all. And there are useful lessons coming out of these. The principles embodied in what Finland are doing, even if they can't be fully replicated, there's a lot that still can be done to push forward on country-led evaluations related to the SDGs. Brilliant, thank you so much. So in a sense, a country-led evaluation, while it has a premise that revolves around national ownership, whenever capacity and resources are not there, we focus on our partners and we capitalize on the partnerships to get it running, to get it done. I mean, from there I'd like to pick up on that discussion with Anna and see and explore with her that although it happens at the country level, but some of the results of those evaluations of the SDGs are picked up also and used at different levels, at various levels. From your experience, Anna, I mean, what other levels do these country-led evaluations, what are they used there? So these evaluations, they can be used at the global level. So for example, to measure progress strategically across all countries, as well as national and subnational levels. So these evaluations at the country level, and particularly the voluntary national reviews or the VNRs as we call them, they bring contributions in different areas. So for example, they outline which governance arrangements exist, which institutional mechanisms are in place to realize the agenda and the SDGs, which policies are being implemented, how multiple stakeholders and this including non-state actors are being engaged or how the country is pursuing partnerships. So all these elements can contribute to global level conversations. And picking up on what Dorothy just said about partnership evaluation, there is an interesting example from the 2021 VNR reports that relate to coordination and partnership in reporting. So there was a coordinated effort between six small island developing states or the SIDS as we call them, in which they collectively highlighted common vulnerabilities, challenges and strategies, which could then be turned into shared solutions and new common opportunities to contribute to sustainable development initiatives through inter- and intra-regional cooperation. So all these would feed into another level of conversations in the regional and in the global spheres. Of course, the country evaluation and the VNR, the relationship between them is more or less complementary. In the sense that VNR might feed into the counter-led evaluations and vice versa. Probably I want to catch up with Dorothy on the use of those counter-led evaluations and VNRs globally, both in terms of accountability and in terms of learning. From your perspective, what lessons have we learned so far after seven years of engaging on the agenda 2030 with whatever we have in terms of evaluation and VNRs? Our positive lessons and negative lessons, I think, that what we've learned over the last years of implementation, that there's no doubt that there has been an increase in what we call meta evaluations, where there's an approach particularly by some of the development partners to look across different countries and look at the emerging lessons related to the SDGs or around particular themes which are important across the SDGs. So say, for instance, meta evaluations on capacity development or on learning, methods of learning. So these meta evaluations contribute very much to the development body of knowledge, and these are used by development partners. We know not only within specific development partners, but there is quite a lot of collaboration across different networks like the UN evaluation group, like eval partners. So those global platforms allow for sharing of information. Similarly, the VNRs that UNDESA does generate a synthesis across the VNRs. And this is a very useful resource to look at what aspects are moving well and which need further focus. But at the same time, there are some lessons around the challenges with that. One is that there are insufficient evaluations actually being carried out, that there has been a strong push to develop national evaluation systems, but perhaps with limited success or very varied success. This means that the number of evaluations being generated are below what might be possible, below potential. Also, there's not always a systematic approach to looking at country-led evaluations, not just those related to a particular organization. Thank you, Dorothea. You've raised a couple of very interesting points that we can probably carry out with Petri. How the evaluation in Finland potentially is informing policy development at the national level? If you give us some examples, probably, to pick up and then share the learnings with others. There are actually two very important platforms in Finland. One being the Strategic Research Council under the Academy of Finland, that is more, let's say, research-oriented, but also has a certain evaluative missions included. The Strategic Research Council aims to identify the critical phenomena at a very broad level, and that also links to the SDG goals and targets. There are big research programs that also contain elements of evaluation, so that's a long-term strategic research and evaluation. Then there is a new instrument called the ENTIAS, which is the new funding instrument located under Prime Minister's office. They carry out several annual evaluations that are targeted to critical phenomena related directly to the government program and its implementation. When the finance comes from the Prime Minister's office, then also it's easier to feed the results and recommendations back to the policy process, because it's located at the high level of government. That's one of the reasons. The second is also that various ministries are very keen on financing their own evaluations, somehow related to the various goals and targets of the SDG. That also feeds to the sectoral processes at the ministry level. I would say these three are important platforms for doing evaluations. If you allow me on picking up on this, obviously it seems that there is a systematic approach in conducting the evolution in Finland, and there is more of a well-crafted process that feeds into the policy making and development agenda. How inclusive are those evaluations? What about the current one? How inclusive it is, and what type of stakeholders and partners are engaged? I have to say that the VNR in 2020 was extremely inclusive. When we started our evaluation mission, we went through the VNR and its main recommendations. I was very happy to realize that the involvement of civil society organizations, various NGOs, and also the enterprises and the business community was included. The way how they included those stakeholders was applying the contribution approach. NGOs and companies reported how they have contributed to the certain goals and targets of the agenda 2030. I think it's a very important approach. What we are doing, we also want to involve various stakeholders and societal groups in our evaluation. Some of the issues that we are evaluating are more administrative, how the SDGs have been combined with performance budgeting or performance management processes. Of course, that's the administrative issue. In other areas, we also carry out several interviews and we organize a number of stakeholder work groups where we want to hear the opinions of various parties. It's included. Thank you so much. Brilliant. If I can turn the discussion a bit at the sub-national level and with other sub-national stakeholders. Anna, building on what Dorothy brought up in terms of the global scene and capitalizing on what Petri just said in terms of the use at the national level. What about the sub-national ones and sub-national actors? What can you bring to the discussion? For sure. I guess we should first recognize that there should be an understanding that all the different levels, say the local, the regional, the national and the global, they are all intertwined. Overlooking the local realities would ultimately mean losing important perspectives, experiences and drivers of local action to achieve the SDGs. As an example, there are colleagues at the United Cities and Local Governments which is an umbrella organization focusing on the local level. They have been analyzing and reporting on local efforts, meaning efforts driven by cities and by territories to achieve the SDGs. They have been doing this for years now. A lot of learning comes from sub-national and local processes. As you've mentioned, Cassin, we see that more countries are mentioning the voluntary local reviews and their national reviews. They're mentioning BLRs in the BNR reports. This is a very positive trend. It directly relates to SDG11 on sustainable cities and communities, but this is also positive in the sense that the BLRs work as an instrument of local efforts and as such they pave the way for sub-national accountability structures. Some examples from 2021 VNRs include Colombia that published a guide to voluntary local reports for the cities and municipalities of Colombia. Another example is Japan that established a VLR Lab, which is a platform that collects worldwide information on VLRs. For sure, this topic is really picking up momentum in several places in the world. Thank you, Anna. I think you've raised a couple of interesting examples of local and sub-national initiatives feeding into counter-led evaluation of the SDGs. It would be wise to reflect on the main big lessons at the different levels. Probably if we can pick up on a couple of lessons from the global level, Dorothy, if you started there, and then rolling it out at the national and sub-national with both Petri and Anna. Yes, indeed. I think the role of the development partners, particularly when they work together, can be very powerful because there is a level of resource there, which if it's combined with a country-led approach, that actually helps to move a common agenda forward. So if we take a few examples, for instance, of the Asian Development Bank that's just completed an evaluation of its approach to climate change, this was a very large evaluation that engaged a number of different countries and particularly the Pacific region as a region and looked at countries there which are really heavily impacted by climate change. So in relation to SDG 13, even though the evaluation was really looking at the Asian Development Bank's approach, it brought together a whole range of different countries to talk about and reflect on their own progress. So there was bringing together country evaluations that had been done, project evaluations, all looking at this approach to climate change. One important lessons from that was that the decision-makers in the Asian Development Bank and in countries were engaged right at the start of that evaluation to really look at the key evaluation questions that wanted to be answered to then feed into decision-making. And that has been successful in that ADB has now intensified its approach and is working across different countries to work with them on a range of priorities. The other good example here is the MOPAN network, which brings together a whole range of bilateral development partners who contribute to a central fund to do evaluations, not just of their assessments really of development partners, but they also bring out some of the themes that Anna was talking about. But I want to raise one other point that isn't always considered at the moment, and that's of the need to involve national evaluators, so people from the particular country of focus, and even sometimes subnational if there are different languages, if there are indigenous peoples for instance within a country or not even within one country but across countries, that those kind of approaches haven't necessarily been picked up enough yet in that link between national evaluations, global evaluations and subnational or thematic evaluations, the integration of language and culture so that the findings of evaluations can be contextualized to the people that really need the lessons arising from these evaluations. Brilliant, brilliant ideas Dorothy. I think you've brought into the picture a couple of very interesting aspects revolving around peer learning among countries within a collective evaluation, engaging decision makers early in the process and even designing and scoping it. Doing some evaluations, more thematic evaluation and most importantly this aspect of localizing the evaluation. And on that point probably I'll pick up on Anna. Given bringing the voice of the CSOs both in the VNRs and evaluations, what can you share with us probably picking up on the localization of the evaluations in such a context? There are many lessons learned in this sense as well as recommendations. So I could probably extensively list recommendations around each one of several topics related to West Digi's implementation and evaluation and what Dorothy has just said is tremendously important. So a key takeaway would be in the sense of the need of working together with non-state actors. So what you've mentioned Cassin of including CSOs but also being mindful of Indigenous groups or those groups who are in many different cases overlooked by governments at different levels. So I would say that there are many actors civil society organizations and other experts as well like the academics that have been collecting that have been producing their own evaluation processes and presenting very comprehensive reports that highly contribute to country level and global level review processes. So with this collaborating with civil society and other groups and advocating for their reports to be acknowledged and given status in the high level political forum for example would be important in terms of one participation and partnership coordination also in terms of transparency in reporting processes and third also in terms of ensuring accountability for progress of the 2030 agenda and STG's implementation and evaluation. Thanks Anna, I think you've shed some light on aspects that are critical to both the DNR process and the counterweight evaluations and we know that evaluations obviously don't happen in vacuum. There are politics definitely but most importantly also there are risks being them the national, subnational as well as global level. If I can turn to Petri with a bit of reflection on how does this current counterweight evaluation in Finland is integrating or accounting for those potential risks. Can you shed a bit of light on those aspects Petri? Going to the risks and the future challenges I think our national commission on sustainable development is doing an excellent job in coordinating all the activities but of course these are very complicated and complex issues where you face immediately the question of the systemic impact on certain things and that is something that goes beyond traditional policy coherence thinking because if you really want to understand the systems change in the field of Agenda 2030 or STG's you also have to see more strongly how these various goals and targets are interconnected, how they influence one another and you cannot do it if you tackle them separately. And now what we have been doing an excellent job in dealing with these issues separately and we have very active ministries who have done a wonderful job but when it comes to interdependencies and systems change and systemic approach there we have still much to do. I think I have to mention that the STG evaluations as a whole have created much closer cooperation between ministries especially the Ministry for Foreign Affairs and other line ministries because the Ministry for Foreign Affairs has a very strong evaluation unit and now the STG's are connecting the various line ministry like the Ministry of the Environment, the Ministry of Employment and Economy and so on together with the Ministry for Foreign Affairs which also gives a more global approach to our national implementation. I think it's a good thing whether it's intended or unintended it doesn't matter but it can be seen as a positive outcome of a process. Indeed, often unintended consequences or outcomes turns out to have much more impact on the change that we all aspire for. Probably on that point, on the points of risks and the era we're living in we've all noticed that there has been a slowdown in terms of the progress on the Agenda 2030 and I'm afraid that this slowdown basically is affecting also the uptake and the scale of evaluations of the Agenda 2030. Quick remarks probably or quick reflections Dorothy if you can if you can shed some lights here. In terms of the slowdown of the 2030 Agenda this indeed is a big issue and one that evaluation as a sector needs to consider and move towards. Evaluations take too long to organise and present at the moment but there are ways around that that most evaluations that we're doing at the moment have some element of consideration of this slowdown they have some consideration of the impact of COVID. So these, what we need is more dynamic evaluations which cut across and draw out some of these risks and the responses so that the learnings can be generated quickly and fed back into programming. Of course, thank you so much. I mean that's critically important bringing the aspect of dynamic evaluation. I think we're coming to an end now and we want to probably conclude our podcast. Can I ask you please to formulate and give us at least one thought for our audience on the value of those counterweight evaluations to inform the stakeholders about the progress on the Agenda 2030 one key takeaway message from each one of you before we conclude. Let's start with Anna and then Patrick and then Dorothy. I just wanted to quickly circle back to what you have said in terms of challenges. And you mentioned a couple of them beginning with C, like COVID-19 climate change conflicts. And I just wanted to add another one which is a civic space and how it has been shrinking in many parts of the world particularly after the pandemic and how many countries started using the pandemic as an excuse to further close civic space. So it would be worth checking a document from the OECD DEC which is the Development Assistance Committee a document they launched in July last year which is the committee's recommendation on enabling civil society. And so this is a common standard for official actors to address civic space and work with civil society towards the 2030 Agenda. So a key recommendation in this sense would really be for countries to not only adhere to the recommendation but also embed it into their practices. And one big concern is that a worrying number of countries is falling short in achieving the global commitments and even to adhering to the UN Secretary General's guidelines for evaluation and reporting. So we have eight years left until 2030 so governments and stakeholders from the global community they should redouble their efforts to address gaps and work towards achieving the SDGs. So I would leave as a final note that there is opportunity for this to further strengthen evaluation and reporting and the way of doing that is making use of comparative analysis from using data from other reports including civil society reports and including peer countries reports so keeping the dialogue open is a major key. Brilliant, brilliant idea Patsy. I think it's extremely important to carry out country land evaluations, the national evaluations because what we really need is evidence for our policymaking and I'm not referring to a very narrow evidence-based policymaking but I'm referring to the more evidence-informed policymaking where evaluation is an extremely important tool for giving decision makers updated information where we are in the certain fields of Agenda 2030. That's the first thing. But second thing is that evaluations if they are carried out in the open and participatory way they are also excellent tools for involving new groups and stakeholders in the process and we saw that in our previous national evaluation carried out in 2019 and one of the key factors was also the participatory approach applied in that evaluation. So I think that's a very important issue and maybe the third one is that when you carry out national evaluations I think it's important to apply somehow the developmental evaluation approach meaning that you are not only collecting data and then reporting at the end of the process but it's a dialogue also between the commissioner of the evaluation and the evaluation team and the stakeholders so you feed the process while you are doing the evaluation. I think that's the lesson learned from Finland. Thank you so much Petri, I think you've shed the light on a very important aspect of course. Dorothy I think if you can conclude with one or two takeaways that you'd like to highlight. Yes, I'd like to build on what Petri mentioned there about the importance of inclusion and participation that if we're talking about country led who's actually doing the leading? Are we talking about country leaders? Are we talking about only one sector? I think we're not. We're actually talking about a broad based country led approach where there's a reflection of the diversity in that particular country or even in that sub-national area. So while country led evaluations of the SDGs are very large and difficult to achieve there are digestible chunks that we can move on with very quickly which is the value of country led inclusion in evaluations for the SDGs so that we can have something that's very manageable of changing the way we do practice of any evaluation to make sure that we are touching base with the key sectors of society within that country. Those who are impacted by evaluations making sure that they are contributing to the evaluations in a way that brings value and brings learning on progress for the SDGs and it identifies innovations and accelerates progress. Thank you, brilliant ideas. I would like to thank you so much for your deep reflections and your takeaway messages. And I want to, before saying goodbye to our audience I'd like to call upon them to sign up for the links and sign up for the different episodes that we've created under this series and to also announce that this network and other partners including Eval partners are moving towards initiatives to reinstate government commitments to the evaluation of the agenda 2030 under a new resolution that promotes country led evaluation and discussion is still there and I believe what we've covered so far feeds and comes timely into that dialogue and those efforts. So thank you so much and final call for our audience to read, check and visit the website of IAED, Eval and Eval SDGs and other social media platform and share widely those episodes and hopefully we can talk and we can initiate new dialogues on new issues around evaluation of the agenda 2030. Thank you and goodbye.