 Thank you so much that after a marked long day of working, we still move to this panel discussion. I really appreciate that. And I think I can promise to do a very very interesting discussion. First, we may welcome also people who are not participating. Who is here from outside? They are not in that participant. Wow, welcome. Thanks for coming. Also the weather is so nice that I need to have a nice view. So thanks for coming and joining. This is the only ethnic event we integrated in the ADAP program. And later on we have some reasons to be here, some why it's so weird. And especially the guests who don't know so much about the program can talk to participants and exchange a little bit with them and connect with them. So you can have a nice talk later on with a nice way of working here. So this is going to be very, very last week of the event here. How old is that building? What do you think? 18, 34. So let me introduce you to our host today. This is Cady Andrews Johnson. She will moderate. She is working at the independent organization world-wide. She is responsible for the relationship with the developers. And I would like to dedicate this evening to the IEGN because they are sponsoring the APRO afterwards. So thank you very much for that. And I would like to turn it over to Cady now. I would like to introduce the panelists. And I hope you all enjoy the evening. Have a nice and interesting discussion. Thank you, Cady. Thank you. Thank you everybody for coming. Thank you to our new gift partners for organizing this event and inviting distinguished panelists and panelists as well really looking forward to having conversations. Let me just start out by saying that development evaluation has long been dominated by donors. Northern institutions and the northern consultants know as evaluators down to staff to evaluate programs and projects. Northern evaluator experts flying in and out to train local evaluators. We have paralyzed the importance of building an evaluation capacity development in developing countries. And now we have agenda 2030. And this places the responsibility of NMA at the country level. Yet at the same time there seems to be a gap between commitments around this agenda and actual investments in the evaluation capacity development. Meanwhile, at the same time, the demand for ECD has really grown and changed this higher demand for the developing world and is a focus on South-South learning. Our discussion today will explore with our distinguished guests, we'll explore the topic, the role of the local South in evidence for development. So let me just start by introducing briefly our panelists and then we'll move on to some questions before we open forward to the audience for discussion as well. First of all I'd like to introduce Adeline Sabanda. Adeline is the president of AFREA which is the African Evaluations Association and IOC and as well she's currently the chair of the Steering Committee for South-South Cooperation and Evaluation Initiative. Brenda Barbour is the manager of Knowledge and Communications at IOT's World Bank Group. The strengthening of this function in IOT has been really important to enhancing the influence of IOT's evaluations. Then we have here Honorable Ananda Kumarasiri and he is the deputy speaker of parliament at Shalaka. He's also the chair of parliamentarian forum for evaluation. He's a very strong advocate for building national and many systems and processes in Shalaka. Welcome. Peter Vila, I hope I said that right. I've been practicing. He's the head of evaluation and control and department for the Swiss HIST for Development and Cooperation, SDC. An important donor and also in the space of evaluation and capacity development, particularly for this event, as well as the fear initiative. Just putting that in, thank you. Peter's been based many years in Africa as well, working six years, I believe, as well as based in the headquarters in Burma, SDC. Stefan Riga, I also practice that one, is the vice president of SAVAR. Now this is the Swiss Evaluation Society and he's also the CEO of Interface, which is a consulting firm. He works a lot to help the public agencies to plan and implement reforms. So let's get started. I'll jump right into the questions. We'll have about 45 minutes for some questions that we've prepared and then we'll give the audience time for Q&A and living comments for the next 30 minutes on how to be a payroll. So my first question. Is there a cultural difference between implementation of evaluation in countries of the global north? The number has to explain the differences between the global south and the global north. And the differences are really shaped by what the experience is that the global south and the global north are one group. And before I say what I'm saying, so I want to make this clear, that these views do not represent IOC views in the 1930s and 30s. So the differences between the global south and the global north, we need to understand the history of the global south. So when you talk about cultural differences, in implementation, we must note that what is specific about the global south is that it is likely saying that the logic has to be done really by the north and driven by the north. But at the same time, when we talk about the global south, we are referring to those countries and I'm going to quote so that we are really thinking about and I'm going to quote from and the same the professor was telling us that we should be talking about histories of global colonialism near realism and the difference between economic and social change through which large inequalities in this country like it's going to be and accessible resources are maintained. So that is the global south. So I'm going to say that based on those histories and based on the different experiences, practices there is actually we need to actually look at the development differently. So it really becomes imperative that evaluators as we implement our evaluation, we must reflect on the cultural implications and practices of our practice and there's also implications of post-colonial development and also the theories that are coming out of post-colonial experiences and practices and now I'll read it to you. Thank you. I'd like to ask Peter if you'd like to share your thoughts on this having looked for a long time and worked also in your own asset about this. Thank you very much. Since this is a panel discussion of course I'm going to challenge Siddhartha and say no, there's no difference but just to launch the discussion as well because I think basically the standards evaluation standards are the same. So we are all committed to do evaluations according to the same standards, regulations and I would say professionalism that we can think about in evaluations. Where we might have a difference a cultural difference is maybe in the use of evaluations. The use of evaluations that is more in your present environment my own role to be a controller as well. The understanding of evaluation to be a controlling instrument is completely wrong. It's not a controlling instrument and it might be that there's none and I don't want to say there's a difference between both the law and the result but the use of evaluations to be misused as an as an instrument for political decisions rather than for learning and for make strategic positions there is probably a big difference between cultural between different environments. We'll go to the next question I'd like to also hear from you guys and also understand how is also changing a bit from very focused on accountability to more learning approach. So that brings me to my second question which I'd like to post Brenda and Peter. How are development evaluations implemented in partnership as an equal process or are there still gaps? If yes, where are the gaps and how can they be overcome? Peter would you like to start on that please? But you can do the call. To be very honest I think every donor or every every actor is designing his own project and his own implementation and then the core development that's also its own its own evaluation process. I have seen in the past several projects that we jointly developed where each and every donor was evaluated the same project individually in parallel and not even thinking about doing a joint evaluation. And that's joint funding partners. Now increasingly and at least in a little bit of discussions I follow that with receiving that. We are talking also about joint evaluations with partner countries and how partner countries can be included in evaluation processes. And this is a still a lot of process and I think really not doing very well with it to be very honest. But obviously there's an opportunity in the past to say there's maybe also a different understanding of what evaluation where it is all about to move on and this is something that I will hope that we can in future develop a little bit more. And I would like to share with IG's experience of Sure. So, IG in the operations and policies of the World Bank Group that our staff lead our region evaluations and our operational evaluations. But they do use lawful consultants to support them and to contribute in those evaluations. And the World Bank Board has encouraged us as part of our evaluation capacity value to use lawful consultants. So we've set a target for these lawful consultants namely to the Board. And we're not FI18 target, but it's not enough. And so Adelaide and I have already talked about how she can help and be accommodated to have these lawful consultants in the evaluations that we're taking out. Thank you, Linda. So, I'd like to now turn to Adelaide again. Adelaide and all good evaluations that fully reflect the knowledge, realities and philosophies of the global staff. Thank you. The first thing that I would mention is really a particular level. It's important that we use local evaluators because they know the context and also the culture of the environment in which the evaluation takes place. That's the first thing. The next thing also in particular level is also to act the tools and methods to ensure that they are completely culturally responsive. So that is a particular level. But we also need to make sure that the knowledge and theories that are coming out of the staff and try and experiment with those theories in the environment. So, it reminds me of one of the teachings that I was giving when I said, you know what some of you wanted to be on, you know, the theory and say, you know, I'm not comfortable with the theory on why the type of knowledge you're looking at and you know, I'm not interested. It's very, very good. And it says you must be careful with those theories and, you know, so it was a learning. But we must be prepared to test and fail because these are not to do that. And we must be open to have that experimentation and also in design. And so the next thing is also to be sure that, you know, once we find that the reasoning about the global sound is not taking risks. We must make sure that we have, you know, high-profile position, team leader, or, you know, not to care as much as it is because in most cases, your name doesn't really matter on that question. But also we really need to look at how, you know, our knowledge must be more integrated by looking at whether the programs of quality are affecting the social network because we often feel that whatever we prescribe or the way we design a program we just look at our results and say we wanted this to happen we wanted this change in the media but we don't really look at whether the program is affected the social fabric because it's very important that we look at that. So for me, those are two ways that I would actually add that to my question. Anybody else want to comment on this question about how can valuations be implemented that more reflect the knowledge and reality seems to us? I mean, the one thing is the use of valuators and that's a challenge to find to locate the valuators that's something that we like to start with and then also to represent the perspectives and maybe the local customers in implementing valuations. Any questions? I mean, the two points I would like to raise first is in the topic of implementing valuations that the evaluation starts by designing it and when designing the evaluation then you have to bring the local staff in control and in order to really ask the right questions in order to really target anything in a valuation property the second point that I would like to raise is the the data and I'm very often shared with my colleagues here in Switzerland that are doing evaluations about Swiss education contracts for example. The data base that are available as a background is much more it's much better than any data base that we can find in those and this is an appeal for strengthening also the statistics capacity in the south in order to have a good and solid data base in order to base evaluations on a good data because evaluations are very often very quality and we may have that later. Okay, good. So let's turn to the role of vocies and let's hear from both the North and South. What is the role of vocies in different partnerships for evaluation and culture? Can we start with you, Stefan? Yes. I felt a little bit more comfortable with these questions because I cannot say much about the past so I was glad to say nothing about the North and say perhaps some small things about the South other questions what we can do to build up a self-development association where we start first with vocies and evaluation culture and we have to bring up with our sentences evaluation culture for years going forward in order to take the policy that we should agree that we are finally beginning with a policy program and that policy program can become one of the future of vocies because of evaluation culture. So what we can do to raise further on is to definitely get a role and we can contribute ways to develop this culture. In the very beginning we started with a service program and the first role was to develop a perfect team of institutions prior to their work. So we have to realize of the county the settings that we have to manage some guidelines standards that we feel or that we feel we talk about the same we talk about the television and now we have a situation where we should go around building up quality recruiting of the older individuals which we are not and we have to build up this also in relation to the south we have a work group which is dealing with that dimension and we have to have a system of guidelines and what we can see is that this is an important role which I think is important is the independence and what you said is actually important data but sometimes we have also we need further understanding of how to be different and then we can start a collaboration these are two different two different things called version here Thank you very much Anyone else like to comment on this version of independence being less current generally as a generalization in the south as opposed to the any ideas or thoughts on the question they say they are all people fighting and coming through the evaluation why are they saying I think it is a solid fact because I believe that in the case of all those evaluation it is very much like saying because we are fighting for a short time and the environment can understand the process and the culture within so they missed out with the positive connection because of the contract so in terms of independence it is very much and our current it is for the progress because it is now in the generalization so we need a different definition as so far what I see is that people are fighting different people who are programming a very different definition but surely there are some normal evaluations and evaluations that can be done in the south as well and any other ideas that are possible to be around a function of evaluation which is usually that sort of that independence but really that is not a fighting independence I mean we heard about also making that is the basic how we understand evaluations what they serve for and the reality becomes a little bit different I can observe that in the recent years evaluation results have been increasingly used for positive allocation meaning the purpose is like the difference it is probably an award mechanism so the ones that are doing evaluations are supposed to do good evaluations they know that the money continues to flow and there is a big danger in misusing evaluations to allocate funds and I am not talking only about development cooperation but policy making in general particularly you might have a particular view of that when parliamentarians take evaluations to decide about actual values and that is a very dangerous thing so actually let's move on to the question around parliamentarians actually it is sacred this is a question for another are parliamentarians best decision to use evaluation lines and recommendations when deciding policies and ways of evaluation let's just ask we are most responsibility of our society in the Asian community and we advocate the members to do better for our country so lastly we have three names one of them is representation of the people who we have created we have a real voice in parliament Oversight Oversight Oversight in our parliament we have no Ossetian reasoning for each one which we have earlier we have a system of Ossetian and it is really not of a specific level mostly in the parliament in India they throw a real gun and the system is based on the Ossetian in India it is really not of a specific level so they have the authority to question that how money has been used in that area and are there pros and cons so this is a reason it is not only somehow ensure that these things are done in such a way that if we focus on Ossetian very much Ossetian we have a power to engage in any practice so that is our mission we decided on Ossetian but there is no need and with that we have managed to make a change thank you so really we have a lot of experience around using evidence to support the role of accountability and Brenda I wanted to ask you a question on the topic clear has been built in capacity of the legislature across Africa for many years now can you share with us some of the emerging lessons the work is about sure can you example of how many of you have heard about Ossetian so we have some of those in Asia, South Asia and many of them have been part of Ossetian for over three years so then we start and training supposed staff and researchers to be in support of part of Ossetian even as part of Ossetian by moving in and out of other roles and now they actually realize that Ossetian is more institutional so they established partnerships with Atnode and many of us so that they are having a promise on building any systems building any processes to support the public who they switch from in the music evaluation and decision I would like to I would like to I would like to I would like to when I give more flexibility they might not be much and so what I am is that they need to train the whole Oss if you are going to make a difference if your working with Atanode and they can train you to so that we can It's really key and there's also a lot of the aspects of the app. Because I believe in some of the things that are available for buying. Clearly, as for example, the management of the app, or maybe as we do not present all the mechanics, this is a lot of variation. So, we need to find ways of making sure that every component of our mechanics can change. Thank you very much. We're getting to your point. We want to make sure that every component that we produce that goes out of the app, is easily used by the different users. So, it's very important that we think about what is going to use the evidence that they're producing and how it's going to work. So, for example, we need to take a few hundred times to use the evidence that's coming out of our evaluation. We need to calculate it in such a way that it's at the same, you know, a specific topic, or a specific evidence that is relevant within that specific environment. So, really, there is a pattern, this pattern pattern is visiting everything, giving people to say, can you find a different kind of pattern for different users at the end of the evaluation? Excellent, my friend. Do you want to add that? Yes. The main answer in one sentence is as we found it, there's a lot we can do to increase the use of the evidence that we generate. The main thing that we need to think about is when and how we may get out of this bill. So, we're going to give you an example. At IT, we need to do these native evaluations. I must know what it is that we need to think. We synthesize what do we know of our evaluations that we should. And we send that to the manager of congratulations. Congratulations on your appointment as the organic nutrition manager getting along there. Here's what ITG knows about along there. And that has received so much positive response because it's in the moment that we need it to have engaging more with the people that are back on their evaluation. So, of course, it's important to look at several processes. Doing an evaluation and making recommendations that are discussed at the board and their management counterparts develop an attitude in response to ITG's evaluations. And that's a very... because ultimately what we want is for the person who's going to re-buy into the algorithm not just a trivial action plan that they can report to the board. We're talking a little bit more about new cases of SAP, the value that was early on in the design. I think this is one of the things to do a formal process also. We have our approach to a bill about evaluation and questions. There are certain points in the process that we check in. And we can obviously agree to disagree but there are the attempts to try to cover that up. And then we have our independence that we have to sort of protect and promote the evaluation that would be a continuity question. But we've also launched new products, which we will say if you think about the learning products, these are actually not accountability products but they're actually collaborations that the board has done jointly with me and ITG and operations staff. So learning is actually added to ITG's mandate. So before it was never explicitly said that a lot of things we were supposed to do is have learning come from our evaluations. But now it is. So there's a comment that we've got that one of the things we launched that happened here is that are learning invasions. So ITG's taken lots to do new evaluation challenges that we can have evaluations that cannot be forwarded on the rest of the challenge. They can apply for a learning invasions grant and get one or two more together on the evaluator to believe what they can learn from the evaluations and we need to start taking our evaluations. It's so positive that you see that all of the evaluations prove as mainstream evaluations so we need to think about them. What is missing in the use of evaluation evidence? The two points are the cost they're cost-effective and they need different points. So we have very often we're tempted to put just anything in any evaluation. Meaning pages and pages of questions. I've seen those of us it's a general illness so to say. We try to put just anything in evaluation. But then of course we ask the evaluators please produce a report of 20 pages. Now what can we learn of 20 pages that has to respond to 15 pages of questions? So let's be realistic also when we design evaluations. Let's make it small and the smaller you get the more targeted you can focus in any evaluation the more you get also as a return of that evaluation. Because expectations are high people want to and they want to be involved in the evaluation process. That's what we call in SDC internally we usually have a co-learning partnership including staff from different entities that are involved in a particular topic that are following the whole evaluation process but not influencing the result of the evaluation of course. But then the evaluation report comes out with every general recommendation for considerations I've experienced. It would be absolutely very nice. Absolutely excellent point. I just want to add something because I'm going to get on so fast because I put on the back I was too supported the IDG page one on our major evaluation reports. So IDG standard evaluation reports can be done more than 50 days. And the board nobody said the evaluation report is the only thing you can produce to communicate with people who you want to learn from your diary. When our major evaluation needs to reach that. So yes, this is the you know how to guide for these people. Do we need an intro product for these people? Check out the daily visualization on IDG website. Do we need to have a workshop with these people to discuss more deeply what they can learn from our evaluation? So the evaluation report is one out of a great tools that you can use to communicate with people that you want to use your evaluation. One more thing is we set aside the thing about the only reason you're doing an evaluation is for somebody to learn from that or somebody to use it or somebody to make a decision about it. So, think about that the money you've set aside for outreach is not a tax. It's not, oh, I can't do all I want to do because I'm second on this out of the outreach. It is the thing that will make your evaluation rich to people. Thank you very much. Yes, please. As a comment, I was listening to this gentleman what I say is spend years and years on the base of current series and then once the budget is passed and then the spending time spending money and then getting the budget is over and then when we in Sri Lanka and this is, we don't get annual reports at the point which is about three to four weeks by the time we get those reports most of the people that we can do results in regard to independent education or the departments. Nobody has come so far. So, what we can offer now, apart from we have in the system we have the public technology and the limited public enterprises which are common enterprises as well as departments where we have a evaluation office where we can get as they said brief reports on evaluations and also if these annual reports are based on capacity system where annual reports means that the department at least one year after which are implemented I feel you can do value to public funds more than countries now in a way especially in Asia. The tiny effects of relevance and that's you, Stefan. There are two points first I would like to confirm that the relation we are going forward first of all we should ask ourselves are we able to complicate in a way that we can understand and still that the office is not the case to complicate to complicate the right this is the first necessary step there are two additional points one time then you would like to use evaluations as a decision making basis so think about that and have this discussion for example if you own the apartment if you basically can't stop the the bigger of the opportunity so you will not have any issues second one and third one you mentioned that before the universal standard there but in my practice nobody wants to give the resource in my practice the evaluation job ends when we give the report to the one who needs the analysis and at this point I'm not prepared they are not willing to spend more money for those who needs the analysis they can do that they have much shared results they can do follow ups they can do group discussions and the people who jump with the programs but in 95% of the majority get down 200 elevations in the land of the state in 95% no follow up balanced by elevations and the people jump