 Also the question, it's a bit of a connected question, so I was wondering and you answered that whether you had the experience of having these actors involved earlier in the planning phase and what would change, but John is also asking because of course the tension is between finding a balance between the short-term gains of the business versus the long-term investment by business. Yeah, I think the question is how we can align the motivation, short-term or long-term of the business people with the long-term vision of the planet, so that's the big question. But the thing, the authority, a player key role here to follow the vision set by strategic planning, so they have to create the more backstage because the solution, the planning objective is not only what we want, but also what should be done to include more appropriate stakeholder involved, that's the very important thing, so that means our solution, that not only what should be done, but also how can be done and the how is even more important than the what. So I think there is a many examples of how to do it, for example in Vietnam, many provinces try to attract investment from the FDI, a foreign direct investment into their province, so they try to improve their governance system to be more transparency, more effective, or they can build some vital infrastructure, highway or port so that they can have the business to reduce the cost, or they can prepare the human capital to provide for the investor. So a lot of things can be done toward this, but the final destination of such a plan is always how to attract the business people into the planning. So depending on the situation, you can think a lot of, you know, you have initiative, you have policy, you have incentive methods so that you can do it. Okay, thank you. I think that that's also a very, let's say hot debate also for example now in Europe with the Green Deal where we're really at the face to try to involve businesses to transform the economy of sorts. And I think it's also a question of what type of business you want to involve in the long-term strategy, because of course some of them do have a very long term. In fact, you know, in the planning phase, we have really some actors who play a role of leader according to the motor map. But if you try to involve the people and like the business people at the very beginning, you may fail because they don't care what you are doing, what the care is about the plan. And if they see any opportunity in your plan, so they react. That's the trigger. So anyhow, you have to be patient, but you have to think of the dynamic of motor. Let's not be the same every time. If the objective change, if the actor change because of a new trigger is coming, so you have to reanalyze the situation again. The motor mapping is maybe not what has been done during the planning phase. Okay, thank you very much. And Leon, I think it would be beneficial to come back to the role of private actors and entrepreneurs in our later discussion because I think that's a really interesting facet on how do we get private actors interested in what we do and also how does government then facilitate that process to be long term versus making a quick dollar. I think that's such an interesting concept for us to try to look at. Yeah. So the conclusion is that if we managed to include also the implementer, the business people at the very beginning phase of the stress planning that may help happen a lot. Yeah. Yeah, interesting. So indeed, let's pick this up in the discussion. Do we have time for one last question? I think so, no. So Manmang is asking how you measured or how do we know the better abilities by sector? Yeah. Any idea about this? Yeah, I think it depends on the nature of the objective because anyhow, we have the only four categories of ability and depending on the phase, depending on the nature of the problem. So I think it's not that difficult to find out which one is relevant and which one is in a higher priority. For example, in the agriculture and maybe different from the industry sector. Yeah. Because in industry sector that require more ability in the market. Yeah. That means you have to understand not only about the financial matter, but also how to how to produce, how to organize the operation and a lot of skills that you require. And but for agriculture, if you are a farmer, that's might be easier. You can just produce it and then that's all what you know require is very simple. So I think that it's case by case. Yeah. It's not easy to find out. Okay. Thank you very much, Professor Phi. This brings us to the next talk. So we're going to now discuss another aspect of extending MOLTA, mainly monitoring and evaluation and it's our colleague Yav Aver. So we'll be talking about it. Yav is working at IGDOT and his research and teaching focuses on water environmental policy development and implementation, as well as policy transfers. And so his talk is about monitoring and evaluation and whether MOLTA indicators can also be part of that monitoring and evaluations. Yav, the floor is yours. And thank you everybody for having joined us here today. I hope you can see my screen. I assume that's a yes. Yes. Okay. Thank you. And also thank you for your interesting presentation. And I think you have made some remarks during your presentation that are very useful also for me in this presentation, because I think I can build upon what you have presented. For now, as a starting question, I think it would be nice for me at least to ask you what are you using your evaluations for? I'll just start thinking a little bit about that. So what do evaluations mean to you and when are you using them? And well, do you find it meaningful, of course? Let me first go to what I'm going to present to you. And the content of my presentation is first that I'm going to share a little story about what is currently going on in the Netherlands. And then, well, basically, we address in this presentation the question about what is evaluation, when are we evaluating who is evaluating and how, of course, is the MOLTA framework helping us in evaluating and improving our projects and plans? Well, currently, there is a big parliamentary research commission going on, and it has as the topic the implementation agencies. And one of the issues that started this parliamentary research commission is the Dutch tax office child care benefit scandal, in which basically the tax office has unrightly suspected all kinds of parents from fraudulent practices in relation to receiving their child care benefit to support them for having their kids at the daycare. And this especially had impacted on citizens of lower socioeconomic classes. And the reason why the tax office had addressing those kinds of groups of people as being fraudulous in their practices to receiving child care benefits, again, related to the automatization of the IT software in the Dutch tax office, which basically already hinted towards that in those classes of people there are more fraudulent practices, but the algorithm, well, got too excited and pointed out too many people as fraudulent, and it had a very dramatic impact on those people. On the right side of this slide, you can see an article from the Volkswagen newspaper, and basically it asks here itself, why are citizens and civil servants so often lost? And it addresses, and also in this commission, this research commission, it addresses this question, why are implementing agencies, civil servants, not able to implement what is the plan, what is developed by the policymakers, but also why are so many citizens not able to adopt those two policy behaviors? And when I have been looking at the interviews or the hearings in this commission, several elements I did notice. So when we are looking at the implementation side, there are many of the heard people in this hearing mentioned, like there is a lack of knowledge and attention by the policymakers, in this case members of parliament in the Netherlands, but I think we could easily translate it also to policymakers in city councils or whatever, or at state level rather than federal level. So there's a lack of knowledge and attention of policymakers for implementation, so they are making big policies and do not very much care or are not really aware of what this means for implementation and if it's even implementable or not by the implementation agencies. And in this case it relates to the tax office for example, but also a program of the Ministry of Infrastructure is part of this whole research and hearing. In the Netherlands what we also see, but I think this is also an international phenomenon, is that there are also under neoliberal ideas as well, is that there is a process of decentralization going on. And so what we see in the Netherlands is that around this issue, but also in the issue of healthcare, is that there is a deep centralization of responsibilities pushed down from central government to local or regional governments, but it doesn't come with additional financial resources. Actually in many cases we see that in the past 10 years there had been budget cuts on it. What I also recognize or what the people in these hearings recognize is that in the political debate and in the development of these policies or these larger programs, there is a need because also in the Netherlands we are dealing with a multi-party system. There's a need to develop consensus around the policy or the laws or the plans. And often that consensus also results in unimplementable policies, what these people say, because there are sometimes to come to a consensus conflicting measures or goals are actually included to keep the different parties happy. And finally, what especially comes from the people heard from the implementation agencies is that there exists a lack of trust between the ministries and the implementation agencies. And that has then resulted in that there is no feedback from the implementation agencies back to the policy makers on giving critical feedback on the unimplementability of the policies and plans that are developed at those ministries. When we look at what the people have been mentioning from the social adoptability side of the question, so for example the Ombudsman was interviewed as well. And they mentioned that there is in the policy development, the policy makers in the Netherlands currently very much rely on the idea that citizens are self-reliant, that all citizens are able human beings who are able to understand the content of the plans and policies and that they are also able to find their ways into this policy domain. Rather, what we see and we've also seen that very much in this childcare benefit scandal is that actually many of the people who became victim of this in this process are actually not fully self-reliant. So they are not IT literate and that nowadays is quite important if you want to deal with the taxations, they do not understand the tax forms, they don't find their way to find judicial support from lawyers to address this. So they become actually a victim of the system and these people are often not empowered. What we also recognize in the Netherlands in this case is that in many implementation agencies they don't see the client relation as the client as the citizen but actually that they rely on their client relation towards the ministries. So the ministries are asking from them they give the assignment so they are actually working for the ministry rather than this idea of them being a civil servant and a servant to the people. So also in that case when we think about evaluations and how they report back it's much more reporting back to those ministries rather than to reporting back to the representatives of citizens which should be parliaments. Another critique that comes from it and it relates especially to the automatization and IT developments in in these agencies is that there is actually a loss of tailoring to the needs of individuals and this is of course always a big dilemma between especially in large plans which are generalized to some somewhat is defined as a general public but of course we see that problems arise in specific parts of the public. For example these people of lower socioeconomic classes who are less educated lower educated are not really self-reliant we see that the problems arise often in those groups. So and basically what it means is that these implementation lost their human touch there's no human relation anymore it's all optimized people are addressed like numbers rather than people and they will get standard responses from these agencies when things go wrong. So what are then some of the suggestions and I've translated a little bit to our MOTA thinking but I saw that quite some people in those hearing actually respond in terms that very much fits our motivations abilities but also the implementability adoptability thinking. So I saw for example one of the members of parliaments who was really saying straightforward we need to have feasibility assessments of plans and oh sorry we need a feasibility assessment of plans really focusing on the implementability by agencies of government and they are actually really proposing that parliaments should invite the implementation agencies in the discussions when new policies or programs are developed so inviting these implementation agencies who in the end will be responsible for the implementation into the development phases of these policies and plans and also what they notify and that's especially also something from a message from the Dutch ombudsman was that parliaments really should move a little bit more away from this idea of people being self-reliant and empowered and really taking into account the ability of citizens. So I think that when the final report from this commission comes out I think more or less some sort of a conclusion like this will come forward if based on my personal assessment of this here and so it shows that it's really meaningful and also fitting in the time we are living in that we are thinking really about balancing the feasibility of plans in the balancing the implementability of agencies but also and the adoptability of citizens. So well let's now move on to then thinking about what is evaluation and why when and how and who is doing it well if I take my standard definition evaluation is the assessment of the object of evaluation in this case that can be a policy a plan a program a process the effect or a result of an empirical observation of a current situation but it's always based on a systematic acquisition and assessment of information to provide useful feedback about the object in our case that would be the plan. But evaluations come in many sorts and many forms you would see that there are evaluation frameworks that focus more on the management I think the log frame approach for example is a well-known form of framework for evaluations but also we have these gun charts and these spurt diagrams that are really focusing on management and implementation but we also have more impact and goal-oriented evaluations like the well-known environmental impact assessments but there are also participant stakeholder-oriented evaluation frameworks like in the stakeholder analysis and the needs assessment. So all these are of course evaluation frameworks and you could also argue that all decision support systems are somewhat evaluation frameworks. Overall we always say that we evaluate to learn and that's the main part of it but and we learn then to basically improve the plan and necessary from that evaluation that we compensate and mitigate maybe for undesired effects like in the environmental impact assessments or actually to steer our implementation towards our desired goals. But another important element evaluations is that they are also often part of an accountability process and that for example we see very much now in this parliamentary research and in this whole process already one minister had to step down so taking his democratic responsibility for this policy failure. Well to think then okay what is evaluating and why are we doing it and also we need to think okay who is evaluating. Well evaluation again because they come in many forms also many actors are actually evaluating. We see that research institutes are evaluating by research. We can present some of our more evaluative kind of research. I think Dr. Kuhn presented last time one of his MOTA research papers which he wrote together with Doreen Corbeil, Leonermans and Hollund V, which is I think a typical kind of evaluation form of research. But of course we also have the government assessment agencies for example for the environmental impact assessments but we also in the Netherlands have the Environmental Planning Bureau which also do these kind of other assessments of impacts. We can hire external bureaus from the private sector that are doing evaluations but also internally when we have project teams or implementation teams we do internal project evaluations. Overall we can say we are all evaluating continuously when we reflect on our day-to-day activity so there is also a combination of formal and informal evaluations. When we are looking at how evaluation and evaluation processes are depicted and I just have taken this one from the UNOCD. We always see that this is presented somewhat as a cycle and I do think that this relates to also how the planning process itself is or the policy process is presented as a cycle often. But this gives an interesting result on how evaluations idealized evaluate in the whole planning cycle and also in this picture I think it gives a nice overview. It represents what many people would agree upon that it is in the center it all turns around to develop a process of continuous learning and that in the planning cycle we have certain we use certain evaluation tools or decision support tools to develop the plan and to assess the impacts maybe so that in the implementation we have also evaluation tools that monitor the implementation process. Our resources allocate are we in progress are we managing to realize our goals and after a while there is a final evaluation or the evaluation of real impacts and that of course the lessons need to feedback in new planning cycles. Well very quickly when also when you would start reading about evaluations basically we make a distinction between two forms of evaluations and those are formative evaluations and summative evaluations. Formative evaluation have the goal of improving and strengthening a plan and rather than to to to judging it so if we look at the previous slide again the formative evaluations are really in this phase of the plan so the needs assessment stakeholder analysis those are all have the goal to improve a plan. The second are the summative evaluation and summative evaluation really focus on the outcomes and the effects so a good example and the most familiar example is again I think environmental impact assessments which really focus on the outcomes of and the impacts of a plan but also social impact assessments or strategic impact assessments but also in the end there might be also cost benefit analysis to judge the outcome of the plan. Other terms that we here often when we are looking at evaluation is ex ante and ex post evaluations and ex ante is basically well is latin for before the event and ex post is latin for after the event so ex ante evaluations are in the first phase of a whole planning process so in that phase of planning and decision making and exposed evaluations are when we are judging the outcome of the planning process so when we are in an operational phase. Well in the implementation phase of course we have also interim evaluations and there might be a final evaluation of when the project may be related to the plan is delivered but overall we also see a continuous monitoring and I think that's basically the main difference between when we think about evaluation and monitoring evaluations are often it's taking a snapshot of a moment or it might look back and monitoring is more this continuous process keeping an eye on the progress. Well I took this slide from from Koen in his webinar one and he presented this process as well as planning implementation and adoption and I think adoption can easily be replaced in this part as the operational phase so and he mentioned that well professionals tell us what we what should be done authorities decide what can be done and social communities need to adopt what they want and do and let's now look at how this relates to evaluation so when we look in the planning and in terms these are ex ante formative or summative evaluation but we see in the planning these are mostly based on science models and expert judgments when and they make assumptions about the impact of the plan on the future during implementation we see that traditionally most of the the evaluation frameworks are then management oriented and so they really focus on the allocation of resources the progress in time so and are we to on track to realizing our goals and in that adoption phase we see evaluations that are actually based on real experience and observations and we see a lot of case studies going on here they can be both qualitative and quantitative analysis but they need to be feeding back on how well the plan was implemented how well they achieved their goals and then give recommendations towards new planning processes just to get a little bit aware again on the the MOTA framework and already mentioned by triggers and perceived opportunities, threats, motivations, abilities and the actions so the interesting part of course of the MOTA thinking here is how to really think about what is the ability of implementing agencies but also other governmental actors but also the ability of society to adopt and implement a plan and I think these are valuable and how that influences action and I think these are valuable elements also to think about when we are thinking about evaluation so MOTA originally it is presented as a decision support tool but that also means that it's actually an ex-anteformative evaluation framework because it helps us in assessing the implementability and adoptability of different kinds of measures of different actions and in that sense we can put different actions next to each other and can assess what is our most desired kinds of action to move forward what's the most feasible to implement and again this is from a slide from Kwon so it helps us next to these other performance based to the more summative evaluation tool so next to assessing the performance that we are also assessing the adoptability and implementability. The tools that we then have for that in the MOTA framework is this MOTA mapping which has been presented to us and in the MOTA mapping we can mostly try to see where is actually and to develop then maybe strategies as well to get the actors in the high abilities and high supporters arrange and so for that sake it's also just like in environmental impact assessments a way to assess how can we not in this case mitigate and compensate but what kind of strategies can we develop to move towards this upper right quarter of the MOTA map so it is a way actually to for planners to to go back to the drawing board and to see if they can mitigate and compensate for the desired impact and in this case the undesired impact would be that you develop a great plan which is not implementable by agencies or not adoptable by local stakeholders and the fits the adoptability framework so the financial abilities, institutional abilities, technical abilities and social abilities are then a good structuring of how to move forward and during monitoring normally it's always about work breakdown, lock frames, it's really focusing on controlling the project but as Thie already mentioned in his previous talk especially when we talk about strategic plans also a lot remains open we know goals but we often do not really know how to do it so I think more recent kind of implementation management frameworks like Agile and Scrum but there are many others with all their own critique are interesting to think about and so what are the must have what are the nice haves and always be aware and again here MOTA can also come into play if we are needing to do less on time or if costs are reduced it will have an impact on the scope what we can do and the quality that we can do so finally so if we are assessing and learning so using MOTA as an exposed summative evaluation so as a final evaluation of the impact of our programs then the goal is of course to learn from the implementation and to improve future planning and the most important part of this then is that we need to stop thinking I think in the planning process as being separate boxes of planning implementation and adoption what we need to do is to integrate further and therefore if we really want to develop learning communities or evaluation communities we need to start to think about integrating these phases of planning implementation and the operational phase just like the one of the messages in the Dutch hearing that we in the planning we need to invite implementation agencies on the implementability we need to invite citizens representations representatives about the adoption and together create a learning community around the implementation the development implementation and operation of our plans so that makes our planning a little bit more complex because we are going to invite more people but it also means that we need to think more in interdisciplinary or maybe transdisciplinary terms so it requires multiple knowledges relating to the performance implementability and adoptability and so does evaluation and this is necessary because we know that complex problems are not solved by silver bullets and therefore it requires experimentation in implementation so evaluations become even more important if so to control and to monitor if you are actually realizing your goals and trying new things and steering by and for that sake a continuous monitoring via motor maps can help to steer towards the desired outcomes well some final things of course in evaluation it needs to create trust and therefore I think also truth seeking is important so coming at common grounds of what are the facts in these implementation processes but also to be pragmatic so really to learn from what works in practice and let those kinds of lessons influence not only the planning itself but also the implementation be reflexive and self-critical when in Netherlands we have the expression the butcher who evaluates his own meat well which is undesired so it's very important that there's always a critical self-awareness and of course evaluations need to be ethical and transparent so also about sharing again this relates to trust building and if we can build trust and learning communities I think the MOTA framework can support very much because it forces planners to think in terms of implementability and adoptability which actually supports participatory approaches to planning because it forces really to take into account the implementing agencies but also citizens and other stakeholders like the business communities and it helps us to integrate these people in the beginning from the planning process into monitoring and the operational phase to really learn around water and environmental issues thank you very much I think it took a bit long my apologies for that but I hope you find it interesting yeah thank you very much it for sure was very interesting has also a lot of activity in the chat on that I think that what I would like to suggest if you're okay with that is that you have to look at the questions and maybe pick one out of them and then we move to the next presentation and we can pick up with the other questions and debate after but I would like to assure that Nishal also still can have a small talk is that okay for you yeah yes that's fine so I think some of the questions they were arising as you were speaking you you kind of answered quite a bit of them yeah yeah I think the so the both Leon and the talk a bit about the adaptation pathways and I think indeed this is a very interesting part and also an interesting thinking and I think that idea of adaptation pathways where you said in adaptation pathways you said how the context of a around an issue is changing as your criteria to make decisions around what to do next and I think for adaptability in implementation processes this is also very important but in that sense you could also use goals as your criteria to change your behavior and goals are then of course not the implementation of an action itself but the goal is of course the desired effect of that of that action so the decision should always be about changing actions in order to come to the desired goals and I think that adaptive pathway thinking is very helpful also in using next to you and using a motin adaptive pathway for example in order to make decisions and changing