 And this one looks at tools for reflexivity. So reflexive monitoring and evaluation, or M&E for short, is called that because it has both reflection and reflexive practices. So to reflect is obviously to look back and thinking about what happened, what does it mean, and how to proceed. So this evening when you'll be licking your wounds from this day, you'll think about, oh, what happened today? And how do we go forward from here? Reflexive is looking at how do we change, change the rules, change the practices, change the assumptions, both of others and of ourselves. So it goes beyond just reflecting. And it usually goes through the cycle of we observe, we analyze, we reflect, we adapt, and on it goes. So it's a double loop, triple loop learning process. Some of the differences between traditional and reflexive M&E. Traditional M&E tends to use static indicators as part of a log frame, say. How many here have worked with a log frame? Yes, how many times did it perfectly reflect your reality? Never. But that's how the donor wanted it, right? So, needed to be done. It also looks at output and outcome indicators, predominantly. We did this, that happened, and it led to this. That's a traditional classic form of indicators in a log frame. It also focuses on achievement of predefined goals. So we have a five-year project, or we're asking for money for a five-year project, and we're supposed to look into our crystal ball. We don't know what we'll be wearing next week. We don't know what we'll have for dinner today, but we know what will happen over the next five years in a very complex agricultural system, and we can even tell you which indicators we will use to count those. And those will not change over the next five years, because we got it all right. The night before when we were hastily typing the proposal because it was the deadline, yeah? You can see that I have a bit of skepticism about this. The structure is set, and once it's set in motion, it can be modified, but it can't be radically changed. So we can say the number of people we reach will not be x, it will be y. But we can't say we're going to reach different types of people now, because that's not what was said in the project. So it's very fixed. In reflexive M&E, indicators are not static. Progress indicators dominate. So not output and outcome, but progress indicators. The focus tends to be on the change. What kind of change are we observing? And if we see really good reason for it, we can also change the design mid-course. In both cases, impact is important. So it's not that one focuses on having impact and the other doesn't. But it's a very different philosophy in terms of how we actually measure the process that leads us to the impact. Reflexive monitoring in action, because everything sounds more serious if you put in action after its name, is an integral part of the planning for platform activities. So you can look at what tools to use, what moment to use, which tool works best for what of them. But the most important thing is that for it to encourage reflection and learning. That's the constant theme that reflexive M&E will promote. So not have you achieved widget 1, 2, 3, but what have you learned from this process? Well, how can you learn more from this process? The role of the monitor is very different from the role of the facilitator. And now somebody will say, but you said earlier, I remember, distinctly remember a slide that said that part of the facilitator, the innovation broker's role is also to document and monitor. I will say yes, but these presentations were prepared by different people. The focus though is not on facilitating the process, but really seeing on what is happening. So that is the primary role of the monitor. So it's often best to separate those two roles so that the facilitator can really facilitate and the monitor can really focus on what is happening, what are the changes that are being observed. It's usually very difficult to do both well at the same time. A big part of it is to maintain the learning history. So when we go through the process, there are many things happening and we'll remember them. I mean, hopefully if this evening you went to the room and you found a paper-based version of the clicker questions, you can all relax, there will be no paper-based version of the clicker questions. But if you did, I'd like to think that you'd remember 80% of them correctly. If one month from now you come back to your home and I somehow sneak into your home that same paper-based test, I can assure you it will be far lower than that. Because we forget. We forget a lot of the richness. We might remember the big picture. We decided to clear the forest to build houses. We'll remember that. We'll remember what was it that the housing developer said that was so compelling at the time that got us to agree to that. We forget the details. That's the role of the monitor to make sure that collective learning is not lost because that's going to be very useful for us and others going forward. So in order to do that, there's a range of tools that are available and I'll run through them relatively quickly so that we just get a taste of it. One is called a fishbone diagram. And why is it called a fishbone diagram? Because the diagram looks like a fish. And it's basically trying to find out causes rather than symptoms. So it's basically asking why five times? Why did you come here? Because my boss told me to. Why did your boss tell you to? Because part of my work is working with communities. Why is it working with communities? Because we really want to get to impact and we think communities have the key. Why do you think communities have the key? And so on. So if you ask anyone anything five times why? One, you might find the cause. Two, that person might not sit next to you for a while. So you have to use that sensibly. Two, is a timeline or learning history. So you've all no doubt seen these. Somebody draws a timeline and it can be on the stand, on the floor. It can be a chart on the wall. And we put a data instead of saying, tell me what happened? Let's think about the project in the last three years. What happened? What were the key moments? And different people draw their own key moments or they stick stickers. And then they can share a few comments on each one. So say this was a big event for me and these things happen. One, two, three. I was hired. That was a big moment for me in the project. This is the moment that the community built the latrines after we've been trying to convince them. That was a huge moment for me. And so on and so forth. Then you start getting that. When you start getting that from the entire group, a picture begins to emerge of what happened, how they perceive what happened. Another is to use things like event logs or meeting minutes or audio visual records of the major events. So these are all things that you're all exposed to in some ways. They systematically can help. And some of these tools, you see, these are sort of simple questionnaires from Google that can be done very cost effectively these days and can help you keep a wealth of information. They can then generate results for you. Google Forms does that automatically these days. So a lot of these tools can be used. And whether it's a Google Drive or Dropbox or Wiki or all of these things, very good for collecting information, sharing information, but then it's really useful when you want to then reflect on what happened, you have all of that trace. What file was uploaded when and so on and so forth. You can also go into more sophisticated ways of data analysis. So you can use descriptive statistics. You can use statistical analysis. You can use text analysis. You can use social network maps or social network analysis softwares. You can do mind mapping. You can do econometric or spatial modeling. You can do anything. You should always just ask yourself, what's the added value? Sometimes there is one. And sometimes there might not be a strong enough one. So before you get involved in more costly, time-consuming forms of data analysis, ask yourself, what is the question that you're trying to answer? And isn't there a simple way to answer that? Don't be the person in my joke earlier about counting the sheep. Don't be that kind of a researcher. And finally, there were a few people nodding about more significant change and a few people that weren't so nodding. So more significant change, you typically identify no more than three types of change that you want to document. Then you collect stories from platform members as well as end users and other stakeholders and you shortlist the best stories that you find. And in order to do that, the story reviewers, you take them amongst the people that you're trying to reach or influence. So it can be your donors, it can be your target audience. And those stories are then the ones taken forward to help both document the change and influence future change. Now that sounds pretty fuzzy in abstract. So in the very short time that we have available, I propose that we take ten minutes, split into groups of two. You should all have this piece of paper on your table. So in groups of two, one person will be the interviewer and the other person will be interviewed. And you have a template there that gives you four simple questions in terms of a most significant change. Now don't worry about not being able to complete this fully in ten minutes, I know it's not a problem. But it'll be good to engage in the process. So take 30 seconds to decide who's going to interview who and then the next nine minutes to complete as much as you can from this process. Actually take eight minutes and then one minute to discuss between you how useful or what was useful about the process. Okay? Having some discussion regarding number four. So how has the work of blank stuff contributed to this? Let's say for example she is a research staff. Is that the question? I'll answer that in a minute in the plenary. Now I want you to stop. I want us to come back to the plenary. It's not important that we finish the exercise. I just wanted you to get a taste for it. But I'll answer that as soon as we get everyone back. Okay, let's come back to the plenary. I see this is a group that really enjoys group work and so tomorrow we'll be very much geared towards that. You'll get a chance to sink your teeth beyond five, ten minutes' exercises. We'll talk about that more in a minute. How useful would you find this process in terms of documenting change that took place reporting on that back to your target audience? Is that something that you would find useful? Okay. What did you enjoy most about this process? And I want to hear from one person who interviewed and from another person who was interviewed. So someone who interviewed, what did you enjoy about this process? Please. Being interviewed I am tracking back what happened to the previous activities. So with that I was trying to think of what happened and what did not happen. So it's looking at experiences. Absolutely. And for most people it's a lot more fun getting human stories than it is to track indicators. So it turns something that can be very dull and meaningless, alive and personal. And you feel often the person doing it would be part of the organization that implemented the work. So it would be, oh wow, my work is really making a difference. Which you don't really get when you're sort of ticking boxes or sending out surveys. How about someone who was being interviewed? Someone who was being interviewed and would like to comment on how it was for them. I was the one interviewed. Yeah. I enjoyed the process where I was reminiscing what happened in the past. It somehow gave me a sense of fulfillment that I was able to do it. And the recall of the story actually made me feel good. Absolutely. And many people also in addition to what you've just said this person came and really asked me what I felt, what I think. That's great. People like being engaged. It's also a very empowering way of collecting the information. Did you find it invasive? I did not. Well, that depends on your interviewing techniques. I can ask you what you had for breakfast in a way that would sound very intrusive. But there was also a question about question four. What does it mean? So question four was overall, if at all, how has the work of staff, staff would be the name of the person from the organization that facilitated the work, contributed to what you achieved? So this basically helps us identify not only the change that took place, but also how individuals within the organization contributed to it. So then we can attribute some of that change to their actions. So it's a very nice qualitative way of getting this information out and packaged in a way that can resonate with a lot of people. Okay. With that, I'd like to thank you very much for your attention throughout this half of the day or a bit more of half of the day. And I'll hand this back to Sel or Nova for assignments and so on. And actually let me be the first one to say something there and I'll be the bad cop. I actually do have homework for you. There's a small, relatively short, very engaging case study about an innovation platform in the foothills in the Himalayas where India borders Nepal. It's called the Milkit Innovation Platform. And tomorrow morning, we will spend a good chunk of the morning discussing a lot of the principles that we covered today in the context of the case study. Yeah? So between now and what time do we start tomorrow? Between now and 8.29 in the morning tomorrow, at your leisure, at any point, please take half an hour to read the case so that we can have a very engaging group discussion. Yeah? Tomorrow morning. So thanks in advance for your cooperation with that. We will hand out the cases now and with this, I hand over to Nova. Venko will be fetching you at around 8. What was it? It's at around 8 o'clock tomorrow. So your assignments.