 Thank you all for joining us this morning or afternoon where they may find you and we're looking forward to hearing about a project and activity that's been ongoing for the last two or four to five years now to give a brief kind of context of where this project and activity exists within USAID. This project, which is a partnership between MIT and GW, is part of the Higher Education Solutions Network site project, which is now part of the DDI Bureau's Innovation Technology and Research Hub, and we're very happy to kind of bring this conversation together and hope you enjoy the larger discussion afterwards. With that, I'll turn it over to Courtney Blair, who is the Field Research Director and Technical Advisor for this USAID Uganda Feed the Future and Market System Monitoring activity. Over to you, Courtney. Great. Thank you very much, Maggie, and I want to thank everyone for being here today. It's really encouraging to see so much interest in systems approaches. Before we begin, I'm going to speak for 40 to 45 minutes and then we'll take questions, but if you have any pressing questions about a particular side, then you can feel free to interrupt. Some of my colleagues are also here and are going to be keeping an eye on the chat as well, so you can put questions in there. In terms of the agenda, I'm going to give a brief introduction to our activity, then a brief overview of our systems approach, and then go over some examples of CLA applications, some brief case studies of work that we've done. We'll be making the slides available. We'll send them out afterwards for reference, so if there's something that you missed, the slides will be available to you. First, just a brief introduction to our activity. As Maggie said, we're the Feed the Future Market System Monitoring activity. We have been based with the Uganda mission for the past five, four and a half, five years. It's implemented by the humanitarian supply chain lab at MIT along with the George Washington University, and our team is primarily composed of systems engineers and supply chain experts. I'm the token social scientist, and the team is bringing principles for systems engineering and supply chain management to development, and specifically that has meant developing methodologies and tools that apply systems approaches across the USAID program cycle. In addition to that, we've also conducted a number of research studies within the Uganda context in particular to help the mission strengthen their evidence base about the market system that they're working in. Why should you be thinking from a systems perspective? Well, if you're here, you probably are aware that the outcomes that you're looking for in development are the product of complex systems, and that you are trying to play in interventions in systems that are constantly changing and that are influenced by forces external to the system, such as a pandemic or even the weather. And as a result of that, it's really vital for you to understand from our perspective the system that you're working in and the dynamics of that system, particularly how that system is changing over time since you're trying to play an intervention that's hoping to see some kind of change over time. One of the benefits of systems tools is that they're agnostic, so system maps can be used to represent any system, and you can create indicators that can be used to measure and track change in the status of any system. So our sort of main takeaway point for today, if you take nothing away from the session, is that system maps can be used to understand how a system is changing and how or why that change is occurring. And so as a result, they can really be invaluable for collaboration for learning and adaptation. And that's sort of the main point that we're going to drive home today is how can you use system maps and system tools for CLA applications in particular. Okay, so now I'm going to give you a brief overview of our system mapping and measuring approach, which we call the system pathways framework. Before I begin, I just want to give the caveat that this is a very high level overview of our framework. It takes more than 15 minutes to explain. It's not incredibly complex, but it does take more than 15 minutes to come to explain. So I just want to say at the outset, that if it seems confusing to you, don't worry, I would usually need an hour to sort of get you up to speed, you know, on the primers for building a map yourself. But it's not essential that you understand the details to be able to get a grasp of the maps themselves and what they're telling you. So I'm just going to go over the basics so you can get a flavor of how the maps are built. But obviously, if you wanted to learn more, you'd have to look at the documentation that we put together. So our approach is called the system pathways framework. It's one that we developed for both mapping and then measuring systems, particularly in international development. It's adapted from causal loop diagrams and system dynamics, which are tools from systems engineering, particularly for contexts that have limited data. And I'm sure you know as a development practitioner that, you know, there's in some areas, there's a lot of data available in other areas, very limited data, not always in the areas you need or the time period you need. And so you're trying to make a decision with often very limited information. Now, the framework that we developed, we specifically link to results chains and development theories of change through the concept of pathways. And this was essential in our early iterations of the framework for working with USAID as a way for the practitioners like yourselves to be able to see your own work in the system. And by linking your work often in log frames and results chains to pathways in the map, we found it was a lot easier for practitioners who are maybe not as familiar with systems to see where their work lives within the system itself. So we specifically tailored it to make it easier for our USAID audience in particular. It's worth noting the framework can be used to make really broad maps of a large system, but it can also be used to make very focused maps. So maybe looking at one particular district or government office in a local area. As I said, early the tools are agnostic. It doesn't matter what you're using them for. And in a similar way, it can be used for a concrete system like a supply chain, but it can also be used for abstract concepts. And we'll go over an example of that later where we looked at household resilience, which in and of itself is a sort of abstract concept that's obviously built up of real-world outcomes, but representing it itself, it's kind of an abstract concept. So you shouldn't feel as though this is only a tool for market systems. It's only a tool for supply chains. We've spoken with practitioners in health. We've talked to the democracy and governance unit. So don't feel as though this doesn't apply to your work. It's an agnostic tool that can apply to anybody's work. That's a good segue. Who is this framework meant for? Hopefully anybody on this call who's listening, and it was designed, as I said, to be accessible to development practitioners. We linked it to your results change in your 30s of change to make it as easy as possible for you to see yourself and your work in the map. It's also a scalable approach. So we've used it with small groups like one or two activities. But you can also use these maps with a broad set of stakeholders. So maybe you have, you know, a whole consortium of actors that you need to bring together. A system app is a way of doing that. So the level of effort and the complexity of the map is going to depend on what you're using it for. It might only require an annual workshop, and that's what you want the map for. Maybe you're trying to update quarterly on your progress. Maybe you're looking at indicators and you're trying to change every month. It's really up to you and what you're using the map for. But again, we want to emphasize sometimes the reception that we get is that this looks so complex, and it seems like so much effort and so much time. And we really want to try to convince you that it really isn't. But it depends on, obviously, if you want to produce very complex analysis out of your map, then it will require more of an investment of time. But if you're using it, say as a communication tool or simply as a way to understand the system itself, it doesn't have to require that much time and effort. And also only requires really one or two people on your team who is familiar with the approach and then they can walk everyone else through it or we can help. So it should be accessible not only in the material, but we also think it's accessible in terms of how much effort is required and how difficult it is to implement. So the framework is made up of two pieces. We have two toolkits that we've built. The first one is about mapping the system. So this is, you've seen, I would imagine what a system map looks like before, a diagram with a bunch of shapes and arrows. This is what we're talking about here. You're mapping the behaviors, relationships, and conditions so they govern a system, finding pathways that lead to key outcomes, and then identifying leverage points that you can use to drive change. Once you've built that map, we have another toolkit for measuring. This helps you monitor indicators on these pathways and on particular outcomes. You could analyze and interpret your data to assess the health of the system. And then of course the measurement itself will inform adaptation of activities and design of new activities. I don't know if I'm going through that quickly, but it will become clear as I go through some of the examples later on. So there are three key building blocks to our system maps, behaviors, relationships, and conditions. In this context, behaviors are actions that are taken by actors in the system. It's a very broad definition. Relationships represent interactions between system actors, and then conditions are just fixed attributes of the system. And here's a little example of what one of our maps would look like. So you have shapes with different colors that signify different types of elements, and they're connected by arrows, and they come together to form what's here a little tiny system map. So the key, I think the key takeaway here is that the maps that we built are very action-focused. So as a development practitioner, you're usually looking for change, and change usually happens through a behavior change by some actor in the system. And so we find it's important for there to be an active orientation to the map, that it's looking at who is doing what, where, under what conditions, and in what relationships with other actors. That it's important to understand the system through that lens in order to influence it in the way that you're trying to. We just to clarify a little bit, when we put, you'll see, when you see the elements on the map, you'll see that we refer to system actors by some role. Sometimes it could be an individual, but usually it's a category or role within the system, like a collector or trader. And the connections between the elements strictly mean that one is enabling the next. So it usually is necessary, but it might not be sufficient. It's not necessarily causal. It's just saying that the presence of this element is enabling another element to take place. We also find that introduces a lot of flexibility, that you're more mapping all of the different influences that are creating a particular outcome. You're not necessarily worried about strict causality. That makes sense. Once you have all of your elements on the map, or as a process of building the map, we find it really useful to organize system map elements into what we call pathways. And a lot of times, these pathways can correspond directly to your results, Jane. So it's a branch of the map or section of the map that, as you can see, almost represents a chain of elements that are leading to that outcome there. Farmers take out loans to improve farming practices. And so this is an example of, when I said earlier, that we try to find ways to embed the existing USAID approaches in our system pathways approach. This is how, one of the ways in which that happens is that you can, literally, if you wanted to map out your results, Jane, using our elements and then kind of glue them all together, plus probably some other elements to build a map, a very simple map that way. And from the flip side, if you have an existing map, you can then go through it and trace. This is where my results chain lives in this system. And it helps to make it a bit more accessible, particularly people who aren't familiar with necessarily systems thinking ahead of time. Then finally, this is a bit more of an advanced topic. But once you have a map, you can use the map to find where there are reinforcing loops in the system. So loops that are either driving positive or negative change that are reinforcing. So, for example, here, this is from a map that we built looking at supply of quality beans that's being sold to traders. And the idea here is that the more supply there is, the more likely there are to be traders who will come to that district to purchase it, which means farmers are going to be more likely to produce beans because they know there's a buying opportunity, which means even more traders are going to come in and you have this sort of positive reinforcing loop in the system. And as I say, this is, this can be a little bit more advanced, a more complex map. But when you're trying to understand why a system is behaving the way it is, it's often really essential to understand the loops in a system. Then measurement. Now, this is a really important, this is the second toolkit that we built is about how do you measure change in a system. So it's so much more valuable when you can actually add data to a map and extract insights from measured elements. So we find you can learn something about a system with just a map, but if you only want to understand the status, you need to add data to it and create indicators. And also if you wanted to track over time, whether or not your intervention is producing the kind of change that you're interested in, then you need to have some kind of baseline and then, you know, have indicators that you're measuring. And also if you're trying to prioritize data collection, this is a way of doing so as well. Now, I'm not going to spend too much time on this because we're going to have an example that we're going to go over a little bit later. But you can just see here at right, this is the pathway I showed you earlier, and this is just an example of it measured. And so very simple stoplight rubric. The red means that the behavior has limited adoption or the condition is of a limited, you know, saturation, it depends on the element itself, how you word that. Yellow means moderate and that can mean, you know, say 33 to 66, depending on where you want to put the cutoff points. And then green means, okay, this is widespread at this level that, you know, we're aiming for. Obviously, you can come up with measurement schemes that are more complex, but we've found that even just this basic stoplight indicator system, really goes a long way towards helping with intuition and analysis about a system and about how a system is functioning. And then finally, there are a number of extensions of our methodology. And what I want to highlight here is that you can use it to assess the impact of shocks to a system. And this is another example that I'll go over a little bit later. So we developed a methodology for very quickly assessing the impact of a shock on an existing system map for, and we did this for COVID-19. There's a link there to the methodology written up that we developed. And then I'm going to go over this example a little while. Any questions before I move on? Okay, I'm going to move on then. So here's the list. Now I sent around a brief that we wrote that contains much of this information. And that's where I pulled this list from of the applications of our framework to CL&A. I'm going to go over an example of each, but some of them I'm just going to do very quickly in the interest of time. And then I'll spend more time on a few of them. But as I said, I'm going to send the slides right afterwards. And so if you wanted to read more closely, I'll learn more, you'll have an opportunity to do so. So first, let's look at collaboration. So how do you build a map collaboratively? Well, here's an example of a workshop we hosted in 2017. We had 168 people in a room. And what we did was took a map, we had built a draft map of the Ugandan agricultural market system, and then set up this workshop first to give an introduction to systems thinking and to understanding the market system in general. And then what we really wanted to do was use this opportunity to collect the knowledge of the stakeholders in the room to make sure that we were building the map, not just in a sort of silo of thinking, we wanted to make sure we were getting all the possible perspectives on the system that we could. So we had other donors, NGOs, the private sector of the government, and we set up the map, broke it into pieces and had little stations around the room. And then the participants could go from station to station, edit the map with the post-it notes that you see in the picture there, and we were able to collect an enormous amount of edits to then make to the map and produce another version that had far higher fidelity to the actual system as a result. So this is an example just to show it might seem so overwhelming to think, how could I possibly make a map with this many people? But it can be done. You have to have a draft to start with. But it can be done. And we've found it incredibly valuable to get as many perspectives as we could. And you'll see as I go through these examples, so there'll be a link to a report or a document of some sort and then also a link to the map itself. So that's what's available here, a workshop report and the map that we created. This is another example. So just briefly, you can use system maps as a way to generate or identify opportunities for collaboration or complementarity across the map. So in June of 2019, we had a workshop where we brought together the Feed the Future and Food for Peace activities in Uganda, who if you're familiar with this space, they don't always communicate about their work. And yet they're often working in the same or adjacent locations. And so it was really valuable for them to come together in the same room. And what we did was have them look at a system map, which you can see there in that image. We had everyone draw on the map for the marker, what they were working on at different stations. And then we brought them all together, layered all their sheets up top of each other. And it let them see, for example, in the lower right-hand corner, you can see there's a lot of work going on. There were a lot of marker lines in that lower right-hand corner. And this created a lot of conversations around, oh, I see once we had this finished product, they could go through and see, well, who had the blue line? Who else is working in this area? And then go and have conversations with other people. So the map itself can be used as a tool. Again, this was at a larger workshop to help different stakeholders identify opportunities to collaborate. And this was at a higher level of, oh, we're both working on extension services. We should have a conversation. This is a similar example. This was specifically focused on the Karamoja region of Uganda. So we had developed a map representing household resilience. This is what I was speaking about earlier, that it's possible to represent what might be considered a more abstract concept as a system map. Again, we hosted a workshop, and in this case, we brought together the activities that were all working in that region, some of whom were feed-the-features, some of whom are future peace. And again, did a similar exercise where if you see around the edge of the map, there's all these green boxes. I know it's very small, but if you click on the link, you can see them a little bit larger. We used, the green boxes represent the activities interventions in the system. And so we went to every activity and had them give us a list of what they're working on and put it on the map. So then again, when they sat down in the room in a more direct way at a sort of lower level of fidelity, they could look at who is doing what. And if I'm working on this element, who else is working on this particular element or on this pathway or in this little piece of the system? And so again, it gave them more opportunities to identify how they could work together within that, even within a particular region of Uganda. There was still room for collaboration. And then this is the most granular possible level. So because they found that exercise helpful, the activities working in the Karamoja region, the Karamoja cluster, had us create even more detailed maps. And so we created two maps for them last year, looking at the market system. And specifically, we made one that looked at the supply chain for iron-rich beans in two districts in particular. And so compared to the market system map of the whole Uganda market system I showed you earlier, this was a very specific, very focused map. But the benefit of doing so was that when we added the interventions to this map, you, we were really sort of at the, where the rubber meets the road in terms of very, very specific and granular opportunities for collaboration. And this diagram right here is showing two different concepts that we discussed with them. One was collaboration, which was okay, as I said on the last slide, who's working on the same element or an adjacent element, where there's a very clear opportunity for collaboration. But we also encouraged them to look at complementarity, which is what we call elements that are sort of down the chain, also influencing the same outcome that aren't necessarily in the same pathway. And so you might not think that they're relevant to you, but in fact, if you're both working towards the same outcome, you really need to at least understand what your colleagues are doing and working on that piece of the system to anticipate how that's going to impact the change that you're trying to make. So we have a report that's forthcoming that we could send around that explained how this could be done and talks about some of the other ways that the map could be used for the cluster. But we've had a lot of really positive feedback from them that this has been a useful tool. And in fact, we've had some interest from the broader sort of NGO forum for the Caramoja region to try to figure out ways that they can add not only USAID interventions, but also what are the other donors doing in this particular region, because there's a lot of donor activity at Caramoja. And can we use the map to map what everybody is doing so that we can make sure that we don't have say overlap or duplicates between donors, which they're pretty sure that there are. And so it would be an opportunity to identify those and maybe try to rectify it. Okay. Finally, for collaboration, and I'll just briefly mention this one, the maps themselves can be used as communication tools. Just very simply, it's a way of showing someone this is what I do. It's in this system. This was a map, as I said, we made two market maps for the Caramoja region. This was the other one that we made. And it was very focused looking at pastoralist livelihoods in the Caramoja region. A lot of the residents of that area are pastoralist or agri-pastoralist cattle herders. And so, but this is something that's not as well understood, particularly because it's a livelihood strategy that tends to be unique to this region. And so we worked with the Caramoja Resilient Support Unit, one of the activities there. And they said, look, we could really use some help communicating what pastoralism means. What is it? How does it work? We need to help better educate people about this. So we made a very, to us, what is a very simple map. It doesn't have a lot of elements. You can sit down in 10 or 15 minutes and get a good sense of, okay, these are the drivers of success in a pastoralist livelihood. And then another thing that we did is you can see on the bottom at right when you click on the elements, it shows you links to resources about that particular topic. And so it's almost a way of, it's like a library catalog. You can use it not only for communicating, but also the catalog and the information that you have. Okay, that was collaboration. So now let me just briefly talk about learning. And this is where we're going to go over the measurement toolkit that I spoke about earlier. So one of the most important applications of a system framework is learning how a system works, period. What is the system I meant and how does it work? And so this is an example of a deep dive study that we did 2019 and 2020 on agricultural finance specifically for smallholder farmers. So it was a topic that USCID wanted to better understand and they asked us, can you dig into this a little bit and try to explain it to us from a systems perspective? So we built a map that you see it right, which represents all of various drivers of access to finance for smallholder farmers. And you can see it's organized into pathways. So that's all of the colorful lines. Each color represents a different pathway. We then went through and canvas all the available data, publicly available data about access to finance or about whatever element it was that we needed to find. So some of it was land titles and things like that, but found all the available data, added it to the map, and then we color coded the element status as you can see there. It was an incredibly useful exercise for us and for USCID, first just representing the system, giving them a way of visualizing this is how agricultural finance works. It was also the first time all that data had been brought together in the same place. So just like you'd be cataloged resources, it's also a useful way to catalog data. And it really contributed to USCID's understanding of this particular piece of the system and the dynamics of drive change. So first you can just learn how a system works. You can also identify what you don't know, where there are gaps in either your understanding or gaps in data. So we were able to understand what's represented gray here. Gray was, we don't know, we're not sure we can't find it, we don't know. And so everything that was in gray on the map, we were able to go back to USCID and say, look, this is the gap in your and our understanding of the system. We also talked to stakeholders, you know, we tried to fill in the gaps as best as we could, but without a more extensive effort, level of effort into this research, it wasn't, it's going to be difficult to fill these gaps. So we were able to identify, okay, where are the gaps? Where are there gaps in the data that we can't find information? And then we were able to prioritize based on what's more important to us for understanding the dynamics of the system, which of these data gaps should we be identifying first? So for example, there wasn't much information from financial institutions, it was publicly available about the kind of outreach that they're conducting or their attitudes towards small holder farmers. And so, you know, the logical next step would be to go and do a sort of limited survey of the financial institutions themselves, something that was out of scope for us to get a little bit more information. You could also use the maps to develop a learning agenda. So beyond just where are the gaps, then question of, okay, well, what do we want to learn? What do we need to learn? And so I'm not going to go over this in too much detail, but we found that demand for loans in particular was an area that they need to learn more about. There isn't much uptake of agricultural loans. And part of the problem is that there is much demand from the farmers themselves. And this is linked, we believe, to access to inputs and other agricultural technologies. And so if promoting more loans is something that you're interested in, then you need to look at demand. Another way you can learn from a map is you can use it to monitor change. So given the data that was available, we were able to create two snapshots of the system, one for 2013-14, and then one for 2017-18. And what you see here is, this is color coded to represent the change. So blue meant positive, orange meant negative, the darker the blue, the more the change. So you can see here this pathway, formal access pathway that I've pulled out. There's lots of dark blue in there. That means there was a lot of change over that five-year period of time. So this helped us to show USAID specifically, not only what is the status of the system, but how has it changed over the past five to seven years. And this can be useful, I'm sure you can imagine. If you want to understand, say, have my investments or the investments of the donor community a large, produce the outcome that we expected, one prerequisite for that is to understand, well, how has this itself changed over that period of time? And so this is an important input to that exercise. Okay, then where am I on time? Okay, adaptation. So this is our last set of case studies. So first, so we've found, we've gotten the most traction. People are most interested in systems maps when something goes wrong and they want to understand why. And so we've been trying to encourage taking a more proactive approach and having a system map in place ahead of time and having an understanding of the system, you know, ahead of time and monitoring. So here's some examples of that. So as part of our work for the cluster in Karamoja, they had a set of high-level outcomes that they're aiming for as a cluster. And we showed them where those high-level outcomes exist on the household resilience map that we made for them. So you can see around the edge there, those four colored box sets of colored boxes represent their high-level theories of change. And then we showed them, this is where it's located in the system. These are the elements that you need to keep an eye on that are the most frequent links to your theory of change, the change you're expecting. And as a consequence, we were also able to recommend, okay, there are certain elements you should be measuring and put indicators on if you want to identify whether the system is changing as you expect. So this can help you then test whether your dynamic hypothesis was right, is the system changing the way expected it to, and hopefully identify earlier when the adjustments need to be made as opposed to after five years, you know, you do an inline or midline and you realize that it's not working the way it was expected. This is something that you could even track quarterly if you have that data monthly. It's going to depend on data availability in the end, but still allows for more frequent measuring of change in the system than is traditionally done. You can also, if you do find that the system hasn't been changing the way you expected, you could use the system map to figure out where or why. So this is an example going back to the agricultural finance study that we did. And one of the key outputs was identifying barriers to change in the system. So where do we see that change has stalled? We could see how it had changed over time over that five to seven year period. So where do we see that that change is not occurring that the change has stalled? And for example, here's two elements in particular that we found were a barrier. This is demand from farmers, right? Trust of formal financial institutions was very low and willingness to take on risk also very low. And so as part of a broader the broader demand loop and the broader dynamics in the system, you're not going to see much change in your outcome, which is more uptake of loans by smallholder farmers if obviously there is no trust in formal financial institutions and the farmers aren't willing to take on risk. And so finding barriers such as these where you have say green elements pointing to them, you're seeing change incoming and then it stalls is a way of identifying what we potentially need to address in order to see the change for you down the line that we're looking for. And then finally, I mentioned this briefly earlier, you can use a system map to rapidly value the impact of a shock. So because we already had a map with the agricultural market system, we were able to really quickly spool up and start collecting information about specifically the impacts of the COVID-19 restrictions in Uganda on the market system. So it was as much the impact of the pandemic, the health impact of the disease as it was the impact of the restrictions that the government imposed such as limitations on movement and closure of businesses and things such as things like that. So we were able to visualize for USAID how the shock would be propagating through the system and then incorporate all this information very quickly and assign statuses to the elements. And this gave them a way of framing the situation. They were able to act quickly, anticipate how the system was going to change over time. We wrote a series of four reports that are available at the link. And then also, there was a version of our map that had all of the shocks added to it. It's also available at that link. Okay, so I know that was sort of a whirlwind tour of systems approaches to CLA, but I'm hoping that this, the slides plus the write up that I sent around, you can sort of explore at your leisure. And if you have any questions, please feel free to contact me. There's that email address. I think my email address is on the invitation for the event. So really, if you have any questions or you'd like more information, we were group academics, we're always happy to engage with people and sort of spread the word about systems thinking. So please feel free to contact us. And now I'm happy to take questions. Of course, we've got a few questions in the Q&A already. I'm going to invite JP Petraud to, you have a series of questions related to the melp and the maps, how they relate and how data relates to things. So maybe JP, you want to jump in and ask your question out loud? Yeah, can you hear me? Sorry. Yes. Oh, great. Thanks. Sorry. I'm not entirely sure about the setup. So many, you have them in writing, but my question is, I guess there's two sides of it. There's, on one hand, how are the maps based on data? And so that's sort of at the end of the production chain. And then at the, or the mail system, and then at the sort of onset of setting things up for around an activity, how does system thinking affect or, you know, how does it affect the map plans around an activity, right? So that the proper data are collected so that you can then do the map. So it's kind of like this interaction between the outputs, which is the map and the input. That's a great question. Thank you. I would say on the one hand, how do we build the maps? We start out with as extensive as possible consultations with the activity stakeholders, beneficiaries, published literature, whatever's available to build a skeleton version of the map that we then can iterate on with stakeholders and try to perfect as much as we can. But then when we do add data, I will tell you if we find a number that seems unusual. So, you know, for example, if we were to add data to an element and it's red and everything around it is green, then obviously we've missed something. There's something influencing that element, causing it to be read that we need to go back and dig in a little bit more on. So there is an interaction between the data and the maps themselves. I would say on the other end, we're actually about certain engagement for the Gates Foundation, where we have an opportunity finally to insert ourselves at that very beginning space where the map is being developed. Because we do recommend often the map is at a very granular level. It's on the ground. We understand that you have indicators that you need to roll up and, you know, that makes sense. But sometimes in a very limited targeted way, there are other diagnostic indicators you can collect for particular elements in the system that will be incredibly valuable to you down the line, particularly if it's something that we know ahead of time doesn't already exist. And so where possible, we really recommend doing this kind of scoping exercise at the beginning of the activity to see what system indicators should we be tracking that aren't already in our map and that the data business already exists for. And that doesn't have to be an enormous lift. But it does make it easier down the line to then evaluate how the system has changed. All right. So we've got a few more questions in the Q&A and one in the chat. And again, people can continue adding questions in either the Q&A or the chat. We'll look at both. So Janie, you've added several questions. The one that's voted at the top is the collaborative workshops. Have this been tried and tested virtually? The short answer is no. The short answer is we've had map workshops, but they've been on maps we've already built or that the participants were already somewhat familiar with. So no, but I have a feeling we're about to test that. So ask me in six weeks and I'll have another answer for you. Got a couple other questions from Janie. I'll move to the chat. We have one from Wadad about measurement. Wadad, would you like to ask the question out loud or should I? Yeah, thanks. Well, my question was at some point in time, you were discussing, you know, that you would collect data to verify the validity of the relationships that were identified on the map. So if you find that, for example, for a given relationship, you found that the data was not really indicating a strong relationship between the cause and effect under consideration, then what would you do next to really understand that phenomenon? Would you, I mean, I mean, there could be different reasons for that to happen, and it could be either related to the design of the map and the understanding of that particular relationship, or it could be linked to the intervention and its design or its implementation, or maybe other things as well. It could be a change in the context that was not, you know, and that was not in mind when the map was designed. I mean, there could be different things. So how would you go with that? I want to make it clear. Yeah, no, thank you for your question. Just to be clear, so the maps are not, we're not assigning, we're not estimating parameters, we're not putting causal estimates on the map. This is not replacing regression analysis or a system dynamics model, right? So if you wanted to try to really pinpoint either the magnitude, you know, or some level of precision about the estimates, you'd have to go for a more complicated tool. What this is allowing you to do is see almost a sort of covariance. What's tracking together? Are we seeing things moving in the same direction? At least what's the magnitude of change on adjacent elements? Are we seeing, you know, if everything up the pathway is changing, are we seeing that change cascading down to the elements below? So there are limitations to the insights that you can get. We think that there's value in at least reaching that level of insight, but if you wanted to go further, then there are more complex tools you can use. And I'd be happy to talk to you more about that. Thank you. Another question from Janie. Janie, do you want to jump in and ask out loud, think more of a conversation? Yeah, sure. And my question is especially, I mean, listening to you and all this thing and how long I'm imagining it takes, is this like an impact evaluation? And then I want to imagine you guys did this for USA Uganda, right? So how long did you work with the Uganda mission on this particular one? Or what's the ideal in trying to do this? And then is this something, is this a process and something that you guys come in and facilitate? Or are there tools and something that you share out with folks and we take them and do them for ourselves? I mean, how does all this really come together? Thank you. That's an excellent question. Let me, I'm going to start with your last question first, which is we have developed documentation. We have two toolkits that we've written. One is for the mapping approach, creating system map. And then one is for measurement, putting the indicators to these elements. The whole purpose of this is to put ourselves out of a job, is to hand this over to you and then we can provide guidance and support and some expertise. But the idea is that you with limited facilitation are able to do it yourselves. How does that work? Sometimes it means we find one person on your team who seems particularly interested and we train them, teach them how to do it, and then they can facilitate the workshop. We've also come in and facilitated workshops and then explained how you can go on and use it further. But then there were cases where USAID said, can you just do this for us? So we just did it for them. So it really depends on what the sort of appetite is for taking ownership of it as opposed to having us do it. But I will say also, it can function as an impact evaluation. That's a great way to think about it. But not every map that you make or measure has to have that level little rigor or effort involved. So you can sit down and make a map in an afternoon, a simple one if you really wanted to. In a week, you could take the data that you know exists and add it maybe two weeks. Or if you really wanted to get into the details and collect data and to have a cross section across time, okay, maybe that takes a little bit longer, particularly if it takes you longer to put together the system map in the first place. But it really depends on the level of fidelity you want and what you're trying to do, but it doesn't have to be a six month long project. Absolutely not. I'll just note too, some of the questions have been answered live in the Q&A. But I'll highlight that Carly asked about the tool we used. It's called Kumu. And I put a link into the answer to that question. So that's a good one. And just in terms of point of contact and so forth. So there's some answered questions there in the Q&A as well. So take a look there. Another question from Chris. Chris, do you want to jump in and ask that directly? About data? Yes, thank you. Am I, can you hear me? Yes. Super. So I'm looking, first I apologize. I learned a little bit differently than other people. So when I see something like this, I kind of want to get into how the practical, how practically played out, how the process played out in the mission. So the first one would be about where did you gather your data sources? Was it something, were they from sources other than you say, were they generated by the project that was associated with those maps? The second question is, several times during the presentation, you noted points where they were having, they were encountering obstacles, such as I think one you frequently mentioned was the desire of farmers to take on loans, which having been associated with farmers for a long time, that does not surprise me. So were there points that the team began to question its theory of change? And if so, how did this map actually affect the implementation of the project? And then, then following it through to its logical, then, okay, now they've done the project, how are they using this mapping exercise to develop follow on programs? And how does that, how are they using the maps in this, the collaborative exercise to build your six through 10 of this effort ever? Thank you. Goodness, okay. So first I would, so you asked about data. For this one in particular, agricultural finance, we did not collect any data. We went out, as you may know, because if you're working in space, there's a lot of data available about finance, digital finance, access to credit, lots and lots of data sets. So I think we pulled maybe 50 sources total or 60, but it was all either data sets or journal articles or white papers that were available on the internet. So we had a, you know, an undergraduate student who helped us pull this all together. So it was effort, but it was not, you know, beyond what would be within an activity's resources, I should think. It's not as though we had to go out and collect all of this. Obviously, if we'd had the opportunity to do so, we could have identified for you a couple of data points that we really wanted that weren't available, but, you know, we were still able to extract the insights from this without having to go out and collect that data. For this study in particular, it wasn't quite, as you probably are familiar, you know, there's a cycle, there's the timing of when the activities are being designed. This was a little bit too late. But I can tell you that it created, it did start to ask those questions, you know, of okay, if we've decided that promoting access to or uptake of finance or small-holder farmers is a goal that we have in mind and it's not happening the way we want, this is giving us some insight into why not. And that maybe we need to be focusing on, you know, linking this to the interventions that we already have in place to do with access to agricultural technologies. And maybe we need to go out and actually ask some farmers, you know, why is it that you feel the way you do about the financial system? So that hasn't led to any concrete, you could say changes yet, but it's led to a mindset shift. There are other pieces of work that we've done that have. So the COVID work, for example, you know, they were given the opportunity to have some of the activities pivot or apply for additional funding or, you know, do some sort of emergency, you know, piece of work. And so this was used to help inform the decisions on whether or not, you know, some of those were appropriate. And then I can actually say, or, Jerry, do you want to speak about the new market system projects that they developed and how our work influenced sort of their thinking about the current project in Uganda, the current five-year market system project? Yeah, well, one of the things that happened in the 2017 workshop was we got, that was our biggest workshop where we gathered participants from various activities, different donors from the private sector itself, lots of different stakeholders to inform what's been happening in the system and had some specific tools to gather information about where interventions might be needed and so forth. And that was a foundation for some of the design work of the new portfolio of activities within Feed The Future. So that broader stakeholder engagement, getting perspectives from outside USAID was really important in identifying, you know, how to move forward in the next four or five years with design of new activities that could address some of the gaps and the opportunities that were revealed by a broad set of stakeholders. So I think that's one of the key things that is, you know, getting that broader engagement can help you see more than you might see from your internal discussions and with a few set of activities. They said that the work that we had done was, what was the word Lari used, foundational to their thinking for the next market, the next project design. And we saw a lot of the concepts and language we've been talking about coming out and said the RFPs for that. So that felt like a victory for us as well in terms of hearts and minds. Including some thoughts on designing new melps and so forth. Yeah, exactly. There's another question here from Kristen about COVID. Kristen, do you want to jump in and ask out loud? Because we're on the COVID shock slide here. Sure. I'm just, I'm really interested in how you did the shock overlay. I mean, obviously data at that point would have been very emergent, let's say. So if you could just speak a little bit more to how you, you did that. This is my dream for activities to be able to do this for all of the probable shocks in a kind, you know, like pick your top three and do this in advance to help figure out what are like the resilience pain points. So I love that you guys actually did this in light of COVID, but can you just speak a little bit more to the how? Sure. And it's, it's good to talk to you again, Kristen. So how do we do this? Well, our philosophy was information that, how do I put this? We appropriately copy out of the, the information that we produced with the feeling that more information for decision making is better. Right. So we said, look, this is based on the information that we can find. It's based on what's available. We can't guarantee that it's correct because this is an evolving situation, but we were clear about the inputs that we use. And therefore, you know, the sort of level of confidence that they should have in, in what that produced. In terms of the shocks themselves, we sat down with the folks at USAID, some stakeholders that we knew, some key informants. Um, so the shocks were easy to identify, right? I mean, it was, I was living here, I could have told you what the shocks to the market were, right? The Borgster flows that, you know, there was no private motor cars on the road. But then in terms of how that's going to move through the system, that's where we relied at first on expert intuition of how do you, where do you think this is going to go? And then as the data started coming in, we could test, okay, was that right? Is it happening the way we expect? Do we need to shift, you know, the status or is it coming through a different pathway? So is it as rigorous as in terms of like, quote, strict accuracy as some of the other studies we've done? No, but in terms of improving on the level of available information at the time, we think it really to do, we think that we are able to really contribute. And if I can jump in here, because I do crisis response work is one of the key things our lab does. And to me, this is one of the interesting applications we had a map already that one things was useful, we already had a map to base this on. And then we gathered news articles, historical reports, any new information coming out, it was all the different sources. And the Kumutu allows us to put the sources in there to give some, you know, documentation. And in fact, it became an online repository of organizing information systemically so that anyone could come in and look at it and try to make sense of what's happening, which is commonly what is a situation in a rapid evolving crisis, you want to pull information together, triangulate and make sense of the best you can get information that can be perfect, but by triangulating, you build more confidence. So I think the way of it's became an online repository and data catalog for for broader understanding of the implications beyond the obvious and how it trickles down into affecting other parts of the system. I think that's all we're going to have time for. But I wanted to thank everyone for coming today for your really interesting questions. I love getting the opportunity to talk to people outside of the sort of group that we normally interact with. And as I said before, please feel free to contact me or I can put you in touch with someone else on my team who might know better, but you know, we're really eager to talk to you about these topics and to help you use these tools in your own work. Thanks so much, MIT and GW team for joining us today and we will be sending out a recording of this session to all the people on the calendar invite. So please share it with your colleagues and anyone else you think might be interested. All right. Well, thank you all so much for attending.