 Hello everyone, good afternoon, good evening or good morning depending on where you're joining us from today. Welcome to Engineering for Change or E4C for short. Today we're pleased to bring you this month's installment of the E4C seminar series. This series was spearheaded by ASME's Engineering for Global Development Research Committee and its purpose is to intellectually develop the field of engineering for global development. We host a new research institution monthly to learn about their work advancing the United Nations Sustainable Development goals and more. Today's seminar is presented by Dr. Jeffrey Walters from the Civil Engineering Department at George Fox University. My name is Yanaranda and I'm the President of Engineering for Change and be one of the moderators of today's seminar. The seminar you're participating in will be archived on E4C site and on our YouTube channel. Both of those URLs are listed on the slide. Information in upcoming seminars is available on our site and E4C members will receive invitations to those seminars directly. If you have any questions, comments or recommendations for future topics and speakers please contact our team at research at engineeringforchange.org and if you're following us on Twitter today please join the conversation with our dedicated hashtag, hashtag E4C seminar series. Before we move on to our presenter I'd like to tell you a bit about E4C. E4C is a knowledge organization, digital platform and global community of more than 1 million engineers, designers, development practitioners and social scientists who are leveraging technology to solve quality of life challenges based by underserved communities. Some of these may include access to clean water and sanitation, sustainable energy, improved agriculture and more. We invite you to become a member. E4C membership is free and provides access to news and thought leaders, insights on hundreds of essential technologies in our solutions library, professional development resources and current opportunities such as jobs, funding falls, fellowships and more. Members also receive exclusive invitations to online and regional events and access to resources aligned to their interests. Please visit our website to learn more and sign up. Our research work cuts across geographies and sectors to deliver an ecosystem view of technology for good. Original research is conducted by engineering for change research fellows annually on behalf of our partners and sponsors and delivered as digestible reports with implementable insights. You are invited to check out our research page. You see the URL listed on the side to explore our field insights, research collaborations and review the state of engineering for global development, a compilation of academic programs and institutions offering training in this sector. If you have a research question or want to work with us on a research project as a fellow, please contact us at researchandengineeringforchange.org. And on that note, I'd like to share an overview of our current cohort who is actively doing ecosystem research in 2020 from around the globe demonstrated here through this map. Nearly 60% of them are women. They represent a variety of skills, including interdisciplinary contributions from aerospace, architecture, civil and environmental engineering, nutritional food science and software engineering from all around the globe. If you're interested in becoming a fellow, please explore the URL on this slide and please do feel free to apply. Now, we want to move on to a few critical housekeeping items. I know most of you around the globe are now familiar with how to use Zoom given current circumstances, but we are really aligned to hear from you and make sure that you are putting your questions and your comments into the right place. So please kindly click on the chat window and share with us where in the world you are joining us from today. We want to see where you're from. So I'm here today in Brooklyn, New York. I see folks from Oregon, from London, from Arizona and California, from Sweden, from Calgary in Canada, and more Brooklynites. Welcome, Texas and Switzerland, DC and Kenya, Nairobi, Denver, Karachi. Welcome. So fantastic. Welcome. We are so thrilled to see you here from Nigeria and Pennsylvania across the globe. So, so awesome. Just as a reminder, you can use the chat window to share remarks during the seminar. And if you have any technical questions, you are very welcome to send a private chat to the Engineering for Change admin. Just search in the participants. If you're listening and you have any troubles, try hitting stop and then start. And then please do use the Q&A functionality for any questions that you have for our presenter today. This will allow us to keep track of those questions. All right. Oh, and so thrilled to see one of our past contributing editors has also joined us as a listener. We're so thrilled to have you. Hey, Sean. All right. So with that, I am so honored to introduce our presenter and also our fearless lead in this as next. So Dr. Jeff Walters is an assistant professor of civil engineering at George Fox University. His research seeks to develop and refine decision support techniques written in complexity science and systems thinking to improve engineering practice and policy for sustainable rule and urban infrastructure system design and management in developing work contexts. His research has been applied within the sectors of WASH. For those of you who are not familiar with that, that's water and sanitation and hygiene, energy, food and engineering education. And the man who needs no introduction, but we'll get a brief one is our co-moderator, the co-chair of our Engineering Research Development Committee, Dr. Jesse Austin Breneman, who's an assistant professor of mechanical engineering at the University of Michigan with his beautiful bio, which you wish you can read at your leisure. So with that, I'm going to go ahead and stop sharing my screen and turn it over to you, Jeff, to share your presentation. Thank you for that wonderful introduction. By the way, this is this is this is really an honor for me. Thank you. Let's not start the problem. We have enough problems in our lives right now. Right. So again, it's it really is an honor to be presenting this research that and to say that it's my research is seems a little bit disingenuous because of how many folks that I've worked with on this work. And so the main goal of this presentation really is to engage in this discourse with you all. Hopefully get some more collaborators on more of this work moving forward. So this research really focuses on applying systems systems tools to improve upon the sustainability of service delivery, not just of water, but it could be it's issue agnostic. It's energy, it's food, it's any sort of structure. So let's get into it. So here's a quick overview, right? And really, I think the case is pretty easy to make for a need for systems thinking and rural water supply could be sanitation as well or energy food. There's a need to understand systems. And I'm going to highlight two approaches that I've been working on with collaborators that hopefully this is this stimulates an interesting conversation. Well, a stakeholder driven approach that is sort of getting stakeholders local stakeholders thinking and systems and extracting systems insights. And then an evaluation of stakeholder understanding, which is a bit of an abstract idea. And I look forward to diving into that a little bit. Do these approaches actually improve stakeholder understanding? And then briefly highlight where the blind spots are and areas for future research and collaboration with Paul that I hope gets us going on a good discussion. So the problem, okay, so you guys have probably seen this many times, but there's a lot of people that don't have access to clean water, right? Despite many efforts, there's still about 800 million people that don't have access to water. That's unfortunate, right? But I think the more unfortunate aspect really, and in my sharing my screen, are you guys all seeing this as well? So I need to get this out of here. No, no, we just see the problem. And I thought you were seeing my chat window. We are not seeing your chat window. Thanks. So yeah, the big issue here is that the interventions in the past have not been super successful. A large failure rate, especially in Sub-Saharan Africa, water points failing, gravity systems failing, etc. And so what comes as no surprise is that these failures are a result of a complex interaction of factors and how actors interact within that system. There's interlinking interactions of factors. There's dynamics that exist between those factors. And ultimately what this is is sort of complexity. There's a complexity that plays a complex adaptive system is what it's been called in the past. That makes it hard to plan for sustainable interventions, both in water as well as other types of development pursuits. So regarding water specifically, how do we plan and manage sustainable water services in light of these complexities, kind of learning to dance, so to speak, with the system? And so that's what my research thrust has been over the past eight years. Is it possible to model the system? And again, these systems are social systems. They're political systems. They're economic systems, environmental systems, technical systems, supply chain systems, all sorts of different systems coming together to affect sustainability. Is it possible to do that with local stakeholders in an accessible way, engaging the intimate understanding of local stakeholders in that process? And then is it possible to evaluate whether the approaches did what we hopefully did in getting stakeholders thinking in systems or at least changing their understanding for how different factors interact to impact systems outcomes? And again, so what I mean by system isn't necessarily just the water system itself, but instead sort of all of the sort of soft systems that surround the maintenance of that system. And then lastly, how does this impact? This is what we want to get at, right, service sustainability and high service performance. How does that change in understanding impact actions that impact service delivery? And really, the whole idea of this research is to automate this process to make it so that you can really go through this process of mapping, evaluating understanding, taking action, evaluating action, mapping, and so on and so forth. It's a cyclical process of adaptive learning and action. So that's kind of the high level view of my research. And I want to highlight, this is research that I'm standing on the shoulders of giants, both my colleagues as well as years and years and years of systems research in this area of complexity science. So I just wanted to highlight that. This is a really, if you guys have a chance to look at this at some point, just look at this up, the map of complexity science by Frank Kostlany. This is an interactive map. You can click on all these different tools that have been used over the past, geez, 80 years, 70 years. The tools that I am exploiting, so to speak, are those in system dynamics and in soft systems analysis. So soft systems analysis is really just getting folks together and talking about and diagramming, could be pictures, could be arrows, could be words, just getting folks kind of thinking in systems in a very, very abstract qualitative way. Now, system dynamics simulation and system dynamics modeling is largely focused on parameterizing these complex systems, actually putting quantitative variables into the model and using differential equations and seeing how things change and doing hypotheses in that way and testing hypotheses. My research focuses kind of at the nexus of these two approaches where the goal is to get local stakeholders with their intimate knowledge, understanding of the system, engaged in systems diagramming and systems modeling to make better decisions based on their understanding. So it's both kind of quantitative and qualitative to make decisions based on leverage points that are identified. So this is known as sort of structural analysis or group model building system dynamics modeling. And the idea is this is sort of a stakeholder-driven structural fact analysis for lack of a better word. And the goal of this approach is to develop a sort of a robust systems diagram or factor map that represents sort of stakeholders understanding of their local system. And then to then, once you have this map, pull out leverage points, pull out feedback mechanisms, influence between factors, centrality as a means to identify leverage points from a systems perspective. So that's kind of the goal of this. This is based on some previous work that I've done in Nicaragua. And for lack of a better word, we're just going to call this stakeholder, a stakeholder-driven structural factor analysis. And this approach that I'm continuing to develop with partners is quite simple. It involves brainstorming factors, getting stakeholders in a room, brainstorming factors that impact a particular outcome, let's just say service delivery of water. Condensing those factors into a digestible form, obviously, you can get a lot of factors, brainstorm, but the goal is to get between 10 and 15 factors to maintain some complexity and nuance, but not having to be unmanageable. And then mapping the interaction between those factors, you can see in that picture that's kind of showing a facilitator doing that. And then performing a systems analysis, and that's what I showed before, really sort of dissecting that systems diagram to pull out sort of leverage points and places to intervene in the system. And then sharing those results with stakeholders. So I wanted to run you through what this looks like really quick. So again, the goal is to get to this factor map that you can then dissect to pull out leverage points from a systems perspective. So first, identify the factors. So you basically ask the stakeholders, what do you think are the factors that in this case influence sustainable water services in your regional context? Where one of those factors is also the outcome, I'll explain why that is important in a second. Okay. And then you engage them in a process that's somewhat arduous. It takes between four to six hours, depending on the size of the matrix and the number of factors that are considered. But you fill out this table. And this is called a cross impact matrix. And you can just draw this on a board. And we've done this with without any access to electricity, right? You just draw this on the board, straw grid. And you get the stakeholders sort of considering every single interaction that could exist in the system between these factors. So let's talk about what this might look like. So one of these cells, let's just say community and operations and maintenance. So what that cell means, and you step the stakeholder through this process, you go and say, okay, based on the current conditions, how do you think community influences operations and maintenance? And this in and of itself is quite interesting. It's an arrow between community and operations and maintenance and influence between community and operations and maintenance. You say, how does that, what is the effect there? Is there an influence? Yes or no? If there isn't, great. There's no, there's no effect. If there is, then you say, okay, is there a positive or negative influence? In other words, if one increases, does the other increase one becomes better? And does the other become better and vice versa? And a negative would be kind of an an inverse or relationship was one goes up, the other goes down. This gets that dynamics. Okay, I mean, we can talk about that. We'll talk about that in a second. And then you ask, what is the strength of interconnection? Is it a weak, moderate or strong strength? And this allows us to really kind of add a little bit of granularity in terms of which factors are the most influential in the overall system. And what's really great about this process is the robust conversations that result from this, from each cell, each of these cells, to my chagrin at times, because it makes it take a lot longer, have had 10 minute long conversations about just one cell. Why does that, does that factor influence that fact? And the idea of doing this in a pairwise way is, is to extract local really rich information about a very complex system. We as people can't conceptualize all these interactions at once, but we can at least consider individually a pairwise interaction between one on the other. And this circles back around. And this would also have operations and maintenance on community as well. So what results is a, yeah, as an impact matrix filled with a bunch of numbers, which is interesting. But what it does is it allows you to sort of draw the factor map or the, or the model and then to pull out those sort of insights, those, those leverage points. So I'll, I'll step you through a couple of these leverage points. But first the factor map, you look at this thing, you go, this thing is, I can't get, I can't get much from this. And that's usually, and that's okay. A lot of times the main thing that we, that we get from this sort of factor map is to show the stakeholder group. Oh my gosh. Look at how interconnected these things are. And a lot of times you just see the look on their faces. Oh my gosh. They're, this shows that this is, this is not just as a simple task here. It's not just one simple factor influencing service delivery, for example. But then we, we stepped the stakeholders through this process of looking at this idea of influence and dependence. A factor influence is how many factors in the strength of interaction a factor has on other factors, sort of the arrow leaving the factor and the dependence is sort of the influence from other factors on said factor. And if you map this in an influence map, it's possible to sort of identify leverage points. So let's, let's go through this. So what we see here is if you have a high influence and low dependence, that would mean that you would probably want to target those factors because they wouldn't feed, there wouldn't be any feedback on those, on those factors and influence this in a very controlled way. Let's look at this example of water service delivery. So what we see here is community and capacity building coordination, for example, would be a place to intervene in the system and really, really fortify, so to speak. Whereas we see in the upper right that policy has both high influence and high dependence, which would mean it's very important. It's, it would have a large influence on the rest of the system, but in and of itself circles, it circles back from all the other interactions back on policy. So there's a potential volatility that that is or an instability if policy isn't protected, this system could spiral out of control. And then in the lower, lower right hand, we see what we would expect where water service delivery and operations and maintenance is highly dependent on sort of the outcome. Whereas the lower left is generally low significance, which in this case would show the users are not super impactful on the system as a whole, which folks may disagree with. But that's just an example. Another way to look at this is as a network and to use sort of network theory centrality to pull out which factors are the most influential on the system, most important. This is using between the centrality or the bridging of the importance of factors ability to bridge other factors. And then looking at that and Sanke, those are the leverage points of the system. So we would see here that financing coordination based on this analysis are the leverage points. And then lastly, the analysis of feedback loops actually systematically pulling out feedback mechanisms, specifically feedback mechanisms that impact the outcome variables of sustainable water services in this case. And then systematically identifying which ones are the most important, and that has to do with a lot of, with normalizing the influences and sort of, there could be hundreds and hundreds of feedback, but the goal is to sort of identify 10 feedback services. An example here is water services continue to function well, operations and maintenance will be easier to do or done in a more effective way, which will make water, water services more sustainable. So that's an example of that reinforcing R3 loop there. So that's another analysis we can do. And again, the goal of this is to look at the system as the stakeholders have conceptualized. It isn't necessarily the real system, but it's a system that is conceptualized by those stakeholders and identify the leverage point based on the interaction between factors, not just saying finances is important, finances is important because it interacts with factor A, factor B, factor C, et cetera. And so doing through these different analyses, it is possible to hone in on really what are, where are the leverage points using these, using these analyses in concert. We might say in this case, let's focus on policy, community and regulations, because it has a high influence, low dependence, high centrality and a high amount of, and a high reference and feedback, or a high feedback strain. So this has been done in a number of different places between East Africa, Central America and South America. And, but it's largely been research that has been done with the help of funds from USAID through channeled through the sustainable wash systems learning partnership, which is working in East Africa and is, was a grant that was won by the University of Colorado, and I've called University of Colorado Boulder and I've been working with them over the past four years on this research. So thanks to those, those folks. So future research in this area stakeholder driven structural factor analysis is to do more of this, continue to, to refine the approach, evaluate impact, we'll talk about that more in a second, and really to, to refine the process. And what this requires necessarily is to develop a software. I'm sure some of you computer science folks are looking at this, and this could be automated. Well, the process of developing these diagrams as a structure, as a, the civil engineer or civil systems engineer, we do is in a very brute force way. And it takes, it can take hours upon hours upon hours to develop these models. And so we don't oftentimes get to share those models to the stakeholders of real time, which is so developing an icon based software that we can run stakeholders through this process. And it's an output real time analyses of the system so that we can discuss them with everybody in the room. That's where the system. Okay, so that was talking about sort of how to get system or how to get stakeholders thinking and systems and extracting systems insights to make better decisions. Now, the other part of my research is focused on, well, what is the impact of these interventions, not necessarily factor mapping, per se, but other systems approaches getting stakeholders thinking about the interaction of factors doesn't actually impact their understanding. So I'll move on to this, this part of the research here. So the goal here is to evaluate whether stakeholders that have engaged in some systems thinking approaches, whether it's talking about social network analysis doing fact talking about life cycle analysis, et cetera, engaging systems in thinking in stakeholders didn't actually impact their understanding of the interaction of factors. And so the unit of analysis for this research is to really say, we can extract their mental model, this comes from social science theory, system dynamics theory, essentially a mental model is the way in which a person conceptualizes the interaction of things that lead to a particular outcome. We all have mental models, whether it's what we're going to go to the grocery store and buy, the food we're going to order, who we're going to marry, what house we're going to buy, et cetera. And so if we can extract this mental model from people, it's possible to evaluate the shifts and understanding that a particular intervention, systems intervention had on that understanding. And so this has been researched, there's been largely with the help of a sustainable wash systems partnership in Ethiopia, Uganda, and Kenya with a number of local stakeholders, we've interviewed these folks. And so we're talking handpump mechanics, local government officials, and we have over 500 pages of text so far. And this is the goal is to interview these folks at three different time steps with the same questions, and to see if their understanding has shifted on the interaction of factors. So this has been done or will be done on three different time steps. So far we've looked at baseline and midline, that's between 2016 and 17, 2018 and 19, the end line interviews have yet to be done, they're going to be done in the next couple months. And the interviews that we asked these folks, in these interviews, we say, okay, what do you think are the main challenges impacting service delivery, water and sanitation? And what are some of the solutions that you would suggest to address these challenges? And in getting them talking, it's possible to sort of in transcribing what they say, extract their mental models. So if let's just say a person said, the sustainability of water services is driven by the ability of the water user community to operate and maintain the water system, that represents a causal statement. And so it's possible to in asking these very pointed questions, to say, okay, that represents a causal statement. So if we can, how do we can we extract that causal statement? And we see there's actually an approach called purpose of text analysis and system dynamics modeling, where you actually code these, these causal statements. So what we would see here is, okay, water user committee capacity causes an effect on operations and maintenance, causes an effect on service sustainability. In other words, you can pull out systematically cause and effect factors that then led insight into how the stakeholders understand how factors interact. And you do that at different time steps. It's possible to see changes in those cause and effect factors to see if there's been a shift in those interactions. And that can be done based on looking at the challenge and solution questions, seeing how those factors are different for those different questions, and also look at different stakeholder groups and see how different stakeholder groups have shifted their understanding over time, among other things. And what's interesting here is you do this enough times, and this takes a long time and I'll talk about that in a second, but you code these transcripts. Let's just say you get 20 causal statements from one interview. You do this with 30 stakeholders for a particular region and you get 600 interactions, right? You get a rich, rich representation of that stakeholder group's understanding of the system, which allows you to thereby evaluate understanding and how shifts and that understanding on factors has taken place. So this is a, a graphic that would show, and again, the goal, the crux here is trying to show how understanding on factor interactions has changed. But this is an example of how we're presenting this for a case in Kenya where the factors that were identified through the coding were operations and maintenance, service sustainability, finances, et cetera. And we can show here a change in the reference, the percent of reference to effect factors and cost factors. So operations and maintenance you can see here has decreased as both a cause and effect factor. Let's look at this some more. We see here the service sustainability. What this, what this chart is showing is if the solid line, it's an increase between the baseline interviews and the midline interviews and a dotted line or dash line indicates that there's been a decrease in reference. So we see service sustainability becomes more influential in the eyes of these stakeholders. Finances becomes more influential. We see coverage and access becomes more influential. It becomes more of a cause factor influencing the other factors. Conversely, we see external support becoming less influential and less affected. And we see here the political influence becomes also less, less, less influential. Again, this isn't saying that the system itself actually is changing in this way. It's showing that the stakeholder's understanding of the system has shifted. And like I said, you do this enough times and you get some really, we're getting some really, really, really interesting insights on how people's conceptualization of the system is changing. It's a very robust analysis based on both aggregated stakeholders as well as stakeholder groups that can be parsed into. And that's great, wonderful. But this takes forever. We have 500 pages of text that we've been coding. And this takes at least two hours per interview. And this is thousands of hours of work. And so the goal is to, in making this replicable in the future, is to automate this process, to be able to allow folks to be able to import transcribed interviews into a software, so to speak, that outputs those interactions to evaluate understanding or to potentially evaluate the system itself. And so what we're investigating is the use of natural language processing to automate that process, importing the sort of the transcribed interview and outputting the factor map and the analysis. So lastly, this idea of seeing if there was a connection between understanding and service performance, right? The holy grail here is we want to impact service delivery. That's what the outcome is, what we're interested in. And at this point, it's still a bit of a black box, largely because to identify and say, okay, whether a stakeholder groups understanding informed actions that led to improved service delivery or sustainability, it takes an enormous amount of time both to do as well as to see if it actually materialized. So sustainability implies that a service worked for a long period of time. So evaluating sustainability in and of itself requires time, and we haven't been able to fully see that yet. Even this USAID partnership, we've only had five years to look at service delivery and see shifts in delivery. And that's not necessarily a long enough time to see if there was actually the impact of this desire. And then this is even harder, attributing shifts in understanding to actions to service delivery. It's a quite spurious and difficult task to do that, but is an important task nonetheless. And so future work and steps forward is to really develop more case studies, more and more case studies, engaging with more stakeholder groups, more practitioners to map the system, get stakeholders thinking in systems, and to evaluate those efforts. And so what I'm doing with a number of colleagues is to develop a consultancy company that develops both the software to automate these processes of mapping and evaluating stakeholder understanding, but also to just get practitioners to have their stakeholders think in systems or getting local stakeholders to think in systems to decide having an impact on service delivery now. We would expect that there would be positive impacts. And it's sort of a separate arm of that consultancy and development of that software is to continue to do research on evaluating the impacts on stakeholder understanding and on service delivery and in refining the process of engaging stakeholders and systems. So that's really where it's at currently with the goal of again, automating the process of mapping, evaluating understanding and evaluating service delivery outcomes in a way that allows you to adapt to changes in the system. The system is always going to be changing. Understanding is always going to be changing, but to be able to be nimble and evaluate that in a very quick way will lead to this sort of adaptive learning and action that will lead to systems change, positive systems change. So thank you for listening. That was a lot of information to throw at you. Again, the goal was to give you enough information so that we can have a more robust conversation on how to collaborate or how to make things clear. And I want to highlight two of my collaborators, Dr. Nick Valcourt. He and I are starting this company that I referred to. And Dr. Walker Orr is a computer science faculty member at George Fox University. He's helping with the natural language processing, the machine learning aspects of this research. So thank you. I look forward to chatting more about this with you all. Thank you so much, Jeff. We're going to invite questions. So please do share your questions in the Q&A window. There's a number of them there already. I'm going to turn it over to you, Jesse. And perhaps I'll share my screen here. Oh, if Jeff, you wouldn't mind stopping sharing yours. All right. Over to you, Jesse. Yes. First of all, Jeff, thank you so much. This is the type of research I wish I was doing because it's amazing. So really great stuff. I really like the idea around systems level thinking. We have a lot of questions here around it. I have my own questions. So I'm just going to hijack everyone's questions, ask my own, because I'm so interested in it. But I'll start with theirs so I don't take all of the time. One of the things that I was wondering about and is asked here is what about situations? So you're talking about Uganda, Kenya, these are places in which people do speak English and we're part of the Commonwealth at one point. So English is a language. So perhaps there's less of this issue. But in places like you mentioned Nicaragua or in South America, I'm wondering about the translation in terms of, you know, you're creating these system models, but the language of the factors becomes quite important then, right? So I was wondering if you could speak on two things. So I'll just start first with just translating in different languages, right? So how do you deal with that situation as you come up with like trying to think about these, these ones, both for interviewing and for creating the stakeholder maps? Right. And so this is, this is great. So when it came to engaging folks in Nicaragua, I did some of this in Chile, I speak Spanish and I was, so I was able to engage with stakeholders in the local language. Obviously keeping it in the local language is really, is key because language has meaning, right? And the meaning of these different factors, while there may be similar factors in different contexts, the meaning of those factors is very context specific. The interactions are especially context specific. So when it came to Latin America, it was, it was, I was able to engage in that way, but when it, it was very much different when it came to engaging with folks in Ethiopia and in Amharic. We don't, none of the researchers knew that language. And so the way that we did that, and ideally the goal is to get local facilitators to do these factor mapping workshops and to engage with the stakeholders, first and foremost, but especially if it's a different language, you need to have different, you need to have local folks that, that know the language, that know the context, that know the stakeholders doing, actually being facilitators, that you don't need to have some gringo doing these, you know, these workshops. And while, yes, certainly there can be, we've sat back in the background and try to not be, not to, not to be a hindrance, because obviously always having someone different in the room makes things weird, and excuse the conversation, but to have someone in the back at least taking notes and then debriefing with sort of the facilitator afterwards. And again, one of the, one of the blind spots here with this, with this research so far has been the ability to, how do you actually engage the stakeholders afterwards and evaluating the system's outputs, because usually we've had to be like, okay, thank you for engaging in the factor mapping exercise. Now we've got to go back and just kind of process these diagrams and then hopefully present them back to you at some point. And in many cases, we haven't been able to present these insights back to people. And the main insight that folks sustainably get is, well, that system's really interconnected and, wow, one or two factors were really influential. But so that said, that the translation and sort of the engaging with the, with the stakeholders is absolutely paramount. And so you need to have a really, really good facilitators to engage with them. Because you're at each cell, each interaction has to have consensus. You want to get consensus with the stakeholders. And it takes a long, a long time. It takes a lot of rapport with the, with the stakeholders. And it's not something that you can just go in and do. It's, it's, it's really, it's really sort of an art that the facilitator needs to know how to do. That's a really good question. And it's still something that we're working on. It's, it's how do you, how do you develop sort of a robust training for facilitators to be able to do these workshops? Yeah, thank you, Jeff. I think that's a really great point thinking about the quality of the facilitator and thinking about the team that you have, the impact on your ability to conduct it. It's not just the process that's important. It's like the team and the implementation of that process. And the details around that seem incredibly important based on what you're saying. And I agree with that. I think that's, that's a really great point. And I think it addresses many of these, these questions that are coming up around perhaps the quality of the interviewer, the style of the interviewer addressing variation as you're collecting this qualitative research data. You know, in those methods, you do have to think about your own positionality and perhaps the nationality of people on your team and their characteristics with respect to the community. And so, you know, I think what you're suggesting is perhaps having local facilitators or people that are really engaged with the stakeholders and have a connection with the stakeholders prior to your work may, may be addressing some of those issues already. So I think that's a, that's a really great point. We do have a couple other questions. I think one question that's important is to think about, you talk a lot about, okay, well, our goal is to get people to do systems thinking basically is what I heard. Right. So this is one process by which perhaps by going through this, people can be reflective about thinking, doing systems thinking themselves as stakeholders. Right. So one question that I would have or that, that came up here is what are some of the common roadblocks to people thinking in the systems way. So do people go through this? Do they improve on their systems thinking at the end? Or, you know, and if so, what are times where that worked better or worse interacting with some other thing that happened? Right. So whether cases where we did this and something else happened such that people really got it and started thinking about other issues in a systems way. Right. Because it can be quite difficult to try and shift these. So you're measuring these shifts and mental models. I'm wondering if you saw some cases where it worked better than others. Yeah, this is, and that really is the goal here is to be able to evaluate, first off, to be able to have solid shifts and a more complete understanding of the local system, but also to be able to evaluate that understanding. And so, yeah, right now, we're still looking at, first off, I don't know if anyone noticed this, but I didn't say anything about was there an improvement in understanding. Just the research at this point was to just look and see if folks were conceptualizing the interaction of factors differently. And whether that's an improvement is still, that's also an area for future research is, how do you know if their understanding has improved? What does that mean? But definitely, we've had, we've had some workshops that didn't go too well. And that was largely because we had people in the room that were kind of antagonizing the workshop participants, not antagonizing them directly, but sort of indirectly folks in the room that the stakeholders are like, I don't really want to say, you know, that the government is negatively influencing me. Right. And so in many cases, it's really a major hindrance is not having the right people in the room. The robustness and benefit of the system mapping exercises lives and dies by the ability to have the right people in the room. And also just, you know, we're seeing it in the United States, right? It's hard to change people's paradigms. This is just a one factor mapping exercise, right? But folks, their mental models have been impacted by their surrounding for years and years and years and years. So a lot of these are, a lot of these, a lot of the understanding that exists is entrenched and will always persist, no matter if you engage them in systems thinking. And so, yeah, I would say that generally folks that don't, I suppose, don't have an open mind or are very much stuck on their ideas are more likely to not have that sort of shift in their thinking, whereas folks that are, were facilitated well in a workshop were with the right people and who kept sort of an open mind to, wow, there actually are these interactions. And wow, my understanding of that interaction wasn't correct in my mind, I guess. And now I've been convinced that I've talked this over with my fellow, you know, workers or colleagues or what, what not. This has changed my understanding. It really comes down to a lot of different factors that go into that. But well, this stakeholder understanding evaluation research is still underway and that'll be a very interesting point out is what approach had the most shift, what approaches had the largest shift in understanding, what seemed to be hindrance to those shifts, if there are any. Good question. So, so what you're saying is you want other people to engage with you on this research, people that were asking these questions in order to to help you try and answer those as a research community. That's what I'm hearing. I mean, I think absolutely to be to do more of these, to do more of these. This sounds really great. So I had, there was a question here and I think this is an important one that you just sort of touched on, but I want you to perhaps elaborate a little bit if you could. So this is a representation of the stakeholders understanding their mental model of the system and how it operates. So I have two questions. Number one, and people are asked that, you know, I'm going to use this classic for people are asking, right? People are asking in the chat around, was there anything that really jumped out at you that surprised you about you went through this exercise and then you're like, wow, the mental model is like completely different than my understanding that I can get just from talking to them, right? So, you know, I think this is a key thing when we think about doing development work is, you know, we're telling people as they do design, go do observation, go do interviews, go do talk to people, we're talking to all these people, you're creating, synthesizing a mental model in your mind, how that system works, right? And now you're creating an explicit representation of how it is in the stakeholders mind. Was there times where you yourself were very surprised or people in your team were surprised, the facilitator was surprised, like, oh, I believe the system to work this way. But the stakeholders all identified this completely different thing that I hadn't hadn't even considered or they thought this was important and I didn't think it was important for example. So can you, was there a time where you were sort of a counterintuitive result was surfaced by doing this technique? Absolutely. Absolutely. So the answer is yes. For the most part in the system's literature that I've seen this a number of times, these sort of modeling efforts tend to produce 90% of kind of outcomes that would confirm a suspicion or by the team, it's like, oh, okay, I kind of saw that coming. That made sense. That really just reinforces my idea. And about 10% of the time it's like, oh, my gosh, I did not see that. And this is a difficult, this is a difficult notion or difficult concept because in many cases, if a counterintuitive result comes out of the system's diagram, which you would hope for, right, that's like, oh, my gosh, like, now that really shed some light on an area on a blind spot at a place where we should be intervening, but we're not. It's difficult at times because a lot of times people see that, they'll see. So for example, there's a large shift necessarily in the sector and the water and sanitation sector towards more sort of utility approaches, sort of preventative maintenance approaches, sort of, I wouldn't say top down, but more not just having the community maintain the system, but engaging the private sector, engaging local governments and maintaining the system, which makes a lot of sense. But in some cases, because we're engaging local community members who have been engaging community-based management, a lot of times we're actually finding that the community, the water committee, is actually really, really, really important in the system in maintaining positive interactions between various components of the system, various factors. And so in many cases, we see that the local water committee and the willingness to pay and things of that sort are really important, but they kind of go against the grain of where existing sort of the paradigm is going with the sector and saying, water committees, the community-based management strategy is no longer as effective as we thought it was in the past. We need to have more of a top-down approach, which, again, is good. In many cases, folks will look at that, that particular outcome, which I say the community is really important, and say that doesn't seem to jive well with what we're seeing in the sector or the direction of the sector. So that would almost point to an error in the system diagram. In other words, this is not represent the real system, that's just what the stakeholders think. And so it's really hard to, there's been a number of times where, yes, it's been counterintuitive, and they influence the politics or the importance of community social dynamics or social cohesion has come up, where the discussion has been along with the lines of, wow, that's a really interesting place to focus. But that doesn't seem to make sense to me. And so really, the difficulty is seeing those counterintuitive things emerge and to not say, oh, well, that invalidates the model itself. Obviously, something's wrong here. And in actually saying, okay, wow, again, this is the open-mind idea, looking at it and going, oh, okay, well, then what does it mean to potentially target that area of the system? Or maybe we focus another mapping exercise around that area of the system to see if we can throw it down and understand it better. But again, to summarize, yeah, in many cases, we counterintuitive things do come up. And the issue is not looking at those things and saying, well, that invalidates the model. Yeah, that's a great point. I think I'm going to ask two questions here in response to that. And I think that was a great answer to the question. So hopefully, people were able to take away and answer the questions, several of the questions that were in the Q&A. So I'm going to touch on two things. Number one, I think what we're really talking about is that this is a representation of the stakeholders' understanding. And then, but as engineers, when we do this complexity system stuff, often we're drawing these connections from like a first principles perspective. So we assume that that phenomena is true, right? Like that it's a ground truth. And so when you say like, this doesn't invalidate the model, it's like, that's assuming that you're equating the stakeholders' paradigm for how the world works with how the world works in some objective sense that the system exists in the real world, right? And so I think one of the things that's interesting about this, and I want to ask a question that was in the Q&A, when you're teaching students about this and you want to change the way the engineers are thinking, how do you do you get across? Like we want people to think in the system's way, but when we're doing that, we have systems that involve people, which is what these are. How do we understand this difference? Like I would imagine that my students, if they're trained in like an engineering mindset where everything is like a physics, there's a right answer, right? Here, this is a representation that we want to use to be useful to create interventions which we think will affect the system which includes people, right? So we need to understand how those people view the system, right? And so I'm wondering about that tension between, okay, this is how the stakeholder group views the way the world works, their mental model versus perhaps the designer's mental model or the team's mental model researchers or and perhaps how like the phenomena, the actual impact of these different things, because you could do like a causal experiment for some of these factors, but not for others, right? It depends on the definition of factor. So I wonder if you could talk a little bit about that tension, both about teaching and then the tension even in practice between objective whatever you want to call it and these sort of mental models, which are more representations of how people believe the system is working. Yeah, and that's a really good question, Jesse. Thanks for bringing that up. And this has actually been something that I've been up against for the last eight years is folks saying, well, you know, that's just you're just pulling out the stakeholder's understanding of the system. That doesn't necessarily mean that is the system. And that's very, very true. And I don't want to use this as a cop out. But there's the kind of age old system saying all models are wrong. Some models are useful. And that is to say, no matter what model we create, some models are more accurate than others, but most models still don't quite get that reality. So what if for a model to be useful, that's the most that's kind of almost more important than being sort of valid. And it's kind of as car before the horse or whatever it's, if it's useful, and it and it actually produces. And again, we haven't been able to show this. This is this is the blind spot. Just glad you identified this is just does the model that the stakeholders identify, allow them to make decisions and to, to effect change that ultimately leads to a better outcome of service delivery. And if the better outcome takes place, you would ideally or hypothetically say, okay, well, that's because their understanding shifted in XY ways, and that the modeling exercise was in effect useful. And ultimately, the model that resulted from their work, the workshop was relatively represented of the real local model. But so so really that the proof is in the pudding, so to speak, it's really that the goal here with this sort of research is to sort of loosen our grip on this idea that all these models have to be fully validated. I would love for these models to be able to be valid and a very scientific sort of engineering sense, right? That's the goal. I would love that. And to actually start to be able to put real quantitative insights into these systems diagrams that develop system dynamics models to be able to do virtual sort of simulations and sensitivity analyses. This is where I would love for it to go. But at the present time, it's, it's, it needs to be enough to just be able to say, did it change stakeholders understanding of the system, they become a more complete understanding of the system. Did it tend to move towards, and this is a bit hard, but it tend to move towards the understanding move towards a more, if the sector is going towards preventative maintenance, and that's been shown to work better, is the stakeholder understanding going towards tenants and aspects of preventative maintenance, in which case we might say, Oh, okay, well, then that seems like their understanding has improved or at least been changed through engaging in these systems activities. But really to be able to have a validated, a validated model will require more and more research. And that's, and I think that's going to be important now bill is sort of the legitimacy of these methods. Sure. So I think we have about five minutes left. So I'm going to be selfish and ask my question and then great questions. Thanks. Well, these are questions from the people and everybody else. Thank you. So this is my question. So it'll be worse for sure. So my question, I have so many questions, we're gonna have to have a phone call later about it. But my question really is about, can we connect this perhaps or your thoughts about the future direction of this type of work? Could we connect some of these things to sort of an implementation science perspective, right? Yes, you're just talking about some of this tension between like the real model, like, Oh, well, we understand that this service thing actually, you know, does this, you know, affects the sustainability of it. But everybody in the community believes that the government's input is actually the most important thing, right? And so the intervention, if you're saying, okay, well, we're going to try and implement this preventative maintenance structure, because we know through past experimentation, that this works better. You may have trouble getting adoption of that intervention or uptake of that intervention, if their mental model thinks that it's unimportant, right? And so I think that connecting this type of understanding and stakeholder analysis, stakeholder understanding analysis to this larger question of, well, we believe this other thing for some other reason, whether it's physics or prior research, actually has a better impact. Well, that discrepancy may prevent uptake or adoption. And perhaps we need to change the way we're facilitating the implementation or the way we're marketing the implementation to those stakeholders to that particular stakeholder group, or provide evidence in a different way such to think about their shift. And now you can measure their shift in their understanding also, right? So that seems really important. I would think that to get to my question, have you looked at creating stakeholder, these maps, these understanding mental model maps for different groups of stakeholders, and then measuring the differences between those maps? So that would be my questions. Maybe you could say, as a team, hey, stakeholder group one, stakeholder group two have very different mental models, and that means we need to change what we're doing. Absolutely. So this area of alignment. So yeah, this is something that we've looked at quite a bit. Very, very good question. And I think you touched on a lot of different things. So really quick, the implementation science aspect of this, I think is critical. And I think that's why I'm seeking to develop sort of a consultancy to kind of promote sort of thinking and doing more mapping, having more case studies, doing more evaluations, looking at different stakeholder groups, how they conceptualize the system differently. Because you're right, that's a really important outcome is seeing, well, these different stakeholders see the system very differently. And these stakeholders have their finger on this button. And these stakeholders still speak on this button. Maybe which system needs to be not shifted or what paradigm needs to be shifted? What sort of understanding is more relevant to the sort of the box or the boundary of the system? And if it's at the local community level, and they're seeing the system completely different from what the, let's just say the regional government is seeing, and they're kind of the ones that have their hands on the first strings, then there's obviously a problem, there's a disconnect there. And that's a place that there would show a place to intervene in the system, saying, okay, we need to engage these two stakeholder groups in a robust discussion about why these models are different. But yeah, ultimately, this is going to just require more and more case studies. And this idea of implementation science implementing, it's both an implementation of a factor mapping exercise as well as evaluating the implementation decisions those stakeholders took as a virtue of or because of that factor mapping exercise. So yeah, this implementation science is really the center of where this is going. And iterating, again, this is all an iterative process of continually going back and meeting with the same stakeholders over and over again, seeing how their understanding has changed, how the system has changed. So, thank you, Jeff, for all of those answers. We're now out of time. So I'm going to turn it over to you. I want to just recognize that there were several more than several questions in the Q&A, which I think were critical about representation, you were talking about who is in the room, while you're doing these things, who is the facilitator, who's on the design team. All of these things are incredibly important to building an understanding of the stakeholder group system as a whole. We didn't even touch on how do you identify who the stakeholders are, which I think is another one, you could create a mapping of the stakeholders in a very similar way, and what are their connections and that might be interesting also. So I just want to recognize for the people that I did not get to your questions, I apologize. We've run out of time was trying to synthesize as they came in. But Jeff's contact information is there. This whole forum is intended to build connections between everybody that's in the audience right now and the people that are presenting. So in discussions with Jeff before this talk, I know that he's happy to try and connect with people and discuss some of these questions in a more in-depth basis. And with that note, I want to thank everyone for joining personally, because this has exceeded beyond my wildest dreams the attendance things that I thought would happen for this type of seminar. So I'm really happy that you guys are all here and asking such great questions. I want to thank Jeff again for his great talk, and I'm going to turn it over to Yana to wrap it up. So thanks again. Thank you so much. Thank you, Jesse. And I whole heartedly echo the gratitude to Jeff. This has been so enlightening and we've already identified some parallels even in our own programs and how we actually train our fellows to conduct interviews. So a micro model of your system's approach. So with that, I do want to thank everybody for participating. We encourage you to join us for our next seminar, which is going to be next month with Dr. Erica Graloff, who's an assistant professor in the Department of Engineering Management and Systems Engineering at George Washington University. So the system's conversations continue, our system's perspectives, which we're so thrilled for. I do acknowledge also, as Jesse said, we have a lot of questions that went unanswered just because of time constraints. So if your question didn't get answered and you're very eager and you didn't catch Jeff's contact details, feel free to email us and we'll do our very best to convey them. We might also consider a follow-up article with Jeffrey compiling some of these questions and providing a synthesis to our attendees. With that, I wish you all a good afternoon, a good evening, or even a good morning, depending where you are right now or when you're listening to this recording. And we hope you'll join us for our next seminar. And thank you all. Goodbye, everyone.