 Okay, so I'm Jane and Aaron and myself, and we've got a couple of our other members online today, Jonas, Catherine and Jerry, and we're all part of a multicultural evaluation special interest group at AES. And so today's seminar is, you know, something that we like to do, particularly on issues that relate to, you know, what is equitable evaluation? How do we do, you know, what does it mean in practice? And exploring the variety of multicultural frameworks and approaches, but I've sort of gone about this the wrong way, and I should just pause to acknowledge country. And in the spirit of reconciliation, multicultural evaluation, Sieg acknowledges traditional custodians of the country and First Nations people throughout Victoria, Australia and New Zealand, their connections to land, sea and community, we value Aboriginal and Torres Strait Islander people's history, cultures and knowledge and pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander and First Nation peoples. So yes, and today we've got Bob, who's a systems thinker and evaluator. And he has been in the systems, using systems, concepts and thinking in his work for over 30 years. And his specialty is complex and difficult projects and programs with a focus on approaches that are robust and rigorous, and but at the same time, produce information and analysis that are good enough for the organisations that need to use that information. And I think rather than me, oh, so, and so the other thing I just wanted to say is that Bob's pre recorded his presentation today, and we're going to pause about 15 minutes in and open up the room for about 10 minutes of Q&A discussion. And then we'll resume for the final 15 minutes followed by the remainder of the session as an open Q&A Q&A discussion again. And both Bob and Aaron and myself will be monitoring the chat. So as the presentation kicks in, if you have any questions or thoughts, please pop them there. And I think we shall get going. OK, let me just check everything before we start up. And just to point out that I'm overlooking the the harbour of the Punganiri Akara in Wellington. But I'm speaking from the Rohe of Tia Tiawa, one of the people of the local area. So welcome, everybody. Thanks very much for giving me this opportunity to to think a bit more about the relationship between ideas from the systems field and ideas from the evaluation field and in particular around the area of equity. As I said, this is I've got a particular throat problem at the moment, which means that if I talk to much more than about two minutes, I descend into the course, which is one reason for the pre-record. But the other pre-record is that the other reason I discovered is that actually, I seem to be far more coherent when I actually have to sit down and listen to what I've said before than just riffing on a particular topic. So the first section of this is about 15 minutes long. The previous course, we have tested it that everything will work. And then we'll have a bit of a chat and then we'll go on for another 15 minutes. So let's go. I'm going to do a couple of things. This presentation and discussion will explore the ways in which the adoption of systemic practice and evaluation can help address issues of equity. Now, I'm going to present a particular framework of understanding what constitutes a systemic framework. And then I'm going to explore three practical ways of using that framework. Now, I don't expect you to get absolutely everything I present to you in the next half hour. There's an awful lot of stuff in this presentation that may be new to you. But however, I do anticipate that you'll be able to take quite a few things away and begin to use them in your evaluations, especially those with a particular equity bias and orientation. There will inevitably be time for discussion as well. Discussions about equity within evaluation tend to fall into two categories. The first category concerns how evaluation looks at the intervention and explores processes and outcomes through an equity in that intervention through an equity lens using equity criteria. The second, which is often described as culturally responsive evaluation, concerns how the evaluation itself promotes equitable outcomes directly as a consequence of the evaluation. Now, in this particular talk, I will focus largely on the second. Why is that? Well, partly the reason is practical. Evaluation has been working on the issue of exploring how to assess interventions through an equity lens for quite some time. And I think the ideas from the systems field, while they can be valuable, and we can talk about them at some other stage, I don't think it's a game changer. I believe the major impact will be in the way in which evaluation can use systemic ideas in the way that it is practiced as a craft. In other words, evaluation by becoming more systemic will actually contribute to equitable outcomes of the evaluations. Why am I interested in that? Well, I think it's an issue of professional ethics. How can we stand in judgment of interventions around equity when we are not ourselves focusing on equitable outcomes of our own practice? As a craft, we've talked a lot, a lot about use, utilization, and usefulness for many years. But really, only recently have we been seriously considering the question, useful to whom? So let's start on this journey of how we can encourage evaluation to become more systemic by stepping back and actually looking at this thing called the systems field. And here's a diagram produced by Ray Eisen, one of the leading figures in the evaluation field, of how he considers, you can describe visually, what comprises the systems field. Now, I invite you first to look at the left-hand side of the column. And what you will see there is a very substantial list of human endeavor, human philosophies, human ideas, and practices that have actually influenced the development of the systems field over the last century or so towards developing the kinds of methods and methodologies that you will see listed on the right-hand side of the diagram, some of which you may be familiar with. Unfortunately, some of those things that you may have heard about have got slightly changed in the telling and have often generated some inaccurate understandings of what a systemic approach actually is and what the systems field represents. What I've got here are three myths, very common myths about what the systems field and a systemic approach is all about. The first one is that systems are universally self-evident and real. In fact, systems are human constructs. One eye could look at exactly the same situation and see completely different systems there. The second one really is that the myth is about the big stuff. How many times have you heard people talk about at the systems level? This is a terribly limiting idea of what the systems field is all about. Systemic practice and the systems are scale-free, essentially. A cell, as well as the universe, can be considered a system. The third myth is probably the most dangerous of them all. That myth is about the systemic approach and systems approaches and systems thinking about including everything. That's actually practically and intellectually impossible. What the systems field is about, actually, and what systemic practice is about, is being about very smart about what you leave out, not trying to include absolutely everything including the kitchen sink, which will only just confuse you. Now, a couple of slides ago, I showed you this enormous range of systems and systemic approaches, very diverse, with lots of different ways of addressing various systemic issues. But what do they actually have in common? Well, what makes them a field, in a sense? Well, let's start at what comprises a system around which there is some general agreement. Now, here is a relatively common definition of a system. A system is a collection of entities that are seen by someone as interacting together in order to achieve something. And here's a visual way of representing this. What I've done is organized the definition of a system around three core components, interrelationships, perspectives, and boundaries. Now, it should be fairly self-evident that that particular shuffling around in terms of interrelationships, a collection of entities interacting together, that are seen by someone, a perspective, different perspectives will look at the same things in slightly different ways. But why are boundaries about achieving something? Well, that goes back to the third myth. We cannot look at every single collection of entities interacting together. We cannot take every single perspective on those entities either, or those interactions either. In order to do anything, in order to respond in any way, in order to achieve something, we're going to have to start deciding which of these interrelationships and which of these perspectives we are actually going to take notice of, and which ones, therefore, we are going to have to leave out. Notice also that the three elements interrelate change our perspectives and we construct different boundaries, change the collection of entities that interact together, and that we'll actually generate different perspectives. And both of those demand quite different decisions over where to set boundaries. However, we don't just want to observe systems. That doesn't really lead us very far, but we want to practice systemic thinking. So in order to do that, we need to add some verbs, some action words. And here are important action words related to systems thinking as practiced. Firstly, we have to understand interrelationships. We have to address the question, what is the reality that we're actually dealing with? In terms of perspectives, it's about engaging with multiple perspectives and ask things like, how do people interpret that reality? And then finally, in terms of the selection of boundaries, which are often ethical and difficult and political, we should also reflect on those boundaries so that we address what is politically and socially and culturally feasible and desirable to do. The remainder of this talk is now going to explore each of these three aspects and how they can assist our evaluations generate equitable outcomes. Now, given that these aspects interrelate, it doesn't really matter where we start, as long as we just kind of keep going around the circle. And since this presentation is primarily about equity and equity can be considered really as a perspective, let's start there. Engaging with perspectives is all about the different ways a situation or intervention like evaluation can be understood. So take this illustration. There are two ways in which it actually can be understood, depending on your perspective. Some of you will see an old woman. Some of you will see a younger woman. The point that I want to make from an evaluation point of view is that the data is exactly the same, but we see different things. Now, I want to make an important clarification here. Different perspectives, taking different perspectives, engaging with different perspectives, is not about handling different opinions about the same perspective or the same understanding, but genuinely different lenses through which a situation or an evaluation can be viewed. Now, I call this particular viewpoint a framing, which, of course, begs the question about what exactly I mean by a framing. A framing is through the lens through which someone views the situation. It addresses the particular issue of how different people would complete the sentence, a system that is something to do with, a system that is something to do with health, a system that is something to do with equity, a system that is something to do with the improvement of a particular transport system. Let me give you an example of an needle exchange program. Some of you may be familiar with the concept of needle exchange programs. They were very common in the 80s and the 90s. Basically, the idea is that intravenous drug users, rather than sharing a needle, can actually take a needle to a particular site and exchange a dirty needle for a clean needle. Now, there are three ways in which, at least three ways, in which a needle exchange program can be viewed, can be framed. The first one is for HIV or indeed any blood transmitted infection. It's the reduction of HIV. The second one is that actually many of these places turned out to be very safe places for scoring. People were actually not hassled for actually taking drugs. Deals were done with the police and so on in practice. And the third one was, they're not always terribly popular in the local area. Local people often see them affecting their rental values, their property values, or at least the sort of reputation of their local area. Now, all these three are different framings. And if you think in evaluation terms, if you were evaluating a particular needle exchange program, depending on your framing, you would come up with quite different ideas about whether the program was worthwhile or not. Now, returning to evaluation, very often and very commonly, evaluations are framed as effectively something to do with accountability or something to do with program improvement or something to do with learning. However, in the context that we're exploring, the implications of taking an equity framing, in this case, our evaluation is a system that is something to do with equitable outcomes. How on earth do we actually unpick this and use the idea of framings? Well, one of the ways that I do this is through the concept of stakeholder. Now, let me clear what I mean by a stakeholder. A stakeholder is someone or something, it can be a thing, affected by or affects someone or someone, something that is of interest to us. So, this is one way that I manage this issue of framings and begin to kind of expand my understanding of the consequences of taking, and say in this case, an equity framing, something to do with equitable outcomes. Firstly, the system has stakeholders and they play a role. In this case, I've just got five example roles of an evaluation on something or other within a school setting. The reason why there are multiple roles, it's why we're talking about roles rather than stakeholders, is just to highlight the issue that actually people can take on multiple roles within a particular system. So, the principle in many smaller schools would actually also be the maths teacher. In many organizations with internal evaluators, they often may play a practical role in whatever it is. And, of course, the maths teacher in this particular case could well be a caregiver, another role that's not there, of some of the students. So, that's the stakeholder role. So, the next thing I look at is, well, what actually are the actions that are associated with that particular stakeholder role in the system? And remember, this is an evaluation system. This isn't necessarily purely the intervention that we're talking about. And I've got some examples there that you can kind of look through as I work through. The third column essentially just reminds us of what the framing is, because, of course, in many evaluations, we often deal with different framings. And multiple framings. And multiple matrices like these. But for this particular example, we're just focusing on equity. And then the final column, which is perhaps the most important one of these, is to try and understand what the motivation and interest these particular roles have within the system, within the system that we are talking about, which is the evaluation system. So there will be reporting requirements. There will be issues to do with school ranking. There would be commitments to equity in terms of the principle. The math teachers, primarily, motivations may be in terms of the overall intervention, keeping up grades, but in terms of the evaluation system outcomes, there may well be a sense of fairness. The evaluator themselves, they may well be motivated by a commitment to the evaluation outcomes, the use of the evaluation and its usefulness. And also motivated by particular aspects of evaluation standards. The student themselves, in the end, maybe just want to be a better math student. The evaluation commissioner has lots of fascinating potential motivations, but here we're just saying that, we don't have a commitment to equity, but they're also interested in their own professional capacity on the performance of students. Well, that's quite a lot to take in on a first go. The issue of what actually is the systems field and various ways of understanding what systemic approaches are, and then exploring in some detail what engaging in multiple perspectives actually means in practice. So I'm going to sort of pause there for a while just to have some discussions. Before I briefly talk about the area of understanding interrelationships and then spend a lot more time on exploring issues to do with the setting of boundaries and reflecting on them in order to decide what actually is feasible and desirable to do in our evaluations in order to create equitable outcomes. So onwards to interrelationships. Right. So I'll just finish my screen sharing here and bring everyone back to some sense of collective understanding. And let's just have a bit of a chat for a while. Please feel free to chime in if you've been thinking about anything or you have a question or even if you want to challenge some of Bob's thinking. Jonas is asking which angle is more important and then Michelle is suggesting with framing in systems thinking and sort of thinking about equitable outcomes. Is there such a thing as one of those aspects being more important than the other? Yeah, good questions. Jonas, can you just explain a bit what you mean which angle? Yeah, thank you so much. My understanding is that when I'm saying angles, the perspective, the boundaries and the interrelationship. So is it the boundaries or is it the perspective that really matter? Or are they all equal? Yeah, that's a great question. If you ask me, I get particularly excited about boundaries. But we can talk about that at the end of the boundary discussion we're about to have. However, I must admit when I first started playing around with these ideas and developing them or when a collection of evaluators and systems people were beginning to develop this kind of triangle idea that really, this isn't the only way of understanding the system. I was really interested in perspectives. So, you know, I think to a large extent it depends where your own particular stance is within your evaluative activity or in your life in general. And the official line would be they are all equally important. But obviously some of them, in any endeavor, some we feel are more important than others. But remind me to address that question after we've done the boundaries because I will then explain why I think within an evaluation context that it's kind of boundaries first, perspective second and interrelationship third. Natalie's got to... Oh, sorry. Yeah, yeah, done. Natalie's got a question. Natalie? Yeah, thanks for that. I guess for me it's that sense of... because I work with Aboriginal communities and being a funding body and also being Aboriginal myself is as we are designing a program I think there is the importance to actually create an evaluation plan for bringing all different types of people to that evaluation plan because sometimes when we say, oh, we want to make sure that it meets people's cultural needs or it's got self-determination and empowerment, we all come with different ideas around what these actual words actually mean. And so what we do is we actually clearly articulate what all the different types of words are as opposed to assuming what they are so that when we put the program together to fund the program, evaluate the program, we're all on the same perspective from workers to managers to funding bodies on what all these words mean and not assume like what we think success for a child is compared to what the community or the people think success for that child is as well. So I think we really got to think about our words because we all come with our idea of what those words mean. So I think that's really crazy, but that's my thinking. No, there's a very important quote that I use in a lot of my books, which is that words don't have meanings, they have uses. It is a myth that we can all share the same perspective. We don't. Your perspective, even though we may talk about what a word means, we still have different perspectives on that. And I think one of the areas within the systems field that made me realize that is to be really cautious about assumptions that we make around shared perspectives. It's almost impossible for us to share the same meaning of even the same perspective. And having said that, I'm not arguing against having those discussions, right? But I think you always have to keep this in the back of your mind. And there is actually a particular systems method that I don't talk about in this particular presentation that actually is very sensitive to try and unhick what those things actually mean. Sorry, I spoke over you. No, no, just the sense to also have, you know, the potential workers or end users as part of this program or whatever you're trying to evaluate so that people understand why it's important to collect this so that we can evidence up that their program works best for their community. But thank you anyway. Thanks, Bob. Yeah, yeah. I think the big challenge, I'll just add to that, the big challenge that we always have and this is what we'll get on to in the next presentation is you can never get everybody in the room. But we'll talk about it a bit later. You will have to in the end make decisions about who's in and who's not. And that's the time. And maybe sort of Melinda's raised something I think that sort of speaks to everything that we've been discussing and that's about power and so power relationships and how they inform the articulations of boundaries. So maybe as we watch the next 15 minutes be thinking about power and then we'll look back to that maybe for the final Q&A part of today's session. Please do because especially when you get into issues of deciding boundaries, power and agency become absolutely to the forefront. And again, I talked a little bit about a particular systems approach that attempts at least to expose these kinds of things, but that's part two. Any more on part one? I don't think so, Bob. So we'll kick on with part two. Righto. Okay. So thanks for those questions. They were really to the point. Much appreciated. So let's share the screen. Let's do that one. Is there a technical problem, Bob? We're not getting there. You're on mute as well. We go here. You got a copy of the PowerPoint. Here we go. Yep, that's it. Let's go onwards to interrelationships and boundaries. Now, I'm not going to spend a great deal of time on interrelationships. It's an area where evaluation has actually put in quite a lot of work in terms of understanding interrelationships within a particular setting. Having said that, the systems field does actually have some unique approaches to exploring interrelationships substantially through diagrams. And I recently published a book that explores six of those particular approaches. And if you're interested in that, then I would encourage you just downloading that book afterwards. These particular approaches would include things like rich picturing, influence diagrams and causal diagrams. And again, I encourage you to take a look at the enormous amount of information on the net which you actually want to look at that in more detail. And here is a way of expressing the learnings from your understanding interrelationships in terms of stakeholder roles. So I've just put a few examples of, in a sense, what that role is influenced by. Not necessarily caused by, but what that role is influenced by. In terms of outputs, what that role itself may influence. So it's a sort of a role based description of the various interrelationships that are within the evaluation system that is something to do with equitable outcomes. So now we have explored the somewhat large expanse of what could be considered relevant to achieve an equitable outcome. We've looked at different ways of managing and identifying engaging perspectives and the range of those and also hinted at the very wide range of interrelationships that any particular situation will display. It's a big palette and we have to start making decisions about what is ethically desirable and practically feasible in order to create in our case equitable outcomes of our evaluation. And in order to do that we're going to have to make some boundary decisions. Now every evaluation is the product of hundreds of boundary decisions. There are boundaries of space, of time, of purpose, of cost, of knowledge, of values, of perspectives, you name it. And the evaluation boundaries we choose will hugely affect its feasibility and its ability to deliver equitable outcomes. To do this I use a framework taken from a part of the systems field known as critical systems. Now critical systems considers that of all the many hundreds if not thousands of boundary decisions that are made in any particular intervention four are particularly important. Especially in tasks like the one that we're talking about which has some specific and potentially contentious goals like equity. Now the first boundary decision that is going to be made will be about who are and who are not going to be the primary beneficiaries of an equity framed evaluation. And those that lie outside of that boundary in the language of critical systems are termed marginalized since they lie at the margins of our focus. The second area of boundary choice and boundary decision is the fact that now we have to allocate the resources necessary to bring about these benefits. Now resources are not infinite we have limited time and money and other things and certainly we cannot include every perspective or every possible interrelationship in the particular area we just do not have the resources and the intellectual capacity to do that. Somebody some body sometimes somewhere has to have the ability to say this is what you have to achieve this is the resources that you have available and these are the rules for using those resources. Someone has to have the ability to control whether or not the resources are used in a way that the beneficiaries do actually benefit and the resources themselves are not hijacked to some other purpose. Someone else then is given the granted indeed the autonomy to decide whatever needs to be done within those boundaries. In other words a boundary decision has to be made between control the ability to decide the resources and the autonomy the ability to manage those resources exactly where that boundary lies in an evaluation and whether it's in the right place to deliver an equitable evaluation is a critical boundary decision and needs to be made visible and then reflected on in exactly the same way that the beneficiaries and those who are marginalized also need to be identified made visible and the choice reflected on. The third is about knowledge. Now as we know knowledge itself the use of knowledge and what is included and what is excluded is these days a very considerable discussion within the evaluation field especially in relation to indigenous knowledge. So it's important to understand what kind of knowledge is critical in our decisions about the use of resources and as I said earlier on there is so much potential information knowledge and skills out there that could inform these particular decisions that we cannot actually handle. So there will always be a need to draw boundaries around which information knowledge and skills are appropriate for the evaluation to deliver an equitable outcome within the resources that are under our control. The final area of boundary choice is legitimacy. Now just follow this logic for a moment. If those equitable evaluation outcomes are to be achieved and sustained then the evaluations equitable outcomes have to have sufficient legitimacy and support. Now in practice the greatest threats to that legitimacy are from those who are likely to be negatively impacted by an equitable evaluation outcome and especially those who have the agency and power to hinder or even prevent the outcome. How much an equity framed evaluation is prepared to trade off or negotiate with those stakeholders again is a critical boundary decision that needs to be very carefully deliberated on. So those are the four boundary choices. Now these boundary decisions raise issues about power and agency who has and who ought to have the power to decide these key boundaries and one of the great advantages of a critical systems approach is that it forces these often hidden decisions into the open for discussion and deliberation. So let's explore what this might actually mean in practice. So we've developed a matrix of interrelationships and perspectives and now we're going to have to try and work out how we're going to marginalize and what we're going to include. So let's say that in terms of purpose that our primary beneficiaries of the evaluation are going to be disadvantaged students and that we are prepared to marginalize in our evaluation the interests of non disadvantaged students. Now obviously there will be some debate around that but that's exactly the point of this sort of table. In terms of resources we're going to say that the school principal has the final say when it comes to the use of resources and the rules around that resource use. Within that the evaluation steering group which could be made up of various stakeholders has complete autonomy to decide whatever they wish to do within those resources available and within the rules that have been set. In terms of the knowledge well let's just say that we are going to include knowledge acquired and information acquired from disadvantaged students and non disadvantaged students and also we're going to include experience of other equity outcome focused evaluations. On the other hand we're just not going to take too much care of and possibly even ignore narratives and experiences outside the school community this particular school community or indeed the experiences outside the education field for instance. And also in terms of the technical side of the evaluation we're not interested at all in any kind of results or information from non-positivatory evaluation approaches. Now when it comes to legitimacy we're actually prepared to negotiate who is and is not evolved in some way in the evaluation process but one thing that we're not prepared to negotiate in order to in relation to ensuring a degree of legitimacy and sustainability is that we're not going to set to exclude the student the disadvantaged student voice in evaluation decisions. So there we have it. There's one example of the way in which you can kind of generate debate around boundary decisions using a critical systems framing. So there we have it, a way of understanding how to design and management in evaluation focused on generating equitable outcomes using those aspects of the systemic approach that considers understanding interrelationships engaging with multiple perspectives and reflecting on key boundary decisions. Quite a bit to take in. I'm going to pause again and invite some comments and reflections. So comments and reflections. Well I might sort of kick in maybe with more of a ponder than a question but it's it's a fair to say that you know like you know we have equity for one stakeholder group that that sort of I mean there's a sort of binary in that you know then it might not be equitable for another stakeholder group and you know how do we resolve this tension or at least acknowledge or manage in our practice have we ever had any sort of experiences with that situation? I think this is where we go back to the question of agency control and the nature of the evaluation economy I have a particular interest in the nature of the evaluation economy because the way in which evaluation has evolved in most countries is that we actually essentially ignore those challenges right? We leave it to somebody else make that decision and I think one of the things that is a substantial challenge to the way in which evaluation is proactive is the development of the range of systems ideas that I'm promoting because it puts these kinds of decisions out in the open and it actually loads a substantial responsibility for generating the discussion around those decisions to the evaluator rather than hidden in the whiteboards of the evaluation commission. Do I have any particular answers? No. The idea in all of this is one of them I believe is to shift the role and responsibility you could argue the professionalism of evaluation essentially away from being a trade doing people's bidding by actually taking part in raising and taking part in the kinds of discourses that I've been talking about because it's not very well to say how do we do it but at the moment we don't actually do it in general. Catherine? Thanks Bob this is great I just have a question around framing recommendations using this sort of approach and I'm wondering if given if you identify the different way framing of a particular issue and you notice that they're quite different but then you're providing recommendations to a donor but you want to use an equitable approach to that could you then frame the recommendations from different perspectives as well so if primarily the motivations of the donor were taken to account you might recommend this if you're taking the primary village people who are going to be benefiting from a particular activity would you then recommend this where you can't necessarily merge those recommendations into one nice neat recommendation could you frame the recommendations in that way from multiple different perspectives or what have you seen done or what have you done yourself this is one of the reasons why I refuse to deliver recommendations in my evaluation I actually believe for a whole variety of reasons that it was one of the greatest mistakes that evaluation has ever made is to play amateur consultants we're not paid to be consultants we don't have sufficient knowledge in the areas that we work in or in the specifics of where we're working in we're usually just in for a few days a few weeks we're actually proper consultants we spend an entire year trying to work things through so in a sense you're asking the wrong person because I literally refuse these days I will help the discussion around options and allow the key stakeholders and this is I think where it gets closer to addressing your question to develop recommendations themselves in a sense what I would I would kind of fudge my answer to you by saying that in a sense our job is to facilitate that discussion amongst the key power brokers and the key stakeholders that have a substantial interest in the outcomes of the evaluation and that goes back again to the whole issue of power and agency because who determines ultimately who has the ability and authority to develop recommendations that come off from any particular evaluation if you are concerned about evaluation consequences there isn't a simple answer to that we can't change the entire power structure of society through a single evaluation but I do believe it is the evaluator's responsibility increasingly if we wish to be a craft or a profession to raise these issues and comment on the power relations that we actually see as recommended as ideas for the future begin to develop I add to the discussion here Bob on this idea of workshopping actions or recommendations with the different stakeholder groups involved it strikes me that we put a lot of thought into when we do focus groups or surveys how we sample or how we include people in the data collection process but perhaps not as much thought into then how we interpret and make recommendations and I know in my practice data parties and the whole idea of bringing people together around a summit or something is something we're well versed at but I wonder if you've ever applied that approach a more nuanced approach to doing the analysis and sense making and action planning within an evaluation setting very rarely because of the nature of the evaluation market I mean I have fought literally fought evaluation commissioners people who essentially pay my wages to develop those sorts of ideas and they just trundle along their 365 page evaluation guidelines report and say this is how you do it there's so no I within an evaluation context no I haven't but I don't always work within an evaluation context and on those people okay yes I have actually managed to kind of pull those things through but there's no you know there is no there's no magic method about this I mean I think one of the weaknesses of all of these sort of slightly more edgy methods and I want you know my brief summing up to raise these things is there is an issue of safety and I think as we move closer to the boundaries you know as our evaluation approaches and methods move much more towards the boundaries of what is what is generally accepted of our craft and of our profession then I think we do actually we do have to keep a very close eye on safety I mean I was doing some work in Rwanda a couple of years ago systems were we were teaching actually that we were trying to sort out a particular issue about kids hating school now in Rwanda you get killed if you go against all the gown me it's as simple as that people follow you anywhere in the world and kill right so in doing this sort of influence mapping thing we had to literally design the process so that it maximized that allowed us to have a systemic approach but maximum ensured that the various participants were organized and the job was framed in such a way that no one was put at risk because I think that's a really really important area as we move closer to the boundaries of what is generally expected of this thing called evaluation does that help just noting we're getting very close to time so I think maybe we've got time for one more question if anybody's got a burning question just looking actually Emily use this pop something up so Emily's asking Bob if you have a read of it yeah I'm trying I've probably read it too fast is what evidence or data can we collect to address those evaluation questions I think one of the things that I've always learned or one of the things that you learn from thinking about things systemically is you can't collect everything and you have to make quite important decisions about what doubt you do and you don't collect know what I provided within an equity frame are the kinds of areas that might actually get you closer to this but in a sort of a day-to-day level say for instance when I'm designing a survey I literally just say this is the question I need to address this is the data that I need to be able and this is the data I'm intending to collect and how am I going to actually physically use that data in the particular context that I've done and I find those sorts of questions being quite specific I set up yet another matrix right and that I think helps me work out ultimately in the end within the resources I've got available with the people I want to benefit and the kind of knowledge that is acceptable within this particular context which actually are the data areas but I sense that I'm dodging the question maybe we can talk about it offline you want my little wrap up I think that might be might be good thank you so much Bob for your time today and for your thought-provoking presentation and I think it sort of speaks to everybody's interest that most people have hung on you know with meetings on the side of this lunchtime session so thank you once again that's okay this will take about two minutes and then we can all say our advice yep I think I'm often asked at the end of these sorts of presentations is well what do we do next where do we start on this kind of stuff and my advice is this firstly be strategic start with an aspect of the system field that interests you if you're interested in interrelationships if you're interested in diagramming if you're interested in some kind of perspectival stuff or if you're interested in the boundary stuff start with that and play around with it first be careful using systems ideas can actually change your relationship with key colleagues and stakeholders just be aware of that be safe for that reason seek low risk and medium reward first and be useful don't just run off and use systems ideas in any way just use them when you think they would add something valuable to your particular alteration be creative I've given you a bunch of ideas but don't be purist just kind of stick to the make sure you stick to the core principles and above all just have a bit of fun because that's what it can be and if you want to find out more information here's some of the work that I've produced in this particular area but honestly there's just a massive stuff out there and I'm more than happy to help guide people through what's available thanks very much and thanks Bob and we just had a question from someone regarding sharing the presentation are you happy to share your presentation slides absolutely right yeah and it's the whole thing the slides will come with the narrative as well so I'm more than happy to keep it around that and as always there's the after sale service some of these ideas may interest you and obviously underneath each of the things in this very short presentation there's quite a lot of ideas and thoughts and experiences and networks that can help you if you as in one of the slide we were saying can we start with something that kind of don't feel you have to do the whole thing draw some boundaries around it and work from there I regard this whole thing as kind of a stepping stone approach you know make the first step see how useful it was maybe the next step so thanks for your time guys and I'm going around with some of these ideas don't hesitate to contact me if you have any particular questions every question I'd like to send to you thank you thank you everyone for staying and being a great audience