 Welcome everyone. Good afternoon everyone and welcome to EY. It's a really enthusiastic bunch of faces I see before me. So welcome and thank you for your time this afternoon and also to those who have joined us online. It's like mission control up here trying to get everyone organised but hopefully you can hear us all okay online there. My name is Mark Elvin. I'm a partner in the firm and I lead Programme Valuation Services for Oceana. But before we dive into the program I would just like to acknowledge the gathered people on New Zealand we are meeting today and I'd like to pay my respects to their elders past, present and also to stand in that acknowledgement to any Aboriginal and Torres Strait Islander people in the room or on the call today. So as I've said it's great to see you all. I think we had over about 500 registrations and there are a lot of people online I believe for this event and that reflects the importance of this topic and a topic that is topical in terms of approaches to evaluation and measuring outcomes in place-based settings. It's also a testament to our fantastic panel of speakers and I'm really looking forward to their insights this afternoon but also the hard work that's been put into making this event happen by Melissa and Paula. AES Simna and the team that are behind the setup today. And just a very quick note on housekeeping obviously you know where the exits are because you came through them but next to the lifts there are exits, there will be a warden in place in the unlikely event of emergency and you can take directions from the floor warden from there. The bathrooms are past the lifts and to the left, down the corridor. You can find those easily enough. And after the session today, drinks will be served outside. And with that I would like to hand to my colleague, Mel. Apologies. We're navigating many different technical... It's a bit like climbing that rock. So as I was just saying, we were just talking through Simna and AES. So Simna has been established to help organisations and their stakeholders understand social impact through education, knowledge, sharing and networking like today's session. And Paula is my co-host today and Paula is chair of Sydney's Simna Committee. I am a member of the AES New South Wales. We are a member based organisation and we insist to improve theory, practice and use of evaluation for people who are involved in evaluation or interested in evaluation. We have more than a thousand members in all aspects of evaluation and performance measurement and we deliver a range of training to our members and those with an evaluation interest. Once per year, we come together as a Simna and AES collaboration and run these events. They've been happening for a few years now and we're really excited to be having this conversation today and sharing it with all of you. We are focused today on place-based approaches and thinking about place-based approaches, it was quite interesting even as a panel, we had a lot of debate amongst ourselves around what a place-based initiative actually is. We're really looking forward to hearing from some diverse speakers today. I'd like to introduce our speakers. We have Leanne Molaumi who is from Clear Horizons Consulting. So Leanne, yeah. Leanne is director of consulting and a partner at Clear Horizons. Sorry, anyone on the line if I can ask you to stay on mute. She's a consultant who specializes in measurement and evaluation and learning and for over five years, Clear Horizon has supported the development of approaches for evaluating place-based initiatives and have worked with a wide range of organizations and initiatives to design and implement quick-foot-purpose evaluations. Leanne's been supporting government and non-government organizations in the environmental sector to bring place-based evaluative thinking to their delivery teams today. So thank you for joining us, Leanne. We have George Argus, George's head of measurement, evaluation and learning at Paul Ramsey Foundation and he supports the foundation's mission to break cycles of disadvantage. George has worked within the university sector for many years, teaching research and statistics and publishing many articles on evidence-based decision-making. He's also worked for several high-level government evaluation frameworks to inform to those, including the Personally South Wales Evaluation Guidelines which I'm sure many of us in this room have worked with over the years and currently is working with the Australian ACT and sorry, everyone online can I please ask you to stay, Nick? It was in the purple. Sorry. That was not a technique. And it's currently working with the ACT government to build a whole of government evaluation capabilities and we have Rachel Bertram in the room as well. Rachel's a social impact and evaluation specialist who's based at UTS. She's co-founder of the UTS social impact toolbox which is a project that's aimed at enabling access to social impact evaluation, education and resources. She works across sectors building capability of organizations to plan, evaluate and report on the social impact that they were. Rachel's also currently undertaking a PhD and in the last stages of that PhD process with a focus on how one measures success and what defines a thriving community. So we're really looking forward to hearing from experiences from George and Rachel on the panel today too. We also have on the line Anna and at some point I will figure out how to get Anna to be up on the screen. Let me just stop my screen share so that we can show you Anna to the rest of the room. So Anna Okay, people online can see Anna Anna is the CEO of Collaboration for Impact and Anna sadly her flight was cancelled so she was unable to be here today in person but we're really looking forward to hearing from her. Collaboration from Impact is a national nonprofit organization that enables people and organizations to transform systems through collaboration. They work with a range of Australian-wide communities, government and funders and other collaborators to build capability, social infrastructure and collective influence for systems change. Anna is leading the design and establishment of the National Center for Place-Based Collaboration and working with communities, government, philanthropic, human services and other enabling organizations to design the national infrastructure for place-based challenges through place. And I think I've briefly mentioned our co-host for today, so Paula. Paula is a social researcher with experience in evaluation, public policy and social impact. She leads program evaluation services and evidence-based projects within the Department of Regional New South Wales and as I mentioned is also a chair of the Sydney Organizing Committee and myself. I'm Melissa, for those of you who haven't met me before, it's really lovely to meet you all. I'm a director in EY's evaluation practice but also with my other hat on, I'm one of the members of the New South Wales Australian Evaluation Society's Organizing Committee. As I mentioned, we do these events once a year and we're really excited to have this conversation and it's really great to have you all here both in the room and on the line to join us. So as I referenced earlier, it might relieve some you to know that despite the level of familiarity that we our panel has with place-based initiatives there was a lot of debate amongst ourselves over many months as we were planning for this event around what a place-based initiative looks like and what measurement in place-based initiatives looks like. I'd like to open today's session by inviting George to talk a little bit about some working definitions of place-based initiatives to help us to frame today's conversation. Now George, I will click through the slides for you if the clicker's not working, so let me know. Thank you. Thanks for that, Melissa. Yeah, we we want to get past the debates about definition and move on to a more important discussion. So just to have a working definition to start with and this is where I from Gestart and Clear Horizon and the work they've done that has some of the broad characteristics that are consistent. Words like collaboration, words like partnership jump out at you, shared accountability, involvement with communities, some of the elements that you'll see in various different definitions of place-based approaches. Now, in trying to be a bit more concrete with that there's sort of developed two archetypes of what a place-based approach might look like and what it involves. So if you want to jump to the next one. Oh, so again some common characteristics of before I get to those archetypes. I won't read out each of these they're there on the screen and we'll provide these slides to you at the end so you have these notes. Just a couple of things that are worth talking about in relation to the characteristics here. One thing I've learned when doing place-based work is not to conflate the community with the place. The two terms are often used interchangeably but there's a difference. There's things that attach to a geographic region that are separate from the communities that live in it. And secondly, when you talk about place you really talk about the community. It's only very specific places where there's a one-for-one relationship between the two. It's often talked about communities, plural in a single place. And that opens up a lot of issues that flow into evaluation and measurement in terms of relationship between those communities, potential conflict, disagreements different values, traditions, all of that comes into play when you think about places distinct from the communities that live in that place. So I'll just flag that because it's something that might come up in the later discussion for you. Happy if there's any points up there you want me to expand on now. Just noting we can't see online. I think there we go. It's up. Thank you. We're jump happy with that unless there's a point up there that people are going to a bit more depth now. Can I just confirm that everyone can see the slides online? So thank you, George, for setting that scene. Anna, I wonder to add a little bit more there from your perspective given that we did have a lot of debate in the leader based on what George has just said. Do you have other thoughts? Is there other evidence that we should be thinking about when we're thinking about place-based what does that actually look like to you and to the team that you work with? On the spot, Melissa, straight into it. So, and hi everyone, it's wonderful to be here and sorry I couldn't be there in person. Look, not so much difference. Actually, more around often we look at this as spectrums as well. So, and it might come in one of the later slides, but there's some great work that's been done actually by a number of researchers. There's a it's not it's been adapted by the Victorian government in terms of their place-based framework that really looks at a spectrum of working on place through the place-based change. So I think there's some helpful distinctions we can make across that about whether we're being focusing on a service on a particular cohort in place or the way through to a community-led collaborative approach that recognises all parts of the system need to come in behind some of the complex challenges. So I think for the purpose of today we're probably talking more about that end of the spectrum. That's looking at multi-stakeholder collaborations that are really genuinely informed by the context of place and led by place more than that. But just to note there's differences and we need all of it just the evaluation and leadership practices look different across that. Thanks Anna. And Leanne, I wondered if you wanted to share an example of one of the initiatives that you've worked with and as we said at the arts that you've worked with a number of different place-based community-led initiatives just to help to contextualise for all of us in the room who and for those who may not have been involved in a place-based initiative what that looks like in practice. Yeah, sure. Sorry, I actually haven't worked on this one but it was considered a good example just because I think it's people can just generally understand it. I've actually got some points that you could share if you wanted to just now. But it's a place-based social initiative and that's the interesting thing of just having a chat with Brian before is that for some reason most of the place-based initiatives that we're seeing are in the social sectors not in the environmental sectors. Even though the environment the environmental discipline of environmental science is very much around systems thinking and thinking about ecosystem so forth. But this one's in northern Mali region of Victoria. That's okay. That's okay. That's all right. Next one. There you go. So yeah, northern Mali region of Victoria and it's addressing the social issues being experienced in the region and it's around improving the health and well-being outcomes for children. So this region the Mildura LTA experiences high levels of socioeconomic disadvantage. There's high rates of family violence, children child protection orders and in and out of home care. So there's a whole bunch of stuff going on and there's a whole bunch of strengths in that community as well which are being enriched through this approach. But hands up Mali or HUM as it's called works in partnership with those who understand and invest. So obviously community and service providers and agencies to identify where actual investment needs are made or have the most impact and then very strong with vibration evidence and decision making and including obviously local community knowledge in that area about the services intervention. So they've got a large consultation period very very extensive. People were engaged in that consultation over a 12 month period and they've come up with a community aspiration. The vision is a bit hard to read there but connect the community families so I can't quite read that. Mata families matter sorry and children thrive. So this vision it's very much a shared approach which is one of the characteristics that George spoke about earlier on this place approaches has a focus on prevention and early intervention and they have different the way the approach works is it looks at both pro-worts you could call them very very young pro to eight years so best start for my young people and so range of different programs within that broader approach targeting those pro-worts. I'm not myself personally but my colleagues have been involved in designing the theory of change or actually for the journey of change for the for the entire hands up rally and voluntary evaluation and learning framework but also tested journeys of change and male plans for different aspects of the programs some evaluation capability building and support mentoring and coaching to do their own good evaluation. Thank you Leon and Anna I believe that there's an example that you'd like to share as well and I'm just going to flick back on this question please let me know if you can't see it can you and you can hear me okay wonderful actually I'm going to say we've got another 30 seconds I'd love to share the slide before that as well I think someone that has a pink in it that's beautiful yeah thank you just that's you know to what I spoke to earlier there's a number of spectrums out there we find this quite helpful in just locating the conversation we're having so I mean in some ways to give an example which you know I want to talk about a multi site so multi community example of place-based change that really looks at what can be addressed locally within the boundaries and geographic place but also recognises there are other system issues that intersect into place that need to be addressed as well so I'll talk to an example in a moment but just so it might be helpful also places a bit more on the why so why you know why we're seeing such a an uptake in place-based change incredible increased investment in place-based change and really off the back of many years in fact some people would tell you know average of colleagues will tell me this has been happening for many many generations communities leading change and more recently we're really learning into what are those different models and how we can better align resources behind and the reason why that's important is because more and more we're recognising that programmatic approaches just aren't working for the type of complex challenges that we have across communities and across our country and really the world now no one I think is saying that place-based change is the magic bullet but it's a really really important part of the equation when we look at models of change for addressing some of those complex and entrenched challenges and so I think where we're talking here particularly around a place-based change is really as I was saying before about recognising we need all sectors to be collaborating together aligning the resources that we do have behind a shared agenda and where we're seeing more traction more impact on early signs of really systemic change is where those efforts are really galvanising community to the point where community have the agency the governance structures, the practice, the resourcing to leave that change so what we're not talking about down that blue end of place but it's changes, it's not just community development and it's not just communities coming up with a whole list of priorities it's really quite sophisticated models where we're aligning lots of different institutions and actors behind those community priorities and it's often it's data informed and it takes many many years to really establish the practice and conditions around it so with that I might go into the next to the example this might be familiar, I just had a quick look at who's online, definitely familiar to some people stronger places, stronger people is one of our really important demonstration initiatives across the country it's working in communities across the country just checking sounds working okay yeah so it's working across 10 communities that are at different stages from establishment right through the scaling so as Leanne talked about with Hands Up Mally the wonderful folks in Mildura have partnered with stronger places, stronger people for a number of years, many of you will know if Bernie works, Logan together so what's I guess other examples of multi-site place-based initiatives as well including Victorian community revitalization working in seven communities across Victoria looking at really quite significant community-led reform and change in regards to economic outcomes so stronger places, stronger people it is really looking at ways of testing the model for multi-led systems change some core principles in there and every community does this in their own way but what is core and common other principles that are being worked to both by communities but also all levels of government that partner with each community and other funders including philanthropy and then any other collaborators that come on board with those partnerships so those principles include the recognition that when we're working with local individuals at the local state, regional, right through to national level another principle around shared leadership and decision-making so recognizing that these big complex challenges do require the not just one single actor to address them we need shared decision-making behind the community priorities often the work that starts is data and learning around what are the key trying to keep that voice sounds like an interesting conversation so shared leadership and decision-making is another principle particularly centering First Nations leadership and self-determination in the process community mobilization and leadership so more than just engaging people to set the agenda it's ongoing mobilization around the change agenda and having community diverse representation of community at the heart of that again that looks different in different context but it's really critical when we start looking at the how and why of evaluation of place-based change in those contexts as I mentioned before another principle behind a shared agenda for change now when you think about that across multiple sites, communities context and across multiple scales it opens really interesting and challenging questions around how we think about Mel in these multi-site initiatives and really critical particularly if we're looking at large-scale change where we can't just address that by putting all the pressure onto communities to address that type of ambition that's needed so if you genuinely believe that we need all parts of the system to collaborate around it we need to be looking at all parts of the system in terms of our learning and evaluation as well so maybe something we can pick up later I know you've nudged me to the next one so this is an example within there which is the Barclay regional deal big shout out to these folks who've done some beautiful work on crafting their shared agenda which manifests in a theory of change which is the slide before I can probably share it out later this is yep so they've got a theory of change which they've also articulated their house so their way of working talks about the different types of levels of governance the working groups and importantly the principles that hold this work together so in addition to the activities in addition to their priorities and their work streams it's the principles that become almost the DNA for how the Barclay regional deal enacts the change across multiple stakeholders and scales so they've done some really interesting work there in that last slide around a rubric that looks at what those principles look like across different phases of time as well and what good looks like for that so really interesting important piece of work that we're going to talk about in a little bit before we go into the next slide and then we're going to talk about the future of the Barclay region so a few headlines there from the Barclay and the SPSP Merle will need you to unmute the room can you all see a minute yes perfect thanks so I was just throwing to thank you for that Anna and thanks for those great examples of rubrics and other ways in your experience how place based approaches change the dynamics and power dynamics within a program and within monitoring and evaluation Power and Agency is a really interesting thing this morning already because by the definition that we spoke to earlier place based initiatives are a democratization of power in how we discuss community how we engage in what's important and evaluation by practice is a process where we say this thing is important this thing is worthy of value this thing is worth our time our energy and our money our resources our energy if we are volunteers in community maybe not money if you're from the not the pocket or volunteer area of town and so power plays a really interesting point when we look at who has that as well who are we allowing to speak when place approaches are an opportunity for multi-stakeholder inclusion before but the people that know place best to speak to what is going to work for them what is meaningful for them what are the challenges of that area and of that place and as Leanne mentioned earlier you can't extricate the social from the political the economic environment everything within place or breaks in relationship to each other as was so beautifully articulated in those examples we need to understand what is happening in relation to people, place, time and power dynamics in order to understand how change is going to occur and so the process of systems evaluation the process of face-based evaluation is really delving in and examining and to do that you have to have time you have to allow community to discern what's important to them and then you actually have to listen and invest which is kind of the part of the puzzle that historically has not been done very well we're very good at listening but now with the investment in the next centre in this movement towards acknowledging the relevance of local place-based knowledge through things like Indigenous knowledge systems traditionally logical knowledge but also just people on the ground doing the work people that are facing those challenges every day it's a redistribution of that power but the other thing that I will say and it was alluded to as well earlier is there is a spectrum and it's very easy to be swayed towards where funding is talking about this now which is a large scale systems level change which is really important to engage with but if you also want some good examples of where it's done well have a look at your local youth centres have a look at your community engagement resource groups that have been doing this for young place-based issues are just initiatives that are reflective and responsive to the communities that they're in and then we get the kind of going and unpacking what that actually looks like how it works and interrogating at which we're going to do next in this conversation Thank you, this has been a really good introduction and definition thanks Melissa thanks for the panelist so far so this second part of the discussion we're going to focus on evaluation and measuring the social impact of place-based initiatives so evaluating a place-based approach is very complex because of the diverse players different levels of work the long-term interactions that we have to have with individuals the dynamic nature of the activities and outcomes and the attribution aspect as well so with so many moving parts it's very difficult to design on one-size-fits-all evaluation model for place-based approach so my first question now so what we would like to understand a bit more now is what is different or unique about place-based initiatives compared to other programs so I might start there with Anna that is online so first question Thank you now I have a slide for this, is that possible to get it up here we will give it a shot while you keep talking thank you because I understand my job here is to set the scene for what is different and challenging about evaluation in the context of place-based change and I think it's fair to say that many of us have been on this journey for quite a while figuring out how evaluation can best serve place-based change agendas particularly community agendas in that so there's been a whole heap of learning and cycling around and huge progress made as well I think it's fair to say that evaluating is applying their own practice of learning through this and here's a bit why I'm trying to slow down before we get the slide up I'll keep going some of the key characteristics and then I'll hand over to the evaluation experts actually in the room some of the key characteristics of place-based change this is of that particular part of the spectrum again I'm just to not talk about the whole spectrum just that particular part around the collaborative place-based change models so that they're complex and multi-layered and they're long-term ideally long-term interventions so we're talking is this generational change a number of the initiatives that you saw on the map I shared are more than 10 years in that current form so that's versus being programmatic so we're not in the territory of cause and effect logic models we've smashed that out we're talking about this type of place-based change sure there are some projects in there within the context of a place-based initiative there might be some projects that lend themselves more to that cause and effect programmatic evaluation but as a whole we're talking about something that's much more complex and with that is another characteristic you know they're highly emergent and developmental we used to say particularly in the early stages but actually it's the way it is as we all notice the way the world works there are some things that are known and some things that are unknown so what do we think how do we think about evaluation serving a context that's highly mental and emergent you know it's where some of the thinking around developmental evaluation has really had some traction most communities we work with and speak to really value the learning part of Mel much more than the MNBE part for example it's often as I mentioned it's often across scales different scales for different initiatives but we want to be thinking about how we understand progress and learning not just of the work that's happening locally and in community you know wonderful community there's some really mature work with their systems change so I'm looking at how do we understand the role of government as an actor, as a player as a partner in their initiative and how do we start to sell more of that story as well it's also the intersection of multiple systems and we're talking about place it's the place in which we all as citizens experience the intersection of systems and particularly in place-based changes about where those systems are inequitable and not serving everyone so that in itself leads to a different type of evaluation considerations and we're talking about intersecting systems and complex challenges multiple stakeholders and that also relates to what we're wanting to be tracking but also relates to multiple stakeholders will bring multiple expectations and needs of place-based of male as well different stakeholders will value different types of data will require different types of reporting still these are some of the upstream challenges that we're looking to address that start to relieve some of the pressure particularly on the communities and the backbones that are holding this work but we're still in a context where often communities if they do have quite good funding and partnerships around them are spending a whole lot of time managing up and reporting up with different reporting templates and tools different types of data being asked for and sometimes those reports are even necessarily I can say this not necessarily having to informing the work they're doing in community they're purely for reporting purposes I think it's going to be conversations about how do we actually think about data reimagined data differently where it does serve our needs and actually genuinely informs decision making a couple other things so really important principles of data sovereignty there's some great work happening in the place-based change field around indigenous data sovereignty and networks forming around the learning of that and frameworks as well and this might be a good segue to George so more than the some of the parts so we want to be understanding not just the some of the different initiatives working groups and infrastructure pieces we're looking to also understand that connective tissue that comes through place-based change and as I think someone else said before really important consideration in place is we are looking at the rewiring of power relationships it's really tricky to evaluate that and understand the progress of it but different communities will value different types of power shifts as well in that highly contextual at times too so I might just stop there around some of the characteristics maybe the next slide just quick George do you want to have a say on that an idea of more given the title of the presentation about outcomes I think it's really important in place-based initiatives to define some set of outcomes that is more than just an aggregation of outcomes that serve individuals in the place community resilience for example obviously community resilience for a place is better for the individuals within that place but there's aspects to community resilience that don't just sit at the individual level when I did disaster recovery work in Queensland one of the Queensland unions that was affected by cyclone Debbie they learned that if they train the hairdressers and the barbers and the people at the front desk of the post office to listen out when they're doing their work for people who are in need after a disaster who are adversely affected and feed that information back to the relevant groups they can address those pockets of real extreme need after a disaster that traditional approaches don't get to and that's an example of so no individual was more resilient but the place was and so thinking about that idea of the parts that hold in more than the parts and defining outcomes that capture that is really important right thank you thanks George now I just want to talk about what are the approach used to evaluate place based initiatives and I'm just going to start with Rachel Rachel can you discuss some methods you have used or come across in place based initiatives Sure I think you'll be relieved that it's not particularly that novel the challenge of place based approaches is the unpacking the understanding where we're doing the action things where we removing resources from investing and doing all that kind of tetris thing of things a lot of mapping and then when it comes to that point that we have that articulation the processes are pretty simple the students that we teach and the work that we do a lot of it people really with that mapping and that's why we rely on the expertise of groups like the horizon collaboration for impact because that is the value of them that they can give now so when we start to look at the role that we evaluate as play that play the government plays and policy plays that is also a recalibration that we have to start accepting and to go if we are going to be engaging from a grassroots perspective what then becomes our role and it's shifting now more to that listening facilitation and ordination in order to then get it to that point where we have a really clear articulation of what success looks like and then we start bringing in all the practices that we know as evaluators around good research methods ethical conduct and research where there is a bit of a trend now in terms of power engaging is not new as well because it is the approaches that First Nations communities have been doing around the world ever since communities started to evolve which is effective listening that's what it really comes down to is are we actually listening to what communities want are we listening to what is happening at the intersection between the different components mentioned earlier we can have a discrete outcome community resilience you know connected communities is a really big one that we see now what the heck does it actually mean this is why we have frameworks and theories of change that go into all the different nodes and modules that we then build out into head frameworks or to go this is how we're going to understand resilience this is how we're going to know a major community cohesion or we come up with a standardized metric for it we like that as evaluators because it gives us something we can put into the report and give a number to so there's comfort in that where we're starting to see the sector move towards is much more what Anna was speaking to is understanding the relationship that is occurring within those nodes and so there is a lot of really good literature out there that engages with indigenous research methodologies which now is being taken on by Western Academy looking at complexity theory systems thinking looking at complexity analysis and understanding the relationship between those two but if you read the literature in indigenous standpoint theory if you read the literature on relationally responsive evaluation you start to see how we can go understand about understanding the patterns that are occurring there that's where we're starting to come in towards that's where we start to see a lot of qualitative and mixed methods approaches but then also the people that we're speaking to and back to power still really like numbers and so matching to because if I was to go to community and say we're just going to do what you want to do and then we're going to operate in a bubble that doesn't allow them to speak to power holders and allow that redistribution to occur because everyone's a bit on edge and so as a whole and this is why these communicable approaches and these click approaches are really impactful is it's allowing everyone to maybe speak different languages but you're getting the best of all of the world and so the skill that we have to have as evaluators, as community facilitators in managing these processes is how do we bring everybody along in that conversation and enable us to then go we're going to use this really cool statistical analysis method that I'm just going to speak to in a sec or we're going to look at this really cool relational standpoint relational standpoint approach that's what's most appropriate to this key we don't have to speak to government because we've got funding from this other person that's really on board so understanding the politics and the connections is really important in how we design out our approaches and what methods we're using but once we get to that point there are very standard approaches to how we go about and methods how we go about measuring that little thing which we can speak to in at length if you go into research methods but I think George has an example that we're just going to share I was going to ask Leanne about something first before we go to the very funky example that George's going to give us because Leanne is going to talk about a place-based evaluation framework that we have something on the presentation as well Thanks, so George mentioned this when he put the definition up so the place-based evaluation framework that I'm talking about here is not the the place-based evaluation framework it's one I've mentioned several but it was produced in 2018 by Cure Records Founding Trust it's that in collaboration with a bunch of people about 150 people were involved in its development like a good framework it's important to be clear what the purpose is it's about providing guidance to how to evaluate place-based approaches it's about clarifying different types of outcomes that you might expect to see relationships, about dynamics being one of them across different phases of place-based approaches and also really importantly guidance capacity building for those for many place-based issues to their own good mission evaluation learning this framework is free and probably available it's online, so we've got some resource links at the end that we'll share with you Yeah, follow them there Most people will come over to this definition or this adopted definition of evaluation from my friend Patton in 1997 the framework uses the adaption of that definition systematic collection of information about the activities, characteristics and outcomes of place-based approaches so we're definitely doing that characteristics as well as outcomes to make judgments about the approach and proof of effectiveness and important about future activities and going back to some of the features that Anna was mentioning that make place-based approaches unique or different in relation to how you might approach from an evaluation perspective evaluation here means many different forms and many different forms of evaluation being used at different times for different purposes so includes form and evaluation includes some of the impact evaluation for tracking, changing, causality and importantly developmental evaluation for informing the development of the place-based approach so I'm just going to quickly share with you the actual framework in a nutshell it's got a few sections evaluation principles to guide the general approach to evaluation planning steps helping you step through and the design of an evaluation framework well-planned framework for your approach and it recommends face-specific evaluation plans because different phases and requirements there's also a really long time frame for place-based approach there's a little visual in there that helps you sort of conceptualise it if you're a visual person I want people to love that visual a generic theory of change the typical types of levels of change you might expect to see in place-based approaches and also what changes you might see along the way in terms of changes in the system including relationships dynamics, policies, practices mindset shifts and so forth and if anyone's familiar with systems evaluations thinking they recognise the typical conditions for systems change a set of evaluation questions a lot which is not normally recommended but of course these are picking and choosing across four criteria and operating at different levels of types of initiative and a toolkit so along with the framework there's a toolkit to help you actually step through the process of evaluation plan the yeah I think maybe yes I have so yeah I recommend getting online and have a look at the generic theory of change model for place-based approaches has been really well received by a lot of people a lot of people really like the ability for that model to help them frame the thinking of how change happens interestingly just recently doing helping an organisation rewrite an evaluation report they're trying to explain they're initiative and I shared this with them and they got really excited they said we never thought about that exactly how we would have thought about it we can't articulate it like that and now we can and so that's been an interesting learning impactful example as an end of how place-based methods can actually drive an empower change I wanted to field a couple of questions to us from the Zoom I've seen that there's a number of questions coming through and obviously we're really keen to cover some of those so we've got Amber just joining on the edge of the panel here just so that the microphone can pick me up do you want to do the questions now or go to George so we've got a question or a comment and a question I really appreciate the point Anna made about the reporting and data collection burden placed on communities especially when that data is not valuable for the community itself for informing decision-making do any of the speakers have examples of using external data sets to measure outcomes or impact while reducing the reporting burden good question I think there's always you're going to use a model and for example there's official data on a whole range of things problem gambling transport use the cynicism health outcomes that you can always draw on to reduce the burden on the community so yeah when we talk about community-led engagement with all of that that doesn't mean that everything has to be from the resources to reduce the burden on them just as a side point I'm actually wary of the use of the term community-led I was pulled up short when we were doing the cyclone duty evaluation in Queensland where we said we don't want to leave, we're not ready to leave that's why you're the government you're here to do that for us I think that idea of a staggered approach and engagement building of all partners so that they can be involved in the way they want to be involved rather than talking about it another question is how do you support First Nations data sovereignty aspects in place-based approaches and what resources do you suggest for First Nations people there's some brilliant work coming out at the moment and a lot of reviews happening within that aspect of it the first area that I encourage you to look at is the Indigenous Evaluation Framework that was launched, well I've lost a year of my life so two years ago I think now where that was the starting point it was the start of a conversation and in that I'd also recommend you go and have a look at some of the submissions that were made because in the submissions that are made towards any problem with your inquiry you get a lot of insight into what's actually happening around the community but there's when we're looking at data sovereignty it's to listen to the people that it affects it's kind of going to be there there are resources around data collection and data management that from your cabinet are looking at at the moment so there will be a report coming out right here on that to keep an eye out but also have a look at Indigenous research methodologies and how they engage in this space as well because there's a lot that Evaluation can learn from engaging with the community that research has engaged in this space really such a sensitive area as data sovereignty data access data reclamation what was the first part of that question the second part there was two parts the second part is about resources and the first part was about data sovereignty yeah so one of the best books that I have read around how you can engage in different research methods as well as a book by Sean Wilson called Researcher Ceremony cannot recommend it behind me I'm interested in looking at systems thinking and new paradigms of how we engage in Evaluation but also just thinking Tarzan Yonkopo's book Centaur is a brilliant read it's also just a really fun read so there's some really wonderful work that's out there if you're interested in the intersection between science and ecology so I'm sure this will be right up the aisle I really am but also Indigenous wisdom and Indigenous spirituality Tarzan Yonkopo is another beautiful book and so there's a lot that we can learn from resources outside of the normal evaluation we can engage with because there's a very dominant narrative in that space and the areas that community leaders are speaking into they're using the access points that they have so for a lot of Indigenous people the Academy may not be welcoming the Government may not be welcoming to them popular discourse is and so starting to do a lot of reading out there go to panels like this listen to people when they're speaking we can learn a lot about what resources are out there but those would be kind of three starting points but also like the framework for Indigenous research and all that other stuff that's already on the front on the toolbox we do have a short list of guidelines relevant to what community demographic you're engaging so I encourage you to go and have a look at the short list of resources there and if you find any do share them with us they're proliferating quite a lot I know the AES also just launched their cultural safety so have a look thank you there are more questions there is so this is kind of combining two here how do you establish trust and transparency with the communities before the evaluation and how do you ensure diversity and equity in place based evaluation I wonder if we want to throw to Anna first Anna did you want to comment on that one? Happy to it's actually a couple other threads that I can kind of bring into this one too first of all a shout out to COA collaboration First Nations owned and led evaluation group not for profit highly recommend looking at some of their resources too and the practices that they use I think it relates a bit to George's point about community which I fully agree with in that I think sometimes our language is a little clunky we're still finding it as a field aren't we community can sound a bit like community led rather than the collaborative change that we actually are communities are calling for every group every actor takes their part and shifts their part of the system more equitable systems and more equitable communities in particular so I think that we need to understand the progress we're making towards more equitable communities we need to be looking at different parts of the system so there's some learning happening in stronger places, stronger people as an example the other one I talked about around the multi-site initiative in Victoria we're starting to understand the different shifts that are happening at institutions including government around that the other part I guess it goes to process that comes to mind when you think about equity communities that have really had time to lean into building relationships across different communities George's point about the plurality of communities particularly where they spend time on finding ways to share power and in some cases really genuinely Aboriginal First Nations leadership in that change of gender which looks like really interesting different models of governance that can do that and often by addressing the priorities of Aboriginal community calling out it's for whole of community as well and folks in Milgeria will talk about that quite a lot so it starts the equity question about diverse voices in the process of evaluation understanding the outcomes they're contributing to really comes off the back of the work that the backbone or the collaboration's been doing for many years to build those relationships across the community and across other enabling partners and collaborators as well I'll just share one quick example of that really practical the folks up in North West Tasmania worth having a look at they did some really wonderful work over the last many years on annual learning circles so they would talk to different collaborators governments from all levels service providers people in the working groups job active providers citizens around them and ask what's different around here we've been doing that for a number of years and some of those principles I shared earlier the result of that the muscle of learning over a number of years and the trust that's been built through doing that in the last year when they asked again they actually started with community and really diverse members of the community and as a community they said these are the types of changes we want to see here locally this is what we need from the decision makers outside community and this is the kind of things we want you to commit to to partner with us as a community and so it was the threading of that then into a learning circle with other collaborators around I was in the room with government funders people who would normally be experienced as traditional power holders hearing what the community was saying really sophisticated thoughts around data sharing what good governance can look like what good shared power looks like in the Bernie context and we had at the end of that day a number of really informed discussions around what those other collaborators can commit to with the community so that in itself might not be outcomes measurement but it is really critical strategic learning that does create some of the outcomes that they're looking for in community and in the wider system as well so I think some great examples around process that goes to some of the equity but my hunch is that Leanne will have some thoughts on this too if I can put you on the screen Thanks and I've been thinking about the different aspects of that question trust and transparency the power dynamic stuff and thinking about how and there's a section actually before this framework example I shared before jumps into the actual nuts and bolts of the framework there's a whole chapter on involving community in the planning for the design of, for example the theory of change and the measurement plan evaluation plan learning plan and so I was thinking about something that's a really practical tool and you showed an example of this before around the principles and what foot looks like in terms of levels of maturity the soil, the seedling and the suckling and then the tree and ground zero and so rubrics just in general are highly relevant and useful in this context but also relevant more broadly but the involvement of the people who matter really get to have a say and whose values get serviced in the process of co-designing those rubrics in terms of describing what we expect to see at certain points on the journey and what foot looks like from a community perspective and so that's a very really transparent way of describing how something is going to be judged or assessed and so that brings that diversity of voices in and is a super transparent way up front the changes we expect to see along the way that's a really practical tool and it's it was fantastic to see that example as well and so thank you I know George, you used some really interesting methodologies from a statistical point of view in these cases and we'd love to hear a little bit about some of those Yeah, it's often thought that because of the intrinsic characteristics of the place it's so unusual that it's hard to make comparison, you're always obliged to use non-experimental methods to understand your impact in the future outcome and of course they're always going to be relevant but we've been using some methodology that controls when going to the technical details of it but basically if it's agreed that certain quantitative outcomes are desirable say properly gambling and you're working in a place with a community to do something about properly gambling it is possible to use this methodology to construct it's not a real place that's why it's called synthetic a mythical place that you can compare your real place to it's done by using some weighted averages of other places that are similar to you could create a reconciling point of view as a counterfactual so it's not a real place you're comparing it to but through certain statistical procedures it has some validity as a comparison I mean the archetypal example of it developed by the guy who used it is quintessentially a place of actually the illustration is what was the impact of German unification on the form of West Germany's economic growth path definitely a place-based approach definitely complex and everything that goes with it but used it to show actually what's the obvious which is its growth path was lower than it would have been against it but as a method we're at the foundation we're using it a lot more to look at and make some assessment not of attribution I don't think you could ever really attribute not to decompose what aspects of what you did with work but at an aggregate macro level some sense that you have been able to shift the dial on some agree or to take conditions I wouldn't dismiss quite the experimental approaches especially that one when you're doing place-based work but always alongside the non-experimental methods or contribution analysis that clear rise indeed in longer than the other methods In the room I'm really interested in questions I'm sure that many of you have been thinking as we've been talking about your own projects or your own work that you are delivering evaluations, supports for any questions from the room that you'd like for us to delve into a little bit more on that from our AES colleagues I've been involved in place-based issues that I'm wondering how people deal with them first is the question of social and it's a government it's particularly a government skill in that when it starts to get complex government property and they use the general conversation not to involve themselves in initiatives the second issue is innovation with overly complex community initiatives, innovation is a very difficult issue in relation to that and the third one and they're probably all interrelated is a long-term commitment for example and I'm sure Dan and I were talking about this before the event I think our involvement in natural resources one time there were three name changes of our government department in one year and eventually got to the department of natural resources which everyone referred to as do not process so that probably leaks in with social so I'm wondering whether the panel with people with the experience how do you deal with that I might jump in quickly I think that's where the not-for-profit sector foundation and I'm not speaking on behalf of them five years just to make that clear but where foundations are restricted in that way by government we're not siloing we're not at the department of trends we have the flexibility to work across areas and organised partnerships and being there for the long term so I think expecting it to be done by government probably not the greatest starting point and where foundations and other not-for-profit can probably I think you said something really interesting around long-term commitment challenge and one of the challenges with long-term commitment is you need a long-term funding to be able to do that and that's where the dynamic is changing because when you see a lot of collective impact systems based change or these place-based approaches require actors on the ground to be taking that initiative before that place to take ownership and lead it that still requires them to have resources and so we're expecting by 10-20 year impact to occur for them to take that on board and run with it and really implement it to have those long-term systems based changes we actually are funding it and that's where people start to really go back and go well maybe not let's see how it goes and so it's it's a real paradigm shift in thinking long-term because we don't exist in cycles that operate long-term so that's why commitment is a huge barrier to people because we're expecting people on 6 months, 12 months on these cycles to commit to things that take them 5-10 years and so we see highly responsive to where the funding is we see funding chasing happening but the other challenge is now we go right back to the power conversation at the beginning is when now every single funding round is focused on place-based approaches the pie is very very small and so again we have to see how we can bring on board our local grassroots volunteer organisations that carry a lot of the impact burden on the community anyway and so when you look at these case studies that we shared today a lot of the work is bringing out voices of the volunteers on the ground they've gone out very quickly when we look at social change as well so it is the resourcing conversation is one that nobody particularly wants to have because it's sticky and it requires a lot of engagement but to have long-term commitment you've got to have long-term investment and you've got to empower people on the round to do the work and in many instances it's getting in the resources and stepping back because then we end up in a situation where we have that oh we gave you all this money but you also have our data that we can put into our investment so it's a deeper conversation against stakeholder management and funding management as well as probably chat later in another panel discussion but to enable long-term commitment we have to be actually enabling organisations through resourcing and that's good for actual hands up thank you just because I want to riff off that I think that's right we consider what kind of methods we use at different stages goes to what we need to expect at different stages as well so if we assume these are long-term initiatives then I think and I know a little bit more of Clear Horizons framework and they do talk about some methods in there that are really suitable for some as early stages Jess Dart talks about them like popcorn wait for the popcorn to emerge so I think Leanne there's something there around the outcomes harvesting I also want to do a shout out to some of the other foundations including PRF Dusseldorf Foundation who are leaning into other more I guess innovative practices because they can for evaluation so there's a story gathering project that Dusseldorf did again with Hands Up Mali and others to start to say well how do we really get to the heart of the type of learning and data that's happening recognising often the limitation of quantitative data although we're all pragmatic and understand that we need to be able to work with different actors and different types of data that they need so Leanne just wanted to see if there's anything around those early stages methods that come there thanks Anna I'm just thinking about there's been so much of the work that's been done in this space is happening through the foundation to commit to these longer term make these longer term commitments not worry about election cycles and so forth it's interesting because we've got a couple of teams in Clear Horizon one mainly works with government and that's the environment team the social impact team mainly works not with government just because that's a nature where a lot of funding is coming from and a lot of the exciting work that's being done by the foundation in my work I've noticed that so many organisations when you start talking to them about what they're trying to do and try to help them run a framework like this around it have no idea that they need sort of every 19 years for policy some shifts in the dark in the things they're trying to achieve so again thanks for referencing that Anna I think thinking about the methods that are appropriate for the different levels of change which can be expected over different timeframes is really appropriate another question in the room I'm a community worker with I understand there is a thing called situational analysis it's a very specific thing but it's part of that understanding the situation and the context it might look like Pat talks about in his book about mental evaluation and Richard Rogers has got some sources on better evaluation so doing that situational analysis is the way to try and address that issue so in terms of actually an ownership of the community starting to get engaged yeah I think one of the important things to acknowledge when we've got place-based approaches is that it doesn't discount what's already happening and it doesn't make the assumption that change isn't already happening in the community it's just how we're capturing it and enabling it to scale the other thing that I want to caveat is we don't want to have the assumption that scale is a good thing it's not everything scale not everything should be scale and one of the challenges particularly where you're a community worker is you're in the field I was working with an agency a couple of years ago kind of before this language that was starting to percolate and they were giving a conference presentation they were putting a picture for a conference presentation and I was kind of coaching them on the language the logical model of this or whatever what the heck is that and it's very easy for me as someone and the evaluators in the room they're engaging in space it's very easy for us to see it in that way that doesn't mean that they're not engaging with that approach it just means they don't have the language set to speak to the conference people that are the newspapers if you don't have that language you don't get visibility at the conference you then don't get to power holders to their mobilized funding the first step whenever I'm working with organizations is let's kind of get skill set in speaking this to be able to speak to what you're doing currently other than reinventing the wheel and trying to get you to do some new innovative thing it's how do we actually articulate what we're doing and so starting to engage in that conversation and George alluded to before these frameworks and much more invitational than other frameworks and so place-based approaches are invitational but they can also really be exclusionary because who gets invited to the table are the people who can speak the language and so the work we're doing at ETS and particularly with the social impact toolbox a lot of it is language because when you would see this what we're going to community as well you go okay we tell us about your theory of challenging people what the heck is that and then you say well tell me what you do and like oh well you know we set up this playgroup and then we realized that a lot of grandparents were coming because they had burden of taking care of the kids during the day and they were getting exhausted they were isolated because the parents were having to work too much because their mortgages were going up and you start to see this whole thing happening and they can speak to that but if you say can map out your theory of change here so post-it notes totally disengaged so part of that and this is where I spoke to earlier what is our actual role as evaluators now one of the models that I really champion is the relationally responsive standpoint model which is flipping the script on how we engage participation it's flipping the script on how we approach community how we engage in communication from an initial standpoint of respect and listening and only once you've gone through that process can you get to taking action so it's bringing football along through so I advise go and have a read from that but one of the key barriers for organizations particularly volunteer and small-scale nonprofits that they face is getting a seat at the table and getting a near of the funders because they don't speak the language but oftentimes they have their big if just why we invite them along to the table to speak at our place-based think tanks it's getting everybody speaking the same language and speaking of speaking I'm conscious that we are over time that we could speak all night on this topic I'm sure for those of you in the room we do have some networking refreshments and drinks and campaigns just outside so we will move out there to have those in a second but please join me in thanking all of the panelists for such an inspiring talk