 Come everybody it is absolutely lovely to have you all here and look I'll start by acknowledging that all the Australian presenters and attendees today are meeting on Aboriginal land and I'm on the lands of the Wurundjeri people of the Kulin Nation and I acknowledge the elders past, present and emerging. We're really thrilled to have such huge interest in today's event and for the opening session of the festival and I'll hand over to John Stoney, the President of the Australian Evaluation Society to officially open the event. John, you will need to unmute. There you are. It wouldn't be a conference unless somebody's done something like that. Well everybody, welcome to festival. Also, I would also like to acknowledge the traditional owners and custodians of the lands where we're all beaming in from today. For myself, coming in from Canberra, I'm on Nunnel Country and I would like to pay respects to elders past, present and emerging. It's great to have an opportunity for us all get together and what continues to be in many parts of Australia and even overseas are challenging times for many. I've just got some numbers and billed before we opened up. We have 991 unique registrations so that's going to cover 23 sessions with a combined attendance of about 6,800 for this week which I think is phenomenal to say the least. And I think that says something about the week's program which I think is an interesting and an exciting one. Over the next five days we're going to have the opportunity to hear from a line-up of speakers who will cover a wide range of critical and engaging topics in evaluation. So we'll be looking at things such as evaluation capability and capacity building, cultural safety, public sector evaluation and design, self-assessment of competencies, multicultural evaluation and ethics. In addition to those sessions, there's also going to be an opportunity to inform we catch up most evenings through the Festival Club. So I'd like to acknowledge and thank the team that has organized Festival and also our speakers and presenters who are going to be here during the week. During this week, the AES is also going to be holding its awards for excellence and evaluation and that'll be at 4 o'clock on Wednesday and that'll be immediately followed at 5 p.m. by our annual general meeting. We're going to be announcing the incoming AES forward and we'll also be announcing our new AES Fellows. And at our closing session on Friday, we'll also be announcing the location and the theme for AES 22. So yes, we will be launching the return of our international conference and a return to being able to get together face to face. So I'm conscious that everyone here is pretty keen to get into things. As am I, I'm keen to hear our three guest speakers. So I'm happy to declare Festival open. Enjoy the week and back to you and on all. Fantastic. Thank you so much, John. It's great to have you to launch the event. So today we have an incredible lineup of speakers to kickstart the Festival with some provocative thoughts about the state of evaluation. I'll introduce our three speakers and then I'll hand over. So our first speaker is Sky Trudgett and she's a proud mother and First Nations woman who's currently completing a PhD in Indigenous data sovereignty and models of care for high risk young people. At the University of New South Wales. Sky is also the CEO of Cower and Black Impact Lead at the National Centre of Indigenous Excellence in Redfern. Sky is an experienced social researcher and brings a unique approach to evaluation which considers sustainable First Nations governance and community. So Sky will be followed by Margaret Crawford, who is the Auditor General of New South Wales. We feel very lucky to have found this time. Margaret's very busy day. Margaret has many years experience as a senior executive across large and complex public sector organisations, including the Victorian Department of Human Services, the Australian Taxation Office, the New South Wales Road and Traffic Authority and Australia's largest local government, Brisbane City Council. And before becoming Auditor General, Margaret held the position of Deputy Secretary at the New South Wales Department of Family and Community Services. And finally, we'll hear from our international speaker, Mary Abdo. So Mary is the Managing Director of the Centre for Evidence and Implementation in Singapore. She has lived in Asia since 2012, supporting organisations to develop strategies and capture innovations that generate impact across more than 25 countries. In a role with CEI, Mary supports organisations to maximise the impact of the programmes they fund, including through evaluation, evidence synthesis and effective implementation. So we know that evaluators are naturally very curious. So please post your questions during using the chat function throughout the presentations. And once we've heard from our three speakers, there'll be time to pose some questions and I'll monitor the chat throughout. So without further ado, I'll hand over to Sky to get us started. Sky, take it away. Thank you so much, Eleanor. And thanks, John, also for opening today. I also wanted to extend my acknowledgement to all traditional owners from the lands on which we're all dialing in from and to our elders, past, present and emerging. I also wanted to extend my acknowledgement to all First Nations people who are here today and who are participating across the week. I particularly wanted to pay respects to AES's Indigenous, cultural and diversity committee who have really been leading the way in terms of changing practice and use of evaluation and making sure that's a safe place for all First Nations people. So thank you very much for all of the efforts that have gone into that important work and appreciate the fact that you've paved the way so far. So, my name is Sky. You've got my intro already from Eleanor, so I'm not going to go too much further than that. But I'm a Camilla Rory woman and I'm coming in from a warbicle country, so pay my respects to those people as well. I was asked to come and present today and I was really excited and then I was a little bit perplexed. How do I do this justice? How do I say the right thing? Because all of this came with the notion of can you please provide something provocative? And I thought, yep, I reckon I can. And so when I got to thinking, I started to try and understand or look back on what was the things that kind of came up all the time in the evaluation practice that we've been doing so far. And what I found was that there was this continual need as First Nations people to do evaluation in what might be termed the right way or how we might see it which could be the right way. And so I started to think about, well, how do I talk about today? How do I bring this forward and to hold my truth? So I sat down for a while and I sent something off to Eleanor and to Greg and then I wondered as I sent that off, how provocative is this really? So the answer is probably a lot more than what you thought and it was definitely more than what I thought as well. So after the release of the program, I guess it isn't surprising that it took less than a day for the chatter to start kind of filtering back to me. And with some people who I heard were pretty interested in what might be being said, there was some thankfulness about the rawness of the title of this introduction. And then there was a number of people who are here were straight up outraged. And that's the nature of provocative introductions, I guess. But it did make me think, you know, why would that be the case? And so I had a lot of people think, who is this about? Who are you getting at? And it's really not as juicy as I think some people might have hoped, but it's still honest and truthful. And the truth is, it is not about one single person. It is not about one single organization. The context of the introduction today is actually about a systemic issue. And it's the fact that we often hear or I personally have often heard in our practice that we would like to do evaluation where we centre self determination where we have this real desire for applying First Nations ways of knowing, doing and being. And that we're really progressive in the face of resistance, sorry. And so, but all of that intent and goodwill seems to be met with this misalignment of values and systems as it kind of gets into the work. And so there's this kind of degree but which I think contributes to the experience of why these things show up. And so often what I kind of have seen is that there's some subsequent behaviours that come along with that. And that seems to look like an attempt to then steer what is our First Nations practice towards something that is a bit more in line with the methodological kind of concepts of the dominant culture. So within this liminal space between our First Nations practice and what it ends up being, this is where I really do see racism raise its face. And so this is the part that I would like to address. This is why the conversation today. And so I kept wondering then, all of that. Why would there be outrage? Why is it so outrageous that an Aboriginal woman would stand up today and say, my path here so far has been really hard, and it's been really lonely. And that is not good enough. Where are all the other Aboriginal people? We need to have a place here. And then why is it so outrageous that I would suggest that patriarchal colonialism continues to show its face within our First Nations evaluation practice still today? It's not good enough. And why is it so outrageous that I would suggest that some commissioning organisations just don't get it. And it's also not good enough. So that being the case, I ask those who felt outraged to sit and think about it for a bit. Sit with a feeling it is your conscience saying that there's some work to do to those people who are interested in what this might be about. I would ask you to hold it because interest is a core ingredient of change. And for those people who really enjoyed the rawness of the title or the introduction here, I encourage you to keep sitting and resisting conformity and ask and inspire others to do the same. Because what we need in First Nations evaluation practice is a revolution. And that revolution has begun. So many of you would know about the Cultural and Diversity Committee at AES and the revolution has started there. And we thank you yet again. Last week we heard from Community First Development who reminded us that it is community who determine outcomes. It is community that tell us what success looks like. And that is a place that we need to listen and be most. This week we have a whole week that is spread in learning the opportunities. Sorry, it's that we have a whole week where we have the opportunity to learn from revolutionaries, from non-conformists and from change makers about what evaluation practice really could look like. So in case you don't know some of the events that are coming up already this week, I encourage you to check out obviously the launch of the Cultural Safety Framework. The building Aboriginal evidence base, our place on country and of course reimagining the multicultural evaluation practice later in the week. Sorry, this is the week. It's the first time I've ever taken notes. I took notes because I actually really wanted to make sure that I got everything. So forgive me if it's a little bit rusty. And the other thing I wanted to share with you, apart from the opportunities this week to hear and learn a little bit more, I wanted to kind of talk to you about what are the instances that we're seeing great evaluation work turn up in our space. And this is not to say that it's the only work that's happening, but it is to say that there's some really inspiring stuff that we should tune into. So in sharing that with you, there's a couple of organizations and projects that are really looking to shift mindsets and amplify the voice of First Nations people in sovereignty and evaluation practice. The first one, getting my notes clear, is applying a flexible approach to Mel for First Nations organizations. This one comes through a trust that we have worked with in the past. And what they're working to do is center both ways of valuation and evaluation and Mel strategies that work for First Nations peoples and organizations, those communities that they're representing and a trust who is funding part of that work. It has been really interesting. You can read a little bit more about that on our NTI website and blog from Black Impact. There's also been some, sorry. There's also been some incredibly inspiring work that's coming out of our Department of Communities and Justice with the Aboriginal Knowledge Program. And then their partnership now with Absec to support the co-design of Indigenous status sovereignty and Indigenous status governance across the outer home care and child protection continuum. The work is happening at the moment. And there's an invitation for all of community to participate in that going forward and you'll start to see that come out in comms later in this month or towards next month. And then we've also been working with a group. Seed Mob, who I'm sure many of you actually know about already and Seed Mob are applying First Nations ways of knowing, doing and being towards their impact framework. And so starting to share with others exactly what it is that they're doing, what success looks like in more of a First Nations kind of concept and way. And I would encourage all of you to look up Seed Mob and understand a little bit more about who they are and what they do. We've been thankful enough to be working with another organisation. I'm not sure if I can share who they are just yet, but another First Nations organisation who are looking to support allies who are funding efforts across the First Nations space to channel their investment in a way that makes good sense and has huge impact across multiple places for First Nations people and communities. And their work has been incredible in looking at how does a dollar investment come in and how does it look different as it goes across First Nations organisations how do we go dollars into values and so that's been a really interesting project that's still emerging now. And then of course there is the grassroots organisation Maranukka who I'm sure most people know about who are trying to weave all these bits and pieces together so they've established their First Nations data governance to enact First Nations data sovereignty principles. They have a platform that holds all of their data and they use this approach to do rapid evaluation of the initiatives that are happening on the ground, as well as holding their whole with funders and government, and then allowing themselves to really focus on what we're calling truth indicators. So this is what are the thoughts feelings just experiences of community on the ground around whether changes happening or not. And finally, there's the effort that are happening in community that I wanted to share with you. And then there's the efforts that are happening in bringing more community into this space and so I would like to just also highlight the array project that clear horizon and ABC Foundation are working on in strengthening the capacity of First Nations women to participate in evaluation and then to know the rules but then to break the rules and feel really confident in doing evaluation practice their way. So I'm really excited that these ladies would be stepping into this space in the future. They're amazing amazing women so please check out what they're doing and get behind that project. So, all of that is my provocative start to the week I'm not sure how provocative it really is. But I would like to encourage everyone to to dig a little bit more into the spaces that I've might have just quickly shed at the moment. I hope you have a wonderful week and enjoy festival. Thank you so much. Thanks, Eleanor. Thank you so much sky what a great start to the week and a great shout out for lots of the important sessions we've got coming up in front of us as well. Margaret I'll pass to you now to talk a bit more about about performance auditing its relationship to evaluation. Okay and thank you very much Eleanor and thanks for the invitation I'm really pleased to be here today and well sky. I don't know about provocative but you certainly provoked me to do better and to learn and to learn much more. So, but can I begin by also acknowledging the traditional owners of the land that I'm speaking to you from today I'm on the land of the Gadigal people of the urination, and I pay my respects to elders past and present. And I'd also just like to acknowledge any first nation person that's joining us online today. Okay, so look I think it's pretty fair to say that we're living in unprecedented times. And in that context I hope everyone is well. But look here in New South Wales, our government is has really been in emergency mode or emergency response mode for over 18 months. They've had to deal with raging bushfires right across all parts of our state, then floods and then and still the COVID-19 pandemic. They're having to act really fast. And there really isn't a script for the for any of this. So in this context, the assurance provided by audits and by other forms of review and scrutiny is arguably more important than ever. So in the very short time I have this afternoon I wanted to tell you a little bit about the role of an auditor general and my performance audits and how these work with the broader system of review and evaluation that you're all part of. So look my starting premise is that at a time when government is is impacting so much of our lives. We've all got an interest in it succeeding. We want to know that governments are doing the right things and in the right way and learning from mistakes. So just stepping back for a moment there's actually been an order to general in New South Wales for nearly 200 years. I just hastened to add at that point that I've only been here five years. So not nearly all of that time. And it's also fair to say that performance auditing is relatively relatively new discipline in that sort of the context of those 200 years. So in New South Wales performance audits have been part of our mandate since around the 1990s. So in essence performance audits go beyond the financial audit assessment of an agency's financial management and reporting to assess how well public money is used. They're looking at the effectiveness and the efficiency of governments programs and activities. So look the the purpose of this work is really two fold. We want to identify performance gaps. So the government can can improve its practices, but also the goal is to support the parliament to hold to hold government to account for delivering what it says it will deliver. The role is incredibly privileged. It benefits from a number of protections that I don't think are afforded to all evaluators. So the key is its independence. So the order to general has a fixed a term and can't be reappointed. And I can't really be sad unless I go completely mad. And most commentators will will always say that orders general always a little bit mad. So nor can I be directed regarding what I choose to audit or to the scope of that work. My office has unprecedented access to information. The legislation actually compels the agencies that we ordered to respond to my requests. And if they don't, there's also an ancient ancient provision that says I can recommend that those individuals pay be stopped. I haven't resorted to that yet, but I'm pretty tempted at times. So look, we, we don't have to bid or compete for work. The government funds my office to conduct performance audits and agencies can't refuse from a choose them. But most importantly, we report our findings to the parliament rather than to the agency we ordered and our reports are always made public. Okay, so given all those privileges, it's it's also natural that we have certain obligations as well. We must comply with auditing, auditing standards and meet professional obligations to be of the highest integrity. And every segment we make must be backed by evidence. And we must allow the agency agency head to respond to the findings we make. And this response is also published with the report. All this means that our work respected and trusted. It is truly independent objective and transparent. We're really beholden to no one other than the New South Wales Parliament. So, so yeah, so it's a pretty privileged position to be to be able to conduct performance audits with all of those privileges. But there are limits. And this is where you come in with the broader where the broader system of research review and evaluation is just so important. For example, performance audits cannot comment on the merits of government policy, only on how well it is implemented. You can. We can only audit government agencies, not the non-government sectors that are commissioned to do much of the work of government. You can. Secrecy provisions mean we cannot speak publicly about matters we have found beyond what we have published in our report. Again, you can. To be truly independent, we cannot provide advice or guidance to try to help agencies to do better. You can. We cannot partner or collaborate with the agencies we audit or with other stakeholders, but you can. And finally, we can only undertake a limited number of performance audits in any given year. And usually that's around 17 or so. And this is clearly not enough when the need to evaluate the actions of government to help improve outcomes for citizens is just so important. So I guess my point is we really need to maximize the diversity of the tools techniques capabilities in this broad system of review and evaluation. I think that's in everyone's interests. So I'm going to pause there and thank you all for listening. And I will be happy to join my fellow presenters to answer any questions that you have following our next speaker. Thank you. Fantastic. Thanks so much, Margaret. And what a great series of ideas and thoughts about where evaluators can play. And particularly some of those extra benefits from the freedoms that are afforded to evaluators. And now hand over to Mary Abdo, Mary's our international presenter zooming in from Singapore. Take it away, Mary. Thank you. Let me just get my slides up one second. There's ever a little bit of trickiness with that. Okay, just checking you can all see my slides give a good nod if you can. Yes. Okay, perfect. All right. So my name is Mary Abdo I am managing director at Center for evidence and implementation in Singapore. I'm an ex consultant hence the slides so can't ever show up to anything without good slide deck. I am going to be covering off evidence and evaluation in Asia massive topic. I will try to provoke some thinking but I will in no way do justice to this enormous topic. So I will try to just peak your interest and we'll see how that goes. So I think some of you will be familiar with the work of CI I'll just give a quick introduction Center for evidence and implementation is Australian headquartered we have offices around the world. We're in London, as well as in Melbourne, Sydney, CI is a mission driven, not for profit evidence intermediary and advisory organization. And our mission is to help get the best evidence into policy and practice to improve the lives of vulnerable people. And we of course as a part of that have a strong focus on evaluation. I'm also always hiring. So please subscribe to our newsletter so that you may learn when you can come and work with us because we want great evaluators to come and work with us. So, I'm going to focus in on three big ideas today. And I should preface this by saying I mentioned I have a background as a consultant. I'm not an evaluator by training I am a new arrival to evaluation land. And as a result some of what you'll see is a bit macro is a bit looking at some of the trends in the region. And letting you know particularly from a point of view philanthropy how I think things are playing out here. So, first, there is an urgent need to identify and scale what works. That philanthropy has an important role to play in addressing the needs that we see and that we as an evaluation community need to start at square one to support this sector. So let's dive in. As you can see a weeks long seminar would probably not be enough to cover any one of these topics so we'll try and do that in the next 10 minutes. So what I want to start by saying is the problems aren't going away. So we all know that we work on social issues every day but I just want to zoom in on one in particular I my background is predominantly in looking at education, mostly in emerging markets. And the chart that you see is looking so we'll just zero in on this particular issue it can help us see what some of the scale of issues are in Asia and more broadly. But this chart is just showing us the out of school population by region and the way that it's changed over a 10 year period between 2007 and 2017. So actually we we start in 2000 and then you can see that we've got a year by year from 2007 and 2017. It's a time that the Millennium Development Goals were created and in the years just afterwards so between about 2000 and 2011, we saw this really precipitous drop in the number about a school children so the whole world is focused on this issue see a massive push in building particularly in places like India, and parts of Southern Africa, and, you know, really attacking this access challenge. And that happens for about 10 years, and in the years between 2011 and 2017, those numbers barely shift at all so in fact, even if we say 2018 and 2019 that number would have gone from say 260 million children out of school to say 259 million children out of school so that implies that at the current rates we have to wait 260 years before we get all out of school children into school, not something that we want to see happen and what what's really going on is that we've got the quote unquote low hanging fruit of the children who are going to go into school and right now what the issue is that we've got a lot of children who are out of school for more complex reasons so for economic needs because because of issues around girls education, more likely to be disabled children rural children conflict affected to have all these kids who are still out of school, and historically Asia has had a really large share of these out of school children. And so this is a massive issue then add the pandemic, just what we all needed. This has made all of this vastly worse so there was an estimate going around that about between about $50 to $60 million excuse me billion dollars would be needed to fill the hole in national education funding. Prior to coven 19 that estimate is now up to $200 billion per annum and new funding that is not currently in the global education system to get these kids in school. There is an estimated 35 million children who dropped out of school in a pack in the last 18 months, and those children, especially from low income families are much less likely to return to school the longer they are away. And this will have an effect on things like child labor child marriage child trafficking exploitation. And of course, hampering the life chances of many millions of children. So why am I bringing this up it's to say that the scale of the challenges that we're facing have really never been greater and I think. That's a backdrop for talking about the rest of this conversation is that we really need urgently to think about what works to solve these issues. Now, enter stage right we have a really exciting thing going on in Asia against this backdrop which is also that we see a massive increase in regional wealth so Asia already has the largest proportion of the world's billionaires. And really the world is the region is projected to be home to a third of them by 2023 80% of the fastest growing wealthy populations in the world reside in Asian nations. And this is a population that has really, I mean, massively diverse population as it's inappropriate to call it a population but let's just say regionally, there are very strong philanthropic traditions often tied to religious traditions in many of the region's major population centers. We see strong traditions of giving. And so it is, let's say good news for the sector that looks to, you know that needs to get additional capital to solve some of these problems that we have an influx of this philanthropy. And what's interesting is that a lot of these philanthropists are the second and third generation, who were shifting from say, more traditional ways of giving away funding through to a much more strategic approach to grant making and looking to mix that up with other kinds of approaches often derived from the market like social enterprise, social business impact investing to try and address some of those challenges, and also using vehicles like family offices to address that so in Singapore alone we saw about 200 family offices open in 2020 that was on a base of several hundred so it's really really rapid growth that's in part Singapore's emerging as a regional philanthropic hub in part because of the flight of capital out of Hong Kong and also Singapore emerging as a really safe regional hub for capital. And so growth in philanthropy implies growth in demand for understanding impact which is where the evaluation community comes in. And so I think it's important to talk a little bit about how philanthropists are different than government as as commissioners evaluation this is all very generally heavily generalized it will depend on where you are it will depend on the type of philanthropist. You know if they're deploying $2 million a year that's going to look very different than if they're deploying $60 million or $200 million a year. But in general, they are more dependent. But that also means that they do have lower accountability if they decide they want to give all their money to right wing causes as many philanthropists in my country, the US would like to do. Then there's really nothing that anyone who disagrees with those political views can do about it. They tend to be agile. So faster speed lower bureaucracy, where you might be in, you know, your years long conversation about doing something government often in philanthropy you can get that moving quite quickly. There's also an innovation orientation which we'll touch on in a minute a little bit more. And often you've got smaller teams in philanthropies with less in house experience and evaluation then you might find in institutional philanthropy countries with a stronger philanthropic tradition of institutional philanthropy. And finally, you do see more of a focus on legacy personal connection with impact. This all also ties back to the fact that a lot of regional philanthropy is, I mentioned not institution but also they might be more likely to be operating foundations. They run their own programs, and they are vastly more likely to be tied to family businesses. So in this case, if you're running a philanthropy or the philanthropic arm of a business, your board may well be your in laws, your grandparents, you know, is a family operation. So there's a real focus on the family's name, family's legacy and its connection with impact. But I will say, looks can be deceiving. So big problem we have big needs for lots and lots of social issues that are happening getting worse because of the pandemic. And philanthropy is one way of potentially helping to close that gap, but we always think about philanthropy is being huge and it is huge. So, even in this is in 2019 foundation assets by country there's an estimate from UBS who did a global philanthropy and they estimated about $1.5 to $2 trillion held in global foundation assets and that number is only getting bigger. So let's say that's a let's call it a solid $2 trillion held in foundation assets and that sounds like a massive amount of money and it is. But if philanthropy is only typically added at the most deploying about 10% of that per annum or about 150 to $200 billion per annum. That's actually not a huge amount of money. That's equivalent to about the GDP of New Zealand, or the state of Oregon in my country. So what that means is that, yes, the capital is very valuable, but it can't fill gaps. So what does that mean in terms of philanthropy's role philanthropy has to be catalytic. That's, I think, just if I leave you with nothing else I would say philanthropy's role is to be catalytic. And it really needs to sorry, a timer for myself just started going off. philanthropy has to be catalytic. So, what we know and says a great quote that I can't take credit for that the road to scale leads through public systems. And so when philanthropy is thinking about how it needs to make an impact that should be looking at understanding the kinds of models that we can test and evaluate that can then get rolled out into public systems and in Asia. And I would argue in a lot of the developing world. That will also include private systems and I know that can sometimes be difficult for us coming from countries that have strong sectors in terms of public provision of services but in India alone. Half of young children go to private schools that's massively more in some countries in terms of early years sector higher education sector, private hospitals, poly clinics are run by private companies and I think we need to acknowledge that and how we think about evaluating and instigating models. And finally, I just want to talk about a complication here and I said, we have to start at square one. So I'll just give you a moment to read this quote in Asia the nonprofit sector is still in variable stages of development. What that means is that in terms of channels of giving ability to deliver effective services capacity on the ground. And even just the basic information on issues and needs is not as well developed in many countries in wider Asia as it would be in other geographies. And so what that means for evaluators is that there is typically a very limited evidence base in the region so most evidence is produced in the global north of English speaking wealthier countries and even if you have a proven intervention, contextualizing it here. It means often adapting in an environment where there isn't great administrative data and where you really have to be testing whether or not solutions developed in other parts of the world are relevant and appropriate for these environments. So going back to what sky was saying earlier, attending to community and context in all of these very, very hugely diverse linguistically ethnically diverse communities that are represented in Asia means accepting that there isn't great evidence already out there. We also might run into issues of different capabilities of field staff there just isn't the body of institutions doing evaluation work in many environments. And there's also differences in terms of infrastructure availability, even to the point of things like digital connectivity may not be consistent in the same way in terms of what evaluators have access to. And also I think it's important also to acknowledge that their scale differences and how we have to think about doing work in some of these countries. I was speaking to a friend the other day who used to run UNICEF projects for girls in in India and North India and she was doing a program on conditional cash transfers and there were 17 million girls involved in the pilot for her program. And I think given that I live in a country with 6 million people, just a totally different way of thinking about scale and needs in this context. And finally, lots of donors are also new so taking it back to philanthropy lots of donors are sort of new visitors to evidence land. And I think there's a tendency as a in this sector to, you know, it can be quite academic can be quite crunchy, it can be quite off putting to people who are not experts and these are folks who are often really well educated coming out of the business community and they are coming in, maybe having programs that their families been running for 20 years. And they, it's, it, these conversations have to be handled sort of delicately to not a scare them off from evaluation or be disincentivize them because you don't want to tell them that a program that they haven't have been running for a long time doesn't work. So that's all contextual factors that are really interesting to think about. And many just don't even know how to do monitoring let alone evaluation that goes for the donors and the community organizations that you may be working with. So I'd say when we're talking about using good evidence we're often starting from square one, that's not to say across the board you have some very sophisticated organizations that are doing this work and the learning curve that they're on that many organizations on is really fast. So there be people who've gotten started as donors in the last five years and within that five year period they've educated themselves to a really strong level and are able to commission evaluation work well but I wouldn't say that's the dominant characteristic of a lot of the organizations. So, so what does this mean for you, like why am I sharing this with you why do I think it's interesting. Why should Australian evaluators care. And look Df giving you a cute little dragon with a cute little kangaroo. The point is that, given the scale of these issues there are real opportunities to make difference by bringing your brains your talents to the region in terms of trying to think about how you can engage an evaluator there. There's also great tailwinds I mean I think Australia is really well positioned to work in the region, because, particularly because you've got really strong diplomatic trade cultural academic ties you've got a new. You've got Asia campus you've got Monash Malaysian got government he Vietnam you've got UNSW Deakin who've got very strong presence in India. So there's really strong Australian ties into the region and obviously all the trading ties that exist. I also think it's just a huge amount to learn as an evaluator in this region. So some of the world's leading organizations doing this work are already active, particularly in South Asia so it's a huge amount of work being done by folks like J pal, and some really interesting innovations in terms of intervention and evaluation design so for example I came across a team the other day who are doing AI to drive behavior change through social media which is incredibly creepy and also probably going to shape all of our lives. And I think there's also good potential for Australian evaluators to help accelerate adoption of good evaluation practices in the region. So we can learn how evaluation might be done a bit differently and particularly how to engage with communities in diverse contexts. And also I would just say there are also lots of donors at home in Australia who are also engaging with some of these questions and involved in some of these that also need good advice. So learn more shameless plug for our conference you can still access some of the conference. I'm the CEO co host of Monash University a summit every two years, and the last one was held in March ticket is still available if you'd like to go there's some really interesting talks that were done there including by the likes of J pal, who bring in some interesting context from the world outside of Australia. So with us, I will close. Thank you very much. Thank you so much, Mary. So we've covered a huge amount of ground with our three presentations today. I'm conscious we've had a very rich conversation happening in the chat. And some leads itself to some some questions and some some were quite practical questions, one which I'll knock over straight away which is Mary can you briefly describe a family office which you referred to in your presentation. I'm probably going to get the technical definition wrong and somebody's just put this in the field but basically, and so Dr Google can tell you more than I can but basically it's a structure that is starting you started to see more and more of which is like a vehicle that allows very very wealthy people to organize their investments, giving and plan for legacy and that often tapes the form of having kind of a professional advisory team that you would bring to help handle your wealth. That makes sense. Yeah, and thank you Brian for posting a definition in the chat. The one thing I did want to point to just as sort of an overarching question for all our panellists is you've all spoken about how evaluation intersects with a lot of other professions and industries. And I was keen to hear from each of you any suggestions that you have about how evaluators can help bridge those gaps and how you better connect evaluation and other professions that we work alongside. It's been a while since we heard from you I might bounce back to you to start with. Maybe if you could make a few comments about about how we bridge the gaps from evaluation across to the people we work with. Yeah. So I think some of the ways that we really try and do that within our practice we have a few kind of key principles and so obviously our overall intent is that first nations people and other people from many cultures step back into what we would call our ancient wisdom. So we really kind of believe in this fact that as first nations people we've been doing evaluations since time immemorial we use evaluative thinking and practice to understand how we make like how we make cultural choices when the ceremony need to occur. How do we care for country and when is the right time to be doing that how do we care for ourselves and and when do we do that also so there's lots of decisions that get made using evaluative thinking. In our everyday lives and that's true for all of us I believe so I think there is a place for everyone and so I feel like one of the key things is allowing everyone the opportunity to build confidence to kind of step back in and lead in evaluation as opposed to us as consultants being the experts who need to keep coming in over and over. In fact we do need to build that skill set within communities to continue doing that themselves. I guess that one is nerve wracking for some because that means doing ourselves out of a job but I think it should at the end of the day absolutely be our intent. But some of the things that we would do is obviously spending a lot of time in community we like to kind of you know spend at least a week just being and so no general outputs to kind of go with that it's just how do we build really good relationships how do we let people you know dig in and understand who we are plus it's out a little bit and make some decisions on behalf of themselves as community is this the right person to be coming in here should they be you know helping us make decisions are they a good fit and so really kind of leaving that decision making in the hands of community. We also try and employ at least one local person on every single job that we do that their payment is the same as any other consultant who is on that job. By doing that what we get to do is demonstrate that local knowledge, a good understanding of the kind of structures that exist in community, the pitfalls, the kind of things that just happen, and the people that you need to be connected to that is a really key person to have in a successful project and so that is not a skill that can be learned to do something that comes from living and being in community and having that local knowledge so it's another really important part that we would encourage people to bring into evaluation and then obviously this part of working across a continuum that evaluations that within community. It's already happening. And so how do we tap back into the moments where community is saying, these are the things that we're experiencing this is the potential ways that we could solve this challenge and maybe it's with program or maybe it's with collaboration or maybe it's with something else that is some policy or some design stuff that we need and so acknowledging what is being called out early on and then wrapping evaluation and development evaluation across all of those parts and playing that support role so that community can continue doing this in the cells is probably how we would definitely see that we we bridge that gap and and then step out as well. That's fantastic points to make this guy. I'll ask the other panelists before we move to a question from the chat. Margaret did you want to reflect on any any thoughts about how we can bridge the connection between evaluation and auditors and other people that we work with. Okay, look, in some senses I think that it's the diversity of the sector that's actually the strength of it, and and really probably that's, you know that's great, and we don't need to bridge that any gaps we actually just need to recognise the various capabilities the various approaches that exist across and I think that that was what I was in part trying to draw out in my presentation because there are certainly things that we do in performance audit that a lot of other people don't do but equally there's so many things that we are not able to do that others can. So I think the real key there is to be just be honest about the different abilities and skills that people bring to this profession overall. And I guess be true, be true to yourselves when you're being, you know, bidding for work or doing work for agencies so that you don't try to, you know, present yourself at something you're not. So yeah that's how how I respond to your question. That's a great, it's a great take on the on the idea of bridging rather more about inclusion and recognition. Lovely reflections. Mary, I'll pass you to reflect on this one and then we have another sort of question emerging from the chat so maybe start with this one and I'll pose the next one to you. Sorry, start with which start with which one. If you're going to start with any reflections you have about how evaluators can bridge the connection between evaluation and the other professions that we work with and perhaps in your case philanthropy. Yeah, I mean, look, I think for the question that will be on everyone's mind is deploying capital, right and that's really grotesque way of just thinking about everybody wants to make social change but at the end of the day, some people hold money, and they have to distribute that to make change in the world. And whether that's a philanthropist or public sector, one of the things that I feel like we're always coming across is just justifying a value evaluation. Okay, that's particularly the case in geographies where evaluation is not as commonly done or where you have a state that has a need to project authority and kind of being the best of all possible worlds. I live in that country that may have a little bit of a concern about evaluation because it might, you don't want to look under rocks. And, and philanthropy also right you don't want to tell up you don't want to look at evaluation and say, Well, well, looks looks like that thing we've been pouring money into for a decade didn't work. So I think it's what I have been trying to do more in speaking to people is really just framing it as learning and about, they were always looking at the best and most efficient ways of getting to the, the outcome that we all want and that the process is should be seen as one not of audit but of learning. And so that does help to reframe thinking and I think that particularly helps folks who are not maybe not as not as who are newer to evaluation whether that's within government or within the philanthropic sector to frame up discussions with their stakeholders to help them understand what the value is. Just to get married I'll do the follow up questions is sort of a series of posts in the chat which are getting you referred in your presentation to the idea of looking for what works. I think that that's been a very controversial kind of concept in in the world of evaluation where there's lots of people looking at things like one of our commenters have mentioned human learning systems. There's sort of two questions in there do you think we should be thinking about this idea of what works and who gets to decide what works. I think it's hard self determination. I might I might just hold with that one. If you want to have any reflections in that space. Do you mean what works in evaluation practice or do you mean what works in interventions. I think you've been talking about what works in interventions but happy for you to take it either way. I think I'd rather take that one evaluation. So as I mentioned, I'm not an evaluator by training so I don't want to get into methodology. It's important that we temper the what works question with the knowledge that's what works is that what is published on what works is often published in places that don't address the reality the lived experience and realities of many communities. So I think what that means is just looking at generating a being aware of that so compiling as much as we possibly can about what does work and synthesizing evidence we can acknowledging the shortcomings of that evidence and then looking to fill the gaps in the evidence and that's why things like systematic reviews and evidence and the gaps are so powerful because they do show us where we need to be focusing our energy to strengthen the evidence base. And so for example, CI and Singapore has just helped to establish a center with a foundation at the National University of Singapore that's going to be focused on evidence translation and early childhood. There is a lot that is not known about how early childhood programs work in Singapore that's, you know, maybe well covered in the international literature. So a lot of the work that we're doing and trialing is about testing how it works in context here, and hopefully then contributing to the learning in the sector. Fantastic thanks for those reflections Mary that's really helpful. Then there's another cluster of questions which is sort of directed for Margaret and this is about the idea of an evaluator general. And there's a few questions I think we can confirm that Margaret space in New South Wales. There's a question about whether you think there might be a state based equivalent of of the Indigenous Productivity Commissioner so in terms of overall governance for this space. Any reflections Margaret. Well, I might just start with the whole concept of an evaluator general which has been around for quite a long time I think it but probably hasn't got a lot of traction. I suppose my reaction to it is really the more independent scrutiny of government, the better. I think. And so, yeah, an evaluator general that could be truly independent and could apply the same sort of professional skepticism that that we all just as apply. And that would always have to publish outcomes, etc would be good. I think there are still some some real challenges or constraints to that role and they sort of constraints that I face. And that goes to really data quality data and good quality performance indicators. I think there's been a lot of lip service paid to those sorts of things through you know various efforts for a program evaluation over many, many years. The government is still pretty immature in those areas and maybe isn't as committed to the notion of evaluation as they perhaps sometimes portray. So yeah, I think the more the merrier, if that was possible, but yeah we do have to start to invest in good data, good KPIs transparent reporting those sorts of things whether it's through an audit function or a evaluator function. The issue of relationships, more broadly with the indigenous community. As I said before, there's some real constraints on my role to be able to collaborate and work with other stakeholders. That said, we certainly in any, well in a number of orders that we've undertaken we have got some expertise from community to make sure that we're not just totally off track. And so the more that we do that and introduce that sort of view into our work the better. Fantastic and I would like to give sky just the final word and that idea of should we have an indigenous evaluator general sky reflections. Yes. I think it's a great idea. I think that the work and the vision, the indigenous evaluation strategy and the concepts that are being floated there are incredible. I think potentially there's some ways that we could look a little bit more around how we preface diversity across how community do a valuation use evaluation and where that kind of comes in and how community coming might be looked at a little bit more. But I think the concept and the way that that could be held at a national level and then also at a state level would be really interesting and something that we should strive for. Fantastic. Thanks so much sky and look thanks to everybody who's participated today has been a really rich conversation in the chat really appreciate your time we are bang on five o'clock.