 I would like to begin by acknowledging the digital owners of the countries that we're all joining this session from today. I'm in the lands of the Gadigal people and I pay my respects to elders past and present. Now, acknowledging country as we've become accustomed to is a really important way of reminding those of us who are not indigenous that we're living and working on land that was never ceded by First Nations peoples. It's stolen land we're all benefiting from. And I guess with this brings a real privilege and responsibility, I think. And in today's session, we are incredibly fortunate to have Associate Professor Megan Williams share with us her reflections and insights into how our evaluation can be connected with indigenous knowledge as a country with a specific focus on practical action that I think we can all learn a great deal from. And so Megan and I are preparing for this session, talked about our real hope that today's session in engaging with indigenous knowledges and country that we move beyond those acts of acknowledgement that we've become accustomed to, to actually a deeper understanding and engagement of all of our roles as people involved in evaluation and in work in this field or interest in this field. While we know that acknowledgement and symbolic gestures are critically important, I guess this is a week in which we are or should be reminded that it's absolutely not enough to just be having these gestures of acknowledgement. It's Reconciliation Week. Yesterday was Sorry Day, the 24th anniversary of the Bringing Them Home report which detailed for the first time on a national scale to a non-indigenous audience for experience of the stolen generations, the absolutely devastating impact of assimilationist policies that removed Aboriginal and Trusted Islander children from their families and the intergenerational legacy of that. And yet we commemorate that day, but we also still see increasing numbers of Aboriginal children being removed into out-of-home care, disproportionately in our youth justice systems, we see grossly disproportionate rates of Aboriginal people being over-policed and incarcerated. Institutional racism is everywhere in Australia if you care to look for it, including in evaluation. And we have a Commonwealth Government Indigenous Evaluation Strategy, and Megan is going to engage with that a bit later on in this conversation. And we're absolutely thinking about and engaging with ways to bring more visibility and accountability around evaluation in this area. But today is also the anniversary of the 1967 referendum in which Australia voted to include account Aboriginal people in our national census and also to enable the Commonwealth Government to make laws for Aboriginal people. And certainly, I think those voting at that time didn't feel like that was just a symbolic gesture. They too really wanted to see practical systemic change from that vote. And while we do have had seen legislative reform, we now do collect detailed data. We can precisely measure the gap in life expectancy in incarceration rates and education outcomes. Do we know enough about whether and how this is actually making a tangible difference in most Aboriginal people's lives? We've had decades of Indigenous policy and programs. You know, how and where is that achieving intended outcomes? Now, I know these are not unique questions or new ones, in fact, but returning to those questions about how evaluation makes a contribution in improving lives and policy and programs, thinking about metrics, metrics of success and approaches and methodologies. I think this is really critical work for us all to be involved with. And I think Megan is such a wonderful voice in this discussion. We'll briefly introduce her, because she's very impressive by her before actually handing over to her to introduce herself in a way that she might prefer to. But it's suffice to say that Megan is an associate professor, the research lead and assistant director of the National Center for Cultural Competence at the University of Sydney. Megan is a radiary through her father's family. She has over 20 years experience working on the programs and research to improve the health and wellbeing of Indigenous people in the criminal justice system. Megan has government and industry funding and collaborations for research, including about health service delivery, workforce development, facilitation, community-driven research and evaluation. Megan is chair of the Justice Health and Forensic Mental Health Network Human Research Ethics Committee, and we will certainly be covering off some of her experiences on that ethics committee and her reflections of that particular area later in our conversation. She's a trained Aboriginal family wellbeing program facilitator. She's active in research translation. She's conveyed Indigenous people's research stories and expertise to professional bodies, to communities, to parliamentarians, students and in the media. Megan's director of and a contributing editor of Health Media Organization, crokey.org. So those of you who aren't already across that should do so. That's a wonderful publication. And also associate editor of Health Sociology, review. Phew. Megan, I know you'd love to be introduced in such detail in that way. But I might just pass to you now to start, you know, introduce yourself in the way that you would like to, but equally to perhaps talk a bit to us about your journey into and around evaluation. Mandengu, thanks, Ruth. And you're a Dumarung. Gidei. I'd like to acknowledge I'm coming to you from Gadigal land and I pay my respects to ancestors and spirits of this land and the elders of the past who carried forward knowledges of the present that elders of this community are able to pass to visitors like me. And I think of all the deadly young people of the future. And as we're adjury, we're definitely encouraged to think of ourselves as belonging to seven generations. And so I know I'm part of four generations with us today in this dimension. And, yeah, so it's not a stretch to think about the next generations coming forward. So, yeah, to like to acknowledge country and all that it holds for us. And I'd like to acknowledge just a couple of people I know online. Josie Newton, who I do call my auntie and we've been through, yeah, many projects and struggles together as we're adjury in mainstream systems. Shout out to Amy Dellet and also Quotsip too, big love for Quotsip. So, yeah, thanks so much everyone for joining. It's very humbling. And I know there's many people on the line with more experience than me in evaluation. But I suppose what I can say is that I've tuned in to really people like in my family who I think are excluded from health services or don't feel comfortable accessing health services mainstream, often not even Aboriginal or Torres Strait Islander community controlled organizations. And so, if you're not participating in health services you're often not in research either. And then on the other side of the coin research using administrative data, data linkage research, which is used in evaluation and how that has another way of excluding Aboriginal Torres Strait Islander people by I suppose assimilating us into the main narrative or slicing us off to deal with in some other special way. So, yeah, sort of been through thick and thin with both those dimensions. But we've set ourselves a really ambitious topic. It's deliberate. Ruth and I've got to keep ourselves amused and we've got to keep ourselves growing and developing and stretching. So sort of partly terrified as well as humbled. And so one project that I'm involved in is the Center for Research Excellence in Stride and that's strengthening systems for Indigenous health equity. And we have pushed ourselves to form the Deadly Poets Society. And I'll just claim that I'm a social scientist and I'm a writer. I try poetry and it's been absolutely fantastic to push our thinking. I know that there's all kinds of learning theories about disrupting our thinking. And anyway, I just wanted to start with this. It's by Gomoroi poet, Alison Whitaker. And she's at Union of Technology Sydney. She's also a lawyer. And this is from her book Black Works by Black Work 2018 by Magabala Books. Which way Asimov? I had to Google Asimov, born in Russia and a professor of biochemistry that lives in the US which way Asimov. So when I read this, just listen for country in it. You start here, ground. Take code from the rivers, grasses where they divert, so shall you assert, so shall you be exponentially. You will live like systems. Feed, die, build. Feed, die, build. No need for better, faster, cheaper. No need, but what was? What comes next? You end here, fend. And tropic algorithm take your back against what? Your task to know. So I love that. I could pop the text in the chat, but again, acknowledging Alison Whitaker, Gomeroi, poet and lawyer. And to me that it speaks that there's code in country and country leads us in all that we need to know, country pre-exists us, country relies on us and how well we do whatever it is we do. And we're all here because we happen to do evaluation and we are crazy if we do not take country into account in evaluation generally because of climate change, but also in evaluation with Aboriginal and Torres Strait Islander people because we do not exist without our connection to country and our identity and our health and wellbeing being related to country. So yeah, from my understanding of evaluation, my Western social science training in the very early 1990s, country was not part of what we were taught to collect data about. Yeah, so lots to talk about, gotta be careful because of time. We do wanna cover indigenous evaluation strategy and some key points on that. We may have Romley Mocat, the commissioner from Productivity Commission online. And if he is, we'll ask him maybe to say a few words as well. And also we do wanna cover ethics. And but mostly I'd like you to leave today with more of a working understanding of NARPINIA. It's an evaluation framework that I'd developed in the early, from the early 1990s. So I'm just gonna do a quick check-in with Ruth. Any particular direction to go in now? What do you wanna jump into NARPINIA? Because I think, you know, most of the people on this call are members of the Evaluation Society. And so we'll have subscriptions to the journal and we'll likely have read the article. I know I've had lots of conversations and feedback from other members of the committee and evaluators about how valuable and engaging it's been. And I know you've had feedback over the years from people as well about how they've learned from and used that. So it might be incredibly valuable if you could take us through that, Meg, and any reflections you've got about the process of developing it and how you've used it in the years since. Okay. Can I just have a show of hands from people who know Stuffle Beam? And give us a wave. And do you say chip model or sip model? Hands up for chip. Sip. Yeah. Anyway, now, because I belong to that Center for Research Excellence Strive, I'm gonna absolutely land myself in it right now and the deadly poet society, but I'm gonna win, you know, probably some kudos. I have written a poem about how I ended up going from Stuffle Beam's sip model to nabinya. And it's called an Evaluation Poem for an Evaluation Presentation, probably never to be published. Dear Mr. Stuffle Beam, you don't know me, but I've been living with you pretty much daily since 1992. You don't know Dr. Phil Crane either who introduced me to you or the excellent evaluation partner, Anna, who also knows you. Phil gave me a dot matrix printout of the sip model back then, one simple stapled set of pages that became marked all over with my pen. I liked your idea of identifying context for projects, don't get me wrong. I just needed to add country, mother earth, the environment, get the landscape into it to whom we all belong. I loved listing inputs of projects. Those lists can be long, but in Blackfella ways, you'd be surprised at the extent of them. Missing what you can't see, surely even from a positivist view is wrong. Our resources used in projects are incredible, they're not easy to see and this new list in Nabiña breathes in the spirit of the community. Your use of processes in the sip model was a bit of a relief. These are the things that can derail a project without processes, how can outcomes be? But Aboriginal and Torres Strait Islander people work so holistically. Ways of working, again, many people, it seems devalue or choose not to see. And products in sip model, well, that's easy to disagree. Aboriginal and Torres Strait Islander people are not a capitalist society, never will be. We're very spiritual and philosophical. We know we go backwards to go forwards with lateral thinking and multi-directionality that has to be seen and acknowledged in processes and learnings to pass them on intergenerationally. But thanks, Mr. Stufflebeam, working on sip since 1993. I've triple-checked and three times over. Nabiña is respectful to you and to any copyright ability. It's all food for thought, all to think locally, all for freedom in our beings based on our own specialty. So that's only my second ever public attempt. But that is the story of the sip model literally walking around with that dot matrix printout and I tore off those side bits that used to get through that printer teeth. And as I worked on evaluations in Gold Coast AIDS and Injectors Newsline, Youth and Family Services in Logan, Brisbane Youth Service in Brisbane, returning home. It wasn't until the returning home evaluation of three post-prison release services with Aboriginal, Torres Strait Islander organisations that I gave Nabiña a name and put it in the report to those evaluations. And shout out to my colleague, Il Sablino, who some of you all know, very experienced in evaluation, really encouraged me to submit it to the journal. And so since then, it's been one of the most downloaded for that year and that addition. And Nabiña has been used among Indigenous cocoa growers in local health districts, in child services, and a number of other places. And it's great now these days to be able to see where. So I wondered about taking, I could show you on the screen the steps really in Nabiña. That'd be great, Meg, the prompts. Yeah, so I'm going to share my screen. And just bear with me while I get it. OK, are you seeing how to use Nabiña? Yes, excellent. OK, so form an evaluation reference group. It sounds surprisingly simple, but it's surprisingly left off a lot. And if that was me, it would be led by and chaired by Aboriginal Torres Strait Islander community leaders relevant to that project or program context and community context. And I'm used to working with people of all cultures, not only all Aboriginal and Torres Strait Islander people on those reference groups, but people of all cultures, because I've been taught just need to have the right people in the right place at the right time. And we know that we all share resources and all have varying roles. But that is really critical. You could call it a working group. I often call it a team and really develop that shared ownership of our process. It's ideal if that happens well before a need for evaluation is identified. But I've had the experience in the last couple of days where there's a large evaluation already funded of a state government program. And they've only just realized now Aboriginal people are overrepresented. They haven't designed the evaluation to use, address, deal with, or engage with Aboriginal Torres Strait Islander people or issues in any way. And so now they are in the position of thinking they need to form an Aboriginal reference group. And they're really only doing that because they know they can't get through ethics without it. So that's a hideously devaluing, but very common experience. So if you know that there's that need for evaluation, ensure that first peoples are first and design evaluation because if you get it right for Aboriginal Torres Strait Islander people, you're more likely to get it right for anyone who's experiencing complex and compounding issues. And you'll get a great creative evaluation design anyway. So together, go through the prompts in the nub in your paper and identify the ones that are relevant. So I'll go through some of these with you after going through this list, but just to give you the global picture. So format reference group, together sit down and go through the prompts. You'll soon work out which ones apply, which ones don't. And you'll get ideas on which other ones suit that local context. These prompts are essentially to identify data from an Aboriginal and Torres Strait Islander local perspective. And that data is needed to answer research questions. So then think about ethics clearance if that's needed, collect and clean your data. The group goes back through the prompts and asks itself, does the data answer those questions? And through doing that, that's analysis happening and interpretation by Aboriginal and Torres Strait Islander people. And that triggers how reporting needs to occur. And all of this can be written up according to the nub in your prompts or a thematic analysis arising from that data or by program logic as well. So nub in your, it means to do something, reflect on it and refine and keep doing it in my understanding in Wiradjuri language. And so that's the same with using the prompts. You don't just use them at the beginning and then never return back to them. They're excellent to go back to to ascertain if we're on track when it comes to reporting according to the question. So it's a framework. So it's not going to do the evaluation. It's not going to change how you might write necessarily, right? You still will write your research questions or you might have your aim, objectives and strategies and program logic can be applied to this. So it doesn't necessarily have to change any of the ways you or might be used to doing evaluation. It's the set of prompts that are informed by Aboriginal holistic definition of health as well as the United Nations Declaration on the Rights of Indigenous Peoples and also other current government strategies in this Australian context so that we've got those kind of parameters in there. And then also say by Wiradjuri knowledge systems are called Norm Sheehan in particular and also other Indigenous scholars as well as that National Health and Medical Research Council's Aboriginal Torres Strait Islander Health Research Guidelines. So I've drawn on all of those as well as Stufflebeams sit model prompts as well as my experience in evaluation to come up with these sets of prompts. So I'm a pretty average poet. I'm really much worse as an artist and this is the best I can do. And but you can see there the landscape, that bigger picture, part of the picture. And then there's all these resources that we put in. There's all these different ways of working and all the learning. So the inputs in the sit model are resources. The processes are those ways of working and the products are these learnings. And at the middle, you can do that thematic analysis and come up with critical success factors about what's working in a program if you wanna do that. So as I said, it's a framework. It fits with program logic. You'll be triggered to think about quaint and quail data and it can be used concurrent to a program delivery, concurrent evaluation. It can be used to reflect back on our program or project and it can be used as a planning tool as well. So there's those landscape prompts from an Aboriginal perspective. They really honor that the history of local community affects so much that happens. I'm about to go work in Aboriginal community, a large town that I haven't worked in or been among or even visited for 25 years. And their history is so different to that of Wiradjuri country. That the first thing I've done is tried to read up and remember what happened to those people. They had at least 16 tribes swept together into a mission by a church and that church pulled out and left that community bereft. So that history is really different to Northeast Wiradjuri country where I'm from, where there were massacres and poisonings and only really children remained of my family and they only really survived because of institutionalization. So we have to ask questions sensitively about the history of local community. It really is integrally linked with local population indicators, which we need data on. We need data on the current published stats about the issue in that location. It might sound like a no-brainer and all of these probably sound really obvious, but in my experience with evaluation design and certainly with reviewing countless funding and research applications, it's surprising how these things are left off. So landscape prompts us to think about all the other services available in an area and their range and their accessibility. Some of those mainstream services have a major influence on how Aboriginal and Torres Strait Islander services can do business and on funding, on confidence of the board, you name it. So we have to get data on these other contextual factors for Aboriginal and Torres Strait Islander services. Same with needs, it may again sound like a no-brainer, but it is surprising how often we don't really ask that local community what are their needs, all the range of their needs as well. We've got to go looking for how we see human rights instruments being used and at play in those local areas. Again, surprising how they never roll off the tongue of government officials or other community leaders as always legislation that affects how things are done. And in my field, which is specifically prison health, Medicare is a major issue and we're constantly trying to think about legislative reform so that people in prison might have access to health services equivalent to in the community to which they have a right. Policy, we've got to look at what shifts and critiques have occurred. So that's sort of our overview. Then in the NABINYA, you can see here how there's many more prompts with all of that detail. So I have this in a Word document which I'm happy to send around. And so I'll have one column with these prompts and then two more columns, an empty column where as we talk about it in the reference group and get ideas like what is the history of the local area? How can we get data about those items and then we'll have that in another column. So we start to get those ideas on how and where to get data from. And that helps of course identify what gaps there are. So here's some prompts. So partly these are here so that we don't have to keep recreating in these lists and doing this thinking every time. I found that I did pretty consistently want to ask and answer these types of questions across my experience in evaluation in Aboriginal and Torres Strait Islander context. And so they're reasonably all-encompassing but you just discard or cross out the ones that don't apply. And as I said before, you'll be triggered to think about which ones to apply. So history of the local area, the history of program establishment while that influences how a program's able to achieve anything. And same with, as I said before, colonization, dispossession. In terms of the environment, that is local population characteristics, socioeconomic factors. Differences and similarities, it's pretty interesting in the lives of Aboriginal and Torres Strait Islander people. Compared to others in the local area, that data can be really telling but it's not very easily available. But if you put it on your list to go looking for it, you'll get a much better picture about what's going on and what's possible. Same with this, it's often overlooked proximity to an accessibility of major health and social services, barriers and enablers. So there's a few handy hints there. You can have a look at local health districts, say in New South Wales, completion rates of respecting the difference, online training, compared to face-to-face training, the online's mandatory face-to-face is optional. And you wonder when you see lower levels of the face-to-face completions in some local health districts, what their workforce might be like compared to a district with high levels of that. So we start to get a sense of, we probably should be looking for Aboriginal Torres Strait Islander staff workforce, numbers and retention, racism, and other signifiers of exclusion. So you can see there's lots of detail in these prompts and same with all of these. So I won't go through them all, but there's a bunch of prompts to get us thinking. When we come to evaluate a program or a project, we do need to think about the impact of other programs and services on it and here's, and also the way the program or service is structured. So here's some prompts about that, some hints from Aboriginal Torres Strait Islander way about what to go looking for. That's really easy to overlook unless you've kind of been in among it. If you've been in among Aboriginal Torres Strait Islander service providers, as I said, you'll be stimulated to think, oh, in our context, this particular thing applies and we'll find some data about that. So I am in no way trying to be comprehensive here, but really stimulate people's thinking. So self-determination, if it's ever mentioned in that program or project you're evaluating, you have to think about getting Aboriginal Torres Strait Islander people's evidence and here's a couple of prompts. Same with policy, you know, really looking at the quality of policy from an Aboriginal Torres Strait Islander perspective. So I'll just do a quick check-in with Ruth. Shall I? Very on Meg, it's all great. One of the things I'm hoping that you'll also get to is we've talked before about the role of what you've referred to as cultural care and healing. And I mean, I don't want to interrupt your flow because I think this next slide will be of particular interest to a lot of people, but it would be great if we can also return to that because there's some really important conceptual work there, which I think is really valuable. If we've got time. Yeah, awesome. Great. So here's prompts about the resources and so stuff will be called these inputs. So resources are a little bit more friendly and really important overall to think about funded and non-funded cash and in kind. And my passion is about peer support. So this is also about looking at informal caregiving, for example. So all the reasons put in program time after time, we see so much Aboriginal Torres Strait Islander elders give all their knowledges, all their yarns. We've got to somehow capture that as a type of evidence because it influences programs and it influences evaluations as well. But yes, the financial, there's always the formal funding and other contributions. Let's look at people power, paid, unpaid, the role of the community. And I'm passionate as well about the multiple hats too. And can we do a multiplier? When I go to the prison, sometimes I'll go in one door as a visitor and I'll stay and I'll go in another door as a professional. And that just wearing the multiple hats and the richness of my understanding of that particular center, how I carry myself there, my confidence there because I'm not just as a professional, it's personal as well. I think that there's magic, there's something brought. It's very mature often of Aboriginal and Torres Strait Islander people who can juggle the multiple roles. Stakeholders love to do a bit of stakeholder mapping. Data in the old records, there's always plans that can be accessed so you can help identify resources, equipment and evidence. Theory is another resource too, explicit and implicit. And I love to see even about coercive control an article that Chelsea Bond has out. And Amy McGuire, we know a durable woman, we know her as a journalist, but she's a theorist. And so I can only but imagine the type of intellectual resourcing she has done to projects that she's living. So I'm aware too, it's around about now. I probably seem like a pie in this guy idealist and we're going to drown in data. But that's why you have an evaluation reference group so that you've got some accountability and you'll have some reality checker people on that and you need the ideas people like me so that power and collaboration. So here's some prompts about financial resources. Consider the adequacy of financial resources to meet the demand for services and support and the needs to support Aboriginal Tri-Strait Islander workforce development, to support program monitoring and evaluation and participation in research. Often we'll do the program delivery and not the evaluation. And also what about the knowledge transfer that needs to happen after? Often we'll get to the end of evaluation and collapse and that's a major problem. Human resources, I've talked a bit about that but here's some more detailed prompts and then material resources as well. And yeah, there's sort of phrasing here about meeting needs and future growth. And I think those two dimensions are really important because about a third of our Aboriginal Tri-Strait Islander population is under 21. And so when we think about Aboriginal Tri-Strait Islander people, we really need to think about them in our minds as primary school and early high school. As a population, we're not an aging population obviously. We're very young. So when we think about that majority of our population we've got to be pushing evaluation to answer questions well into the future for us not just answering questions now. So it's kind of very surprisingly different way that we would need to do things from Aboriginal Tri-Strait Islander evaluation perspective. Only I suspect we probably need to do that with any population but anyway. Yeah, I mean we do hear a lot of people talking about evaluation needing to be data driven but there's a nuance to what you're saying there which is there's demographic information that's relevant for us to think about but we have to bring that in in meaningful ways in terms of what the reality of people's lives but as you say, planning for that age bracket and thinking about having those voices also included perhaps in that evaluation work. That's right. And Aboriginal perspective I think leads me to know how rare evaluation is and how we really need it to work for us to pitch forward. So here's ways of working prompts. Big long list there, holistic advocacy take into account our diversity and then here's all the nuance prompts about that holistic caregiving. It's beyond the individual that's probably one of the biggest challenges to evaluation is how to not pathologize that individual. The family is our unit of intervention. What tools do we have to collect data from family perspective? That's very few. Quality caregiving is what we're after and that's really hard to assess as well but here's some prompts that hopefully help signify that from the Aboriginal way of working. And staff support can just never underestimate the power that the 97% have the non-Indigenous workforce and the power that they have in programs, in government and funding, procurement, you name it. And I think underscoring all of this is that relative powerlessness that Aboriginal Torres Strait Islander people often experience, certainly in research and evaluation. Learning, so yes, it's outputs, reflections, critiques, relationships, progress. How can we just measure progress? Yes, the results of activities, the extent to which principles are implemented, barriers, enablers, all really valuable information for the future, not just whether this project met its aims. So here's lots of... Can I just briefly ask you about the principles point? Because I think that's a really important one. It's a bit like theory. People kind of say, oh yeah, that's all very well but how do you actually apply that in principle in an evaluation? And I'm interested in any reflections you might have about measuring the implementation or the manifestation of a principle or series of principles in work you've been involved with. We've got to take you away from the slideshow. No, that's fine. If there's an example that you can think of easily that... Yeah, two, one is to do interviews and collect that the nuance qualitatively. But another is a matrix and like a scale. If you need to do that among large population and judge people's perception of the extent to which you witness or experience that principle in action. So it can be as simple as that type of say, participant or stakeholder feedback but there's nothing like eldership and their deep reflection on principle. As well. And if there is a knowledge hierarchy, I would privilege elder insights very much. Yeah, definitely. So, yeah, thanks for my little insights, I suppose. Lots about learnings. So I won't really go too much more into those except say when you do, as a group, decide on data and then you've gathered that data, hopefully Aboriginal and Torres Strait Islander people are involved in that and involved in that whole process of cleaning it. Really important as well that Aboriginal and Torres Strait Islander people are involved in designing the analysis. What's going to get run if it's quant or how it's going to get analyzed as well as well. And then that next step about interpretation, that's so what factor. It's really, I hope I'm not belittling by saying that but it is surprising how it's not resourced. I suppose we might want to do that and respect that process but resourcing it is a different story. So that means sitting fees, terms of reference, good facilitation, safety, group safety and reflection on group safety and debriefing as well. I've led a group of people through, we've worked with some of the most horrendous data that I could ever want to expose people to about impacts of parental incarceration and in the context of program evaluation and yeah we needed very careful facilitation like that. So you have to have a skilled person then who needs to be paid to do that. So now is probably two when it starts to all sound too hard because it's too expensive but if we want what we've always had we'll do what we've always done or if we are a country that wants something better for our own identity as Australian people we'll do a valuation better and we'll fund it better too. So I think that's about it from me in terms of that. Thank you, Anna. And the article we've put the details in the chat to of which journal that came from but there is a lot more detail in that but it's always really useful to have the author kind of talk through they're thinking around it all and certainly on our New South Wales committee we've talked a lot about bringing to life some of those articles in the journal and actually bringing in some discussion of them too so it's really valuable and really appreciate your kind of reflections around all of that. And I know you said at times people have kind of said that this seems a bit boutique but I'm really reminded and struck by the point you were making about actually if we design things well for Aboriginal and Torres Strait Islander people the likelihood is actually this is gonna be a much more meaningful framework and response for everyone. But I really feel like that idea of something being boutique is a way of trying to dismiss it perhaps or not engage with it comprehensively but there's nothing in what you were saying that as you said this might sound like common sense but there's so much of evaluation that's done so poorly actually setting out a comprehensive framework like this with these prompts is a really useful guide and I feel like there'll be a huge amount for everyone to learn from that having heard you talk through that. Yeah, thanks, it is really thorough. So that's one thing when you're doing evaluation is you do wanna feel like we've given it our best to be thorough and I still use it in the way anyone else would use it like sit down with the tables, fill them out go through it all and go back to it to help understand it all. And every time I think, oh wow this is such a lot of work. And I think who designed this? And then I think I'm glad it's comprehensive and I feel confident because it is. That's it. And there's a way that the prompts all lead to the next prompt. So I mean, you can just divide up the use of it into say four pockets of time, one domain at a time over four meetings if you need as well. Fantastic. Megan, can I ask you to talk a little bit about the new Commonwealth Government Indigenous Evaluation Strategy? And I know that this is something that the AES has been really engaged with and supportive of. And I know you mentioned, I know Romley Mokak actually registered. I'm not sure if he's been able to make it, but it's really interesting I suppose or it's wonderful to have an Indigenous Policy Evaluation Commissioner. That's an extraordinary kind of step and obviously it's development and release has been a significant development federally. But really interested in any reflections you might have about its development, its role, the procurement strategy I suppose that by the National Indigenous Agency and just anything that you've seen around that it's use and relevance in evaluation more generally. Yeah, thanks, Ruth. It was a fairly short process when you look at the timeframe here a little over a year and a half I suppose from when the letter came, issues paper was released, submissions were due and when the final strategy was released to government. But I do urge people to have a look at the 200 or so submissions. There's many quotable quotes in those submissions if you ever need leverage in funding application or anything that it is a good resource. But there's this quite an interesting progression paper that's come out of it. And I've just popped it in a little table. So for these central ideas, centering Aboriginal trust under people, perspectives, priorities and knowledges, building credible evaluation practices, improving evaluation usefulness, building ethical evaluation practices, improving evaluation transparency in this strategy progression pathway document. Each one of these is documented according to what unsatisfactory practice looks like in centering Aboriginal trust under people, perspectives, priorities and what does developing practice look like? What does mature practice look like? What does leading practice look like? So it is a way that we can assess will all be different and will all be different for different projects and different groups. But it's a really important critical reflection that we can do there. It is also a way of calling up unsatisfactory practice and it's a way of positioning and pitching to, people higher up in funding bodies. So it's pretty useful. So the procurement strategy of the NIA does ask people applicants to demonstrate how they do use Indigenous evaluation methodologies, how they draw on the Indigenous evaluation strategy. NABINIA is mentioned actually as one of the methodologies. It's in brackets, how you, you know, Indigenous methodologies with a name in brackets and NABINIA is included in there. So that is, yeah, astounding. It's astounding that it's 2021 okay, it's a funding that obviously hasn't been evaluated and the great risks that have come from that. So it's setting a high bar and really calling all of the Australian governments a whole of government framework. Another lever we can use, government have committed to that on paper. Let's watch and call it out. Put Australian government agencies, that's astounding. And it's for selecting, planning, conducting and using policies and programs. So it's very high bar and what we don't have. Meg, I think we might have just lost you briefly there. There are others having the same issue. Yeah. I wonder, Meg, if you turn off your camera briefly, whether that might help? Just so we can still hear you. Sure. At least it's that time of day where the NBN gets reary. It's getting busy in this neighborhood as well. So that's a little snapshot of that Indigenous evaluation strategy. And yeah, probably several people on here have made submissions or used elements of it. But it is there to use and we must go to it each and every time we're thinking about evaluation or research. I think it's really valuable and it's a really fantastic development at a national level. And I think rather than people viewing it as something that doesn't necessarily apply to them if they're not directly involved in Commonwealth government evaluation, using it as a guide is incredibly useful and embedding it in practice. Megan, we've got a fantastic question in the chat here which I wonder if we might be able to go to next. Yeah. Someone asking, how do you cost up the value of Aboriginal? Oh, you can see it. How do you cost up the value of Aboriginal and Torres Strait Islander knowledge and contributions to gain a true insight of program costs, such as lived experience and community connections that are inherently brought to service or program delivery? How to value this in evaluation to aim for more sufficient future funding models? Yeah, excellent. Well, I've got a quote here by Bill Jonas and Marsha Langton and IAXA staff, yeah. From 1994, from that early edition of the little red, yellow and black and green and blue and white book. And if you don't have a new edition of that, I might put my camera back on. Yeah. Yeah, you'll find that. We can hear you. So do get the recent copy of that. Get copies for others in your work area as well. And the quote from this, since 1788, the newcomers to our country have often assumed that indigenous people's cultures and societies were worthless because of their efforts to understand us were too superficial. The impressions they gained were those of tourists passing through, not those of neighbors. As Australians, we begin to see themselves as Australians begin to see themselves as part of the Asia and Pacific regions rather than as a satellite suburb of Europe. They are also seeing the first Australians through new eyes. So that to me speaks about value because of the word worthless, and so that question from Quotsip, obviously, evaluation has the word value in it. So it is a really important question. The answer for me is to do that with your local evaluation reference group and ask them to set that price. What price? Whether it's scratching together actually what wages people earn and estimating what elders are a bit like a professor. A professor gets this much per hour. Would you value your input at this much per hour? Oh, no, no, no, half that. Like you can actually have those discussions and come within a green local figure. There's also the government figures for participating in committees. And so they are good leverage to use as well for costing that. But I think absolutely be bold and do try to get an agreed figure. The safety is in the agreement. So the proper process has to be used and the right people to manage that process to get that figure. And if it's an actual figure because absolutely unless we do cost that out then it's invisible. And we will not make progress. Yeah, and I think Megan, I'm just looking back to the slide from Nubinia, your framework where you have those list of resources. And it would be, I mean, you may well have already done this or others have, but to actually work even working with a good economist, not an evil economist. A economist who could actually do some of that really comprehensive work around thinking and not reducing everything to a dollar figure because we also know that there's kind of traps in that too, in reducing things to a kind of economic argument. But we do know the pragmatics in which policy decisions are made and resources are allocated. Yeah. I did enroll in a postgraduate economics degree recently. Nothing like another degree. The textbooks just there. New Australasian macroeconomic theory. And I thought, oh, great. It'll definitely have Aboriginal Torres Strait Islander people. So, you know, go to the contents list. There's no chapter or maybe they've used indigenous. I'll go to that, you know, index. No, no. So, yeah, I did. Of course, being a nerdy mature age student asked the lecturer, how come? And they said that, oh, there's no one, like no one's written it. Well, do you want to write it? I'm like, I've got to pass the macroeconomics maths first. But, you know, that's, that invisibility is, oh, it's so painful. And why, like how hard it is to try to get through a book like that and to not be included. To be so disregarded all the economies that have flourished here. Imagine a creative chapter in there to bring macroeconomic theory to life in, you know, right across this Asia-Pacific area. And as well, so, yeah. Finding an economist is, yeah, another very hard issue. So, yeah, we, you know, a lot of this really, because like I said before, our population's so young, you know, we really ought to be thinking and the society could be thinking about how do we plan for curriculum that is going to have a process for young Aboriginal and Torres Strait Islander people to see research and evaluation as a career to want to do something like economics. If any of us here have responsibility for curriculum, for student recruitment, we've got to start thinking what is in our domain sphere of influence for us to change. Or we won't see an increase in researchers. And in my time being trained in the early 90s, I only got sent because I was terrible at health promotion going to Gold Coast nightclubs with a eight-foot condo man and a basket of safe sex things. I hated it. But I was lucky enough to be trained with Indigenous peoples from all over the South Pacific. And that was a three-week, what we'd probably call microcredentialing now. It's just one of those rare random projects that a university offered. And that we've got an alumni from that. And, you know, how can we start thinking about sharing, you know, maybe across institutional arrangement where we do have those smaller degrees so that we can build that workforce of Aboriginal and Torres Strait Islander evaluators. We are coached to think by our elders that Aboriginal people were the first researchers and the first evaluators. And we've got this one sort of word that encompasses something so holistic and powerful. So it's just trying to push ourselves in our roles now to not just do evaluation but to grow the sector for the next sort of three to four generations ahead of us. So it is something that we need to make a plan for, I think, as a sector. I think that's something that all of us involved in evaluation. And certainly the AES and the various committees should be turning their heads to. I mean, I think that's on all of us to think about that, you know, enabling, supporting, equipping work and mentoring. And it can come in many sorts of forms. So it's certainly a role, I think, for the evaluation society and for those of us involved in the various committees and in this sector, too. Yeah. Just briefly, Megan, we had Ken from ARTD has just made a note in the chat that ARTD recently set up an Aboriginal Reference Group to oversee an evaluation of housing products or projects, I guess. And we use the New South Wales Government's Public Service Commissions Guide to Aboriginal Protocols to help determine remunerations. Yeah, that's great. I do turn to that New South Wales Government Public Service Commission for regularly for workforce data and also their workforce plan ahead. So really valuable resource. Great mention there, Ken. ARTD. Got another very important reflection in the chat, Megan. Someone who inherited the condom and costume from you. No, you're working very well in that space. Thank you for allowing a peer evaluation. Oh, yes, we did too. We did pyramid selling, actually. It was so good. Yeah, worked really well. We did get in trouble by the health department at one point, but it was fine. Yes. And Megan, can we, in the time we've got left, just perhaps turn to ethics, because I know this is one you and I both feel really strongly about. And it's not just because we're also academic researchers. It's the really important role that ethics plays in accountability, in reflection, in reporting and being accountable to community and other partners and doing things properly as you've outlined. So talk to us about why evaluators should have ethics approval. Give us some of your worst examples to demonstrate that and just why it's important and useful to embrace ethics as part of evaluation. Yeah, sure. Well, I'll have to start with another poem, Shame by Kevin Gilbert. Some say shame when we're talking up and shame for the way we are and shame because we ain't got a big flash house or a steady job in a car. Some call it shame when our kids, they die from colds or from sheer neglect. Shame when we live on the river banks while collecting our welfare checks. Shame when we're blind from Tacoma. Shame when we're crippled from blights. But I reckon the worstest shame is yours when you deny us human rights. And those ethics guidelines aren't human rights as such, but being on a human research ethics committee, it is a proper place. They are ethics guidelines that people have to adhere to. And I suppose what I've seen is every trick in the book that people try to get around, first on the human research ethics application online, if you do click yes, Aboriginal Torres Strait Islander people are going to be participants in this, then there's a whole heap of other dropdown fold out boxes that have to get answered about how and when and what resourcing. And it's very common that projects are planned without Aboriginal Torres Strait Islander people in them at the beginning. And so for someone to just tick no there and hope that the ethics committee won't notice. And probably people hope that they won't notice because there aren't that many Aboriginal Torres Strait Islander researchers. So maybe there's an assumption there's not that many Aboriginal Torres Strait Islander people trained on ethics committees to help with this accountability. So that really hurts so bad when we are kind of that you see that initial assumption. And then that's if an evaluation project, you know, an ethics application is put in for evaluation project. So there's yeah, one quite recently, well, actually I'll start with a great experience. For an evaluation I'm involved in with two or three government organizations and a consortium of Aboriginal Torres Strait Islander community controlled organizations. We've got three layers of governance with Aboriginal leadership at each layer. And so of course we were really confident in our evaluation design and it's intervention, research and evaluation. So it's really great use of research funding. And we didn't have any issues getting through ethics clearance. So the point is while people think that it's gonna take them longer to have Aboriginal Torres Strait Islander people in that design and the setup, you can get through ethics easier like quicker when all of those elements are in the right place. And again, it sounds like a no brainer but it is so surprising how often that doesn't happen. And we've had, you know, in the field I work in inexperienced people designing programs that are, you know, about extremely sensitive issues among people with many issues, mental health, suicide risks, suicide attempts, poverty and trauma, and to not have Aboriginal Torres Strait Islander people involved in the design of that evaluation, you know, really presents a risk of death. And you know, for people in prison just experience major long lockdowns to prevent the spread of COVID, to then have to participate because there's also that one project you couldn't quite tell the difference between the service delivery and the evaluation. And implicit in it was that because you're a participant in the program, you're a participant in the evaluation and that not being made clear. It's unethical, too. And it risks people's wellbeing. So yeah, so that was a pretty dirty, dirty one. And I'm just mindful of once again in that process, Megan. I mean, it's fantastic that you and others are sitting on those committees but that's an enormous amount of your time and energy that's then taken up with having to call individual researchers and evaluators to account. Yeah, yeah, that's right, really time consuming. And also, you know, I've had to do a lot of work on my own self, too. It's, I don't come, well, just the risk of over disclosing chronic people, please, and insecure, got all my own insecurities. And so my need to make sure when I provide a response, it's a squeaky clean, palpable response that'll stack up. So I'll be quoting the national statement and these guidelines and that one. And that's the enormous amount of work as well, me dealing with me in the midst of that and not the confident voice I might have if I had a much straighter identity as a, you know, non-indigenous educated person, me being English on mum's side, black on dad's side and having education. And it's I'm full of shame, too. So it's also dealing with those elements of ourselves. I don't have that many peers and mentors, you know, I cling to people like you, Ruth, because of knowing that you've done the work as well. So it's also, yeah, I guess the call out to be a great support person and ally to Aboriginal and Torres Strait Islander evaluators and researchers and really supporting those who show an interest because Aboriginal and Torres Strait Islander people in my experience are just such sharp thinkers with a very honed radar for principles and never forget when things don't go well, either and always remember and celebrate. So a real repository of knowledge, too. Yeah, yeah, I was going to say, Meg, I mean, we don't have any Aboriginal and Torres Strait Islander people on our South Wales committee. And as you're talking, I'm thinking that's that's incredibly telling and an indictment on us and as a committee, something that we really need to shift and work around and also think in more systemic ways, how we as a committee and as a discipline seek to address some of those issues. So thank you for your generosity and vulnerability in raising that. And I do feel like we need to take up that challenge. Yeah, one, one suggestion I've got there, you know, because a lot of us at UNIS and in these roles are pretty overloaded. But if there's ways of having young people involved, too. And because I still I find young people can be very outspoken, again, with that honed radar about principles. And and I have found and others have said Aboriginal and Torres Strait Islander people can be very loyal and will stay over the long term when there's vision for the role that they might fulfil and that safety and good treatment, too. So while, you know, there may be people who might not have the academic qualifications or the experience, when we boil it down to core things that underpin evaluation like principles, then we see the similarities in how and what people could contribute looking beyond qualifications or or roles but to some of the core skills. Yeah, thanks, Megan. That's really valuable. And I think it's something for us as a committee and on that flow from the New South Wales Committee has added in the chat the link to our Mentimeter, which is the committee's way of getting feedback from people. And we know that this is the time of day where people start to peel off and cook dinners and comfort disgruntled children and pets and things. But for all of you and certainly, Megan, I've been making lots of notes of you've been talking to bring back to the committee, but for all of you here and of course, you too, Megan, any feedback you can provide through the Mentimeter is really valuable to us. So just as a note before you peel off if people could do that, that's really valuable. But I'm also taking notes on what you're saying, Meg, so I don't feel like you've got to do that. No. As well as actually this whole talk. But there is a fantastic message someone sent me through on ethics and I would love to put this to you, Meg. We're running out of time, but this is a really gold one. Someone's saying with ethics committees, there are current processes and mechanisms that do work, but there's also ones that go against self-determination. There are a lot of non-indigenous evaluation agencies doing evaluations in Indigenous settings. And from my experience, ethics has been a reason they use for not sharing data, community data in particular with community organisations. And so that information or data is never owned or shared or accessible with community or providers that directly work with them. How do we work through this? Any insights? Yeah, and my working understanding of the national statement, NHMRCs and other guidelines I think we would find that it's unethical to not have that data sovereignty vested in the community. And it actually is up to an ethics committee to be skilled enough to request information about how that is realised. So, yeah. And I think if we go back to your point about those of us who would seek to make not an Aboriginal people, not Indigenous people who make a contribution, how to be reasonable allies, and actually just ethical professionals in our areas actually listening more and learning more from that work that's already been done and thinking about actually embedding that in our practices constantly and having that as a framework and a reference. And I think evaluations are good at reflection often and we should embed that more as a discipline and create room for that to do collectively and also calling each other out and embedding accountability in that work. We see poor practice, that's terrible for all evaluators actually where you see people taking shortcuts and being unethical and actually deepening institutional racism rather than challenging it. Yeah, the other question that's in the chat about, you know, Linda Klein, I mentioned differences between Aboriginal countries. And what do I suggest we suggest when a project spans many different Aboriginal countries like across New South Wales and ACT, but the reference group consists of only like six to eight self-nominated Aboriginal people representing a sample of those countries? Is this an issue? And it is an issue. It's both an issue in what it produces and does but it's also an issue in what we, how do we avoid that? It's only avoidable if we've got funding allocated for payment of somebody to coordinate Aboriginal trust at all of people's input and payment for people to, for their time. And not all Aboriginal trust under people require funding to be, payment to be at a meeting, especially if they're waged and they can be part of that as a job. But there's a lot of Aboriginal trust under people with one RDO work closely with. She's got a bachelor and a post-grad degree but she didn't work in that sector. She's absolutely razor sharp when it comes to understanding our topics. There's people like her that universities and government departments do not have easy ways we can fund her. She'll have to get on the HR system and all those logins and contracts and that puts an end to, you know, like the issue Linda's pointing out those six to eight self-nominated, they're probably people who can work their way around either not being paid or they can manage how to be paid. And if we've just had more streamlined administration processes and funding for somebody to help do this work, we will get people from all over who want to participate. If we set the project up well, Aboriginal people who can facilitate group safety and we respect Aboriginal people and we really show the value of their input, we will get people, but we won't get people if we don't have the right way to pay and the timeframe issues and all that. This is where the actual administration of research needs to change a bit as well. We keep trying to think of the same way that we do research business and Aboriginal people are gonna come stick to it. But if we can somehow loosen and reformulate a process of research or evaluation, then we'll be able to get a lot of input too. And young people, that's again passionate about getting young people involved. So that quiver and that condo man back to that old work, we really did do pyramid selling. It was a community-based organization. You got paid $100 for every survey that you got completed. But they could go to places we could not go as service providers and get the data. And yes, we did get criticized, yeah, but they could have just gone and made up and filled out all the surveys themselves. But I don't think they did. They were involved in designing the survey. They were involved in setting the targets of how many we needed. And in the process of analyzing the data, they didn't want dodgy data made up. They wanted all the pride they had in that data that we were collecting. So that's a critical point to, isn't it Megan, that we're talking about reference group or reference board, but actually what you're describing in the integrity of that process is actually having Aboriginal people involved from the outset and from the beginning in the program design and the methodology in all of that work. Hello to the Beagle. But I think it's, that's really critical, isn't it? But that's what we're moving towards. So it's not just about advice or it's actually about Aboriginal people undergoing and leading and being investigators in that work. Yeah, and so that work is about partnerships and it's the same for me, when I moved from Brisbane to Sydney, starting again and asking who are the elders here, where do I fit in, how do I be a good partner and what have I got to offer and slowly, slowly building those relationships. And so we like let's do that work because we appreciate and respect that work rather than because now we have to squeeze a project through an ethics committee. It's an insult. Absolutely. We've got, just before we wrap up, Megan, we've got a quick clarifying question in the chat about asking you the national statement in ethics. What was that that you referred to, just if you could re-reference that? Yeah, sure. That's the National Health and Medical Research Council's national statement on ethics, on research, human research ethics. And so I'm used to that because I'm from health. So that's why I always go to that one and that that NHMRC has got its Aboriginal and Torres Strait Islander health research guidelines. And so say for the ethics committee, I chair and I'm a part of, it is our responsibility for all applicants to meet all items in the national statement that apply to their projects, as well as those Aboriginal Torres Strait Islander health research guidelines because they belong to the NHMRC's national statement. There is wording about evaluation in those and my understanding others will probably have that clarity is if something like research is for the development of program quality, it may not need clearance by an ethics committee. But in New South Wales, if it's about health and it's about program delivery, it is protocol for us to get clearance by the Aboriginal Health and Medical Research Council's human research ethics committee. I just think it makes for better research and evaluation. Well, I guess another dimension to that is that also encouraging people to punt to publish evaluation findings. And so going through the ethics process means that you've got structures in place to support that and making the findings more accessible and distributable and accountable to. Now, Meg, I've just noticed you've got another little gem in every medium adding in generous little tips. So the other one is about alerting people to cultural isolation. You're kind of talking about getting more than one Aboriginal and Torres Strait Islander people involved in the process is not just one person. I think that's a really critical and important reflection. Yeah, that's right. I got invited. You know, me, I think it sounded awesome to be invited onto a university Senate committee, a curriculum review on implementation of the Child Sexual Abuse Role Commission. And got invited onto that. But to me, that's horrendous. And no questions asked about what my relationship might be to any of the anything. And so asked, I had to ask as well, can I invite someone else onto it? Because it's a Senate committee. You know, you can't just bring people in and out. So even that process was a bit brutal as well of me having to kind of ask, can I too? And but anyway, we've got another Wiradjuri Williams on the committee. So we've got a lot of nonverbals between us. And we are actually using Nabiña in that. And and it's a fantastic experience. So can turn out well. That's wonderful. Megan, thank you so much for your reflection, your generosity. There's really stimulating questions. You know, that kind of point you're making about how to get the right data, how to talk to the right people, how to ask the right questions in the right way. It's just so valuable. I think there's a lot for us to go away and digest and think through on the committee, but others more generally. There's so much and so many people in the chat passing on their gratitude. Thank you for your poetry, which is extraordinary. I really do think you should publish it fabulous. But we have recorded this this session. And one of the reasons is Megan asked in the interests of sharing information and using it as a teaching tool in the future, which I think is another sign of your generosity. So thanks again. Really appreciated. Meg's got her Twitter handle here too. You can follow on. You do some great tweeting about issues about public health and prisoner health, but also policy and evaluation questions, which I think make pretty fascinating reading. Yeah, lots of fantastic comments and gratitude here for you. But yeah, we really appreciate it. You've also got something that the naming game. Did you want to just? Yeah, it's something in the crowd, but also to sum up any final messages for everyone. Yeah, thanks. Yeah, I'm used to doing group facilitation, where we'll go around and talk about our relationship to Aboriginal Australia. And we've also teams I've worked in have had like a 10 point, you know, choose your own adventure on getting to know Aboriginal and Torres Strait Islander Australia. And so we've actually just flipped that with some prompts for us to fill in the rest of the sentence. My favourite local food and smell and taste is. And so all of this is about that closeness and confidence that we can have with our local Aboriginal Torres Strait Islander people and and Aboriginal and Torres Strait Islander Australia, you know, so that we can be good partners in evaluation and where we can understand what those ethics requirements are for us and we can advocate for doing better in the future. Yeah. So thanks so much for having me on and yeah, I can tell I can talk for a living and really appreciate the work of many people who've supported me and come before me. So yeah, truly grateful things.