 So good evening everyone. I'd like to open the session by acknowledging the traditional owners of the land upon which we are all streaming into this session from. So for myself that is I'm streaming into the session from Narn, so I would like to acknowledge the Wurundjeri peoples of the Kulin Nation. I'd like to pay my respects to our past and present and future and extend that respect and acknowledge their continuing culture, knowledge systems and contribution to our community. I'd also like to welcome and respect anybody who identifies as an Indigenous person from within Australia and outside of Australia who may be joining us this evening or who may be viewing this session afterwards. Look it's really an honour of mine to introduce our speaker this evening, Dr Alison Rogers. Alison is a very experienced evaluator, probably known to to many of you already. She's been on all sides of evaluation, practicing evaluation, commissioning evaluation, reviewing evaluation and conducting research on evaluation, which we're very happy to have her present this evening on. Alison has worked across many sectors, particularly health promotion and again working in evaluation, but also doing public health work as well. She's led many much important work here in Australia. I'm including the Indigenous Australian Evaluation Program at the Fred Hollows Foundation. We've been very lucky to have Dr Rogers as a part of the AES for many years, and some of you may remember her from organising the incredible conference that was held in Darwin in 2014. In the last several years, Alison has been completing her PhD in evaluation at Centre for Programme Evaluation at the University of Melbourne, which is where I've been very lucky to observe and connect with her over her research journey, and it's just fantastic to have you here this evening, Alison, presenting where you've ended up, but I'm sure also where you're looking to progress into the future. So before I hand over to Alison and we all hear her incredibly exciting presentation, I do just want to remind you that Alison's work has been published many places. I know Alison's going to be sharing some links and some references, but there are places for you to get more information about her work, including the AES website and several journals, including the Evaluation Journal of Australasia, but she's also on Research Gates, so you can see all of her publications there as well. Wonderful. So thank you, Alison. Over to you. Gee whiz. Thanks, Ruth. That's the most wonderful introduction I've ever received, so I appreciate that. I am a white Australian living and working in Port Stevens in New South Wales, and I'd like to acknowledge the traditional owners of the Waramai peoples as their land, as custodians of this land, but also pay my respects to the traditional owners from wherever we may be zooming in from today, and also a big acknowledgement to all the people that contributed to this research and that gave their time and knowledge and expertise into contributing to the findings here today. As Ruth said, I've recently, very recently, completed a PhD with the Centre for Programme Evaluation, and this presentation today captures that some of the findings from my research with non-evaluators who advocate for evaluation in Australian non-profit organisations. So I'm going to share the screen now, bring up the presentation. Ruth, have we got the thumbs up that you can see that? Yep. Okay, great. Thank you. And as Ruth said, there's my profile for Research Gates in which you'll find many of the preliminary pieces of work that contributed to forming this research today. And the slide bank of this presentation will also be made available to those attending today, and I'll also upload that onto the Research Gate. So I wrote a blog for the AES a few years ago, and you don't have to have read that blog to understand the presentation today, but the fable that I penned at that time really does, for me, was about demonstrating that convincing colleagues in non-profit organisations to do something that they don't want to do is a really difficult thing to do. And on an animal farm or in the non-profit context, evaluation is often one of those things that people don't want to do. There's a range of interpersonal challenges that prevent individuals from engaging with evaluation, but most interestingly, there are individuals who can really make connections with evaluators and find ways of being able to see the benefits of evaluation, the value of evaluation for their organisation. And in this fable, people like that were represented in the dog called Champ. And this fable was really very representative of the anecdotes that I heard from evaluators when they talked about how they were able to have success in non-profit organisations. Having people, dogs like Champ, on their teams really helped them assist in gaining momentum for change around evaluation and making things sustainable. So the fable represented, I wrote it at a time when I was really trying to investigate, okay, what do I want to know about these key individuals that can help build that bridge between the people that are resistant to evaluation and to the wealth of knowledge and expertise that evaluators can be able to bring in. So this fable was the starting point really, and it helped me capture what it was that I was looking for to and really show that I was really interested in understanding how individuals who are non-evaluators are effectively able to persuade their colleagues to get passionate and enthusiastic about evaluation and embed it into their routine operations. So the purpose of today's seminar, I'll briefly describe the problem that I saw in organisations, give you a very brief introduction to the methods that I use for this research. I'll take a pause there and you can ask me any questions about the research process, but then feel free again at any point to put questions into the chat box as we go through the presentation. But I've left plenty of time for a really interactive session for the second half of this. So we can have a discussion about some of the findings, the discussion points and the implications. It really is this, it's focused on understanding these interpersonal issues even though the bigger research project was focused on understanding the characteristics of that. So that's what I'm going to focus on today. In the implications sections, I am going to look at how my work, this research that has actually aligns with and where the difference are with the AES's cultural safety framework. Because I think although that the cultural safety framework wasn't published at the time that I undertook their research, I did draw upon the NH and MRC guidelines for conducting research with Aboriginal and Torres Strait Islander people. Because there were people that identified as Aboriginal and Torres Strait Islander as well as people from other culturally diverse backgrounds. And I did my best to uphold those guidelines and make this research as participatory and as interactive as possible. It really was about interpreting, understanding and making judgments in a way that co-created knowledge and really was a two-way process in interpreting the findings, especially around their own social interactions. So that's an overview of how the seminar will run starting off with a little bit of background about what the issue is and what problem this research is attempting to try and address. So I don't need to convince this audience really that organisations can use evaluation to improve, demonstrate effectiveness, demonstrate their change and communicate their achievements. However, there are often structural issues, resource constraints and interpersonal challenges that actually make this very difficult. Specifically related to interpersonal issues, people may resist becoming involved with evaluation because of the lack of dedicated resources, disconnected priorities, negative previous experiences, feelings of distrust and anxiety and difficulties with evaluation, terminology and concepts. And particularly in not-for-profit organisations where people are trying to solve difficult social problems, they're operating in competitive funding environments. They have to produce information for multiple audiences. It can be a really challenging setting in which to engage staff with evaluation. And there can often be resistance to change in non-profit organisation and this translates into resistance to undertake evaluation. It can also be compounded by challenging communication issues. We know in culturally diverse settings when communications issues can arise, when there are people who have different worldviews and have been socialised in different ways to understand knowledge and have had distinct cultural experiences. So this makes the not-for-profit setting makes for a very difficult setting in which these interpersonal challenges are prominent. So but to overcome these issues, therefore understanding more about effective interpersonal strategies is vitally important when evaluators are trying to engage with non-evaluators to embed evaluation. And there's lots of focus of the evaluation societies. They're trying to understand and unpack what this means, the interpersonal skills in terms of competencies and identify which skills such as cultural competence and communication, facilitation and conflict resolution, what that actually means in practice and whether it can be taught or not. So this research that I conducted attempted to fill that gap in the literature by collating some really practical strategies of what that looks like in practice of people that are able to do that effectively and be able to advocate and champion evaluation in their organisations. So like I said, it was part of a larger project that focused on the characteristics of these people who championed and advocated for evaluation, but specifically the part that I'm going to present today is around answering this key question, what interpersonal strategies can make evaluation more appealing and engaging for non-evaluators. I will now briefly provide an overview of the methods and really just saying that I took a really broad interpretation to what evaluation meant in an organisation right through from embedding evaluation systems through to one-off projects through to really thinking about it in terms of pro I didn't want to eliminate any forms of evaluation that was taken because I wanted to capture as many people doing different things as possible as long as that they were doing it effectively and to find out that that was a purpose in sampling strategy was that these individuals to be part of the study had to be nominated by an evaluator or by a colleague as someone who championed evaluation in an organisation. So they weren't nominated as having certain characteristics and those characteristics to build that profile were derived from the literature. The diagram that I'm showing here is just an overview of how the research was undertaken. It did involve interviewing 17 participants and undertaking case studies with four of those participants. The qualitative methods provided a way of understanding an increasing understanding by generating detailed information about the strategies that the individuals used and like I said before this research was based on creating and interpreting the findings in a joint co-creation way but I also did draw upon a formal theory from social psychology to understand whether the findings resonated with a broader evidence base. A little bit of detail about the participants. They all worked in non-profit organisations, human service non-profit organisations that were aiming to improve people's lives by providing health, community development and social services. The majority of the participants were at project level with no direct reports. All had over 10 years of working in the sector. Of the 17 participants, 10 were female and seven were male and the participants were either of the cultural linguistic group that was the focus of the organisation or that worked with the target audience for more than five years. And in total six participants identified as having a culturally linguistically diverse background four of which identified as being Aboriginal and or Torres Strait Islander and two grew up outside of Australia from the UK and a Western African nation. So look I'll just press pause there and see if there's anyone has any questions about the research design or wants a little bit more information about the background before I dive into the findings. You don't have any questions in the chat box yet Alison? Like I said before please if something springs to mind please pop something in the chat box and then I'm really happy as soon as I finish these slides we can really open it up for a discussion so even if you've got something there we can take the mute off and just have a conversation at the end but I'm happy to keep going if there's. There is a question here from Janet. I've got a question from Janet Conti. Yep Janet the theory from social psychology was social interdependence theory and that the specific part of that theory that I really use was the elements of cooperative teamwork so that was a theory that really helped me to look at what was happening and identify whether there were whether I could see the presence of the elements of cooperative teamwork from that social interdependence theory. You saw the one from Lucy? Yes with the the seven that are self-identified as having culturally diverse backgrounds so I there were 11 of the participants white Australians. They may have been white passing but it was there were seven participants that clearly made it very clear that they did have those issues with communication and with cultural issues so that was how that was determined. Yep okay I think we'll let those roll through and I'll go back to the slides and we'll kick off with exciting part which is the findings. Okay now to briefly summarize the findings I will give you some examples that the participants really drew out as as to what they were doing and how they were able to work. So one of the main things that the participants did they connected evaluation reasoning with the colleagues lived experience and used examples that were relevant to their life to be persuasive about the value of evaluation. They translated different types of knowledge between different people across the organization and beyond and they used the knowledge the skills of knowledge brokers within their organization to make evaluation terminology and evaluative information relevant and meaningful. Participants asked questions to probe for greater understanding understand the situation of colleagues and encourage colleagues to hear each other and think in different ways. They asked questions to probe for greater understanding and that they really made it personal by sharing analogies and individual journeys and stories to connect individuals and evaluation to organization goals and they also incorporated casual prompts about the benefits of evaluation in general conversations with a wide range of people. Now I could go on and on and on and listening and it would be great to listen to my voice the whole time but what I've actually done and if anyone knows my work you know I love lists and tables and producing information that's in digestible formats to be able to look at and what I've done at the end of this slide pack is created a bank of these types of examples and strategies that people can refer to to get inspiration about what these participants did or discuss with their colleagues about what might be applicable to be able to do and like I said it will be emailed out to everyone and it has a list if that's what you're looking for in this presentation today there's certainly a great deal there that can you can read through and get down to the nitty gritty of what these actually look like and in the in the research we've actually looked at some of the evaluation competencies and when it says things like listen to colleagues or communicate effectively well these type of strategies and examples that have included there actually help to bring that down to well what does that look like in practice so please we can I can show you that at the end of the presentation that it's attached but really know that it's there as a resource to be able to turn to I'm really interested now to also go on just to dive a little deeper into how those practical type of strategies really intersect with that social psychological theory and the literature but just to highlight that the value of this research really was about that investigative people who had genuine relationships with their colleagues that resulted in effective engagement around evaluation and they these participants were able to achieve the feat of persuading their reluctant colleagues to incorporate evaluation into their routine operations and look if this was a podcast from one of those organizational psychology work interviews that you I'm sure people here would have often listened to the next slide that I'm going to talk to is like the top five strategy top five specific strategies that these individuals did to be successful in their organization to be able to do this so I'll talk them through one to five but essentially this is the take home message out of combining it with the theory in the literature these were the strategies that enabled these individuals to be able to be effective and do what they do and the first one is understanding their colleagues as individuals so mutually beneficial working relationships were foundational to all their interactions they highly value the positive working relationships with their co-workers and they took the time to make really strong interpersonal relationships they understood their colleagues individual aspirations and the underlying purpose of their interaction their key motivational drivers and their enablers and barriers to participation and it really was on an individual level but they were also very inclusive they had a very inclusive orientation and they engaged on this individual level to find out not only what their aspirations were but identifying their individual needs their preferences their learning styles showed them how evaluation could assist they said that no one size fitted all and that participants considered who was on their team and how breast to bring out those talents perspective and worldviews this third one is really key um they're all key but this one is is my favorite because they really were able to find the shared goals that linked those individual aspirations to what the organization was trying to do and also how evaluation connected to that so they they looked at what what did that individual want did they want were they looking for a promotion within the organization were they looking to impress the boss were they looking to feedback meaningful information to the community beneficiaries what what was it that was driving them and then connected that to evaluation often in a non-profit organization there is such a synergistic alignment between the individual goals and the purpose of the organization because often people work in an organization because their values align with the organizational values so it was a matter of finding out for personally what that individual how they saw that coming to fruition and then linking evaluation um providing encouragement was very similar to also the way that they incorporated conflict resolution strategies so that participants promoted evaluation they provided reassurance enthusiasm support they made evaluation appealing um they gave praise feedback suggestions and they also provided help assistance support anything that would keep the momentum going and and to be able to um facilitate for moving evaluation forward and into the into the organization although there were minimal formal conflict resolution strategies the way they dealt with conflict drew upon a lot of these other um key strategies so by finding out about what made an individual tick they also knew about what made that individual um why would they triggered in certain situations and how they could prevent that from becoming you know a full-blown conflict or what was underlying some of the issues of why that they were reacting in a negative way or they turned someone who was um challenging what they said and drew them back in by taking on those challenges as something that could help the organization to be able to prove so they really did use a lot of the first four strategies to be able to either um offset or reduce the the chance of conflict but we do know that um in you know embedding evaluation can they they did acknowledge that this is an area where conflict is bound to arise but they they they looked at other strategies as to how they could do it so really in essence this research found that by finding those shared goals providing encouragement from a basis where meaningful relationships um and colleagues understand each other as individuals they're the basic tenets for effectively being able to encourage others to do something that they don't want to do and really this slide kind of encapsulates well you know this this research was done on people um looking at evaluation as the thing that they were trying to get them to do but obviously there's lessons can be learned in this from right across from from um you know drawing upon this way of looking at things from a social psychology point of view that may be applicable in other areas. Diving a little bit deeper um just related to that those um ways of identity of working with people on that individual level um participants really didn't um well they use their social skills to be inclusive and engage the wide variety of their diverse colleagues around this evaluation and in the literature this way of working is called individualized consideration or understanding their co-workers as individuals and it's when the person pays um a great deal of attention to the differences among peers and discovers what motivates each individual so for communication skills in practice they they really tailored their communication and they use their intuition and ability to adapt to the context and capacity to tailor the information on an ads needs basis um to be able to look at how that could promote evaluation but also prevent the underlying conflict um and in regards to cultural competence um participants understood the context they privileged multiple perspectives of their co-workers and they displayed um they highly valued those personal relationships um and in alignment with individualized consideration they displayed an approach that fostered involvement promoted equity tailored engagement and created a supportive environment for participation no matter who was on the team um and although these strategies you know you might hear as I'm saying them they are very similar to the extensive references from the evaluation field that working um that illustrate working in a ways that do respect cultural diversity and appropriate to context and we might call these um cross cultural culturally responsive or culturally appropriate evaluation techniques however the participants didn't think that these um labels adequately encapsulated the diversity in regards to age gender levels of education preferences for learning engagement levels of attitude with technology and administration and the participants actually didn't want to use the singular cultural dimensions to define their peers but preferred the option of acknowledging the multiple intersections of cultural diversity and I'm going to return to this point in just a moment to kickstart the discussion because I think this is where it gets really interesting when we're looking at the cultural safety framework and what this finding means but I'll just cover some other implications in regards to our practice and then return to that point to kickstart the discussion so implications for non-evaluators and for internal evaluators so these strategies um could form the basis of self-assessment activities and a lot of the examples that are provided in the slide pack at the end could be the basis of seeking formal feedback or informal feedback to determine the extent to which these strategies are being implemented appropriately in practice and the examples may provide inspiration for alternative ways of interacting or as topics of discussion with colleagues or other evaluators and in communities of practice um but for external evaluators who may not have opportunities to develop relationships understand the nuances of differences within specific cultures and interact with individuals on the same level over the long term because quite often these relationships of these people were over many years they may need to support engage the support of individuals who can enact these strategies and I think this is where we return to the fable and understanding how the postal worker or the evaluator that was on the outside of the fence was able to engage with champ as one of the non-growing aggressive dogs at the fence to be able to form that that interaction and and how can we identify these people and how can we engage them and recruit them and be able to to know how important they are to have on a team and so part of my research was developing a definition and a field guide for evaluation advocates these people who motivate others provide energy interest enthusiasm by connecting about evaluation with their colleagues aspirations and the organisational goals so this table is the who why what and how of how these individuals work and it really is a snapshot of how these individuals were effectively able to do that because obviously the the key top five strategies can be really applicable to work people working internal to organisations or with the same group of people over a long time but it really might be important to think about identifying and recruiting evaluation advocates to support some of these interpersonal things when it's not possible this is under review again with the open access journal of multidisciplinary evaluation but look you have it in this slide pack and feel free to use it for purposes of recruiting or for discussing with other people or for anything that you think it might be useful um so returning to the implications also for educators of evaluation the five strategies might be a useful way for presenting the interpersonal skills required to do evaluation like I said breaking down those high level terms into practical ways of of actually explaining what that looks like in practice and I think these this information might help evaluators to invite colleagues to assess their interpersonal skills to garner a more accurate sense of self potentially finding creative approaches to using the strategies to elicit constructive feedback it may help to overcome some of the positive self assessment of competencies in this area that evaluators may tend to give themselves and using it as a constructive feedback tool might be one example um but for theorists and researchers I suggest that further investigation is required to help elucidate those intersections between these strategies and cultural safety and this is where I'm now just returning to those points about these individuals not wanting to put that label of um culturally responsive evaluation on the work that they were doing I'm going to to link that actually now to um the Australian Evaluation Society's First Nations cultural safety framework that most people would have been aware of just recently launched um on at the um festival conference uh only last month so I'm going to consider what the implication this means for for this research in relation to that framework um and I have to say that you know when I looked at the frame there were many aspects of the research that I've conducted that align with the framework for example the the participants in this research express really similar sentiments to those things um depicted in this figure three you know advices listened to and acted upon knowledges and skills recognised and supported cultures are centred and valued and experiences are believed and validated and presence is welcome and respected so you know these things strongly align with with where this research place focus and another the excerpt from page 11 when a diversity uniqueness when it says in the in the framework that um Australian First Nations cultures are diverse as our First Nations within nations avoid assumptions one size does not fit all that could almost be a direct quote from some of the participants in this research because it resonates so strongly with what they were trying to what they were saying to me and in relation to time take time to develop relationships and build trust in every project even if you've previously been involved with community and organisation did not rush the process again um you probably you know me and just those few examples use the word time but that's you know one of the core things that is exactly uh relevant and relates to that um however um the participant strategies illustrate that they do work in ways that respect cultural diversity appropriate to context and are in alignment with the culturally responsive evaluation literature because they were reluctant to use the terminology such as cross-cultural culturally responsive or culturally appropriate and they really didn't want to um separate cultural competence from other types of social skills I think this is a really interesting point that is worth for the discussion and for the research um because many researchers have actually been critical of approaches that don't pay enough attention to culture to racial differences specifically and to underlying power dynamics so acknowledging everyone is diverse um potentially doesn't go far enough to address the underlying power imbalances and may well continue to perpetuate inequalities among staff so recent literature is calling for cross-cultural evaluation approaches and participatory ways of working to go beyond just looking at the team dynamics and the situation in front of people and think about the underlying system um the possible negative outcomes of encouraging participation where there are power imbalances in the workplace and determining if individuals would be willing and able to increase the emphasis on the social and emotional needs of colleagues while paying attention to those social inequalities um is a really key area that would be really worthy of further exploration and and research and I think it would probably be a great point in the conversation um to open this up now and just to see what people think and um to get perspectives on on what that means and how and what this information resonates with people so um like I said we've got plenty of time so um I'm I'm happy Ruth I'll just point out before we go that the key references and related readings are there available as well as link to research gate and these are the types of specific slides at the back that are that you can definitely flick through to see you know taking time to build rapport um so there's there's a bank of slides at the end of this block that really might help people really break down what um what these um type of strategies look like in practice um and and how that actually works but I won't leave all these words up on the screen now I'd much rather see people's smiling faces and have an opportunity to um hear from people about what they think how this research resonates I mean Alison I don't know if you saw Kahewa and apologies for pronunciation Kahewa made a comment but it was earlier in your presentation so I'm not sure that you may want to add to that Kahewa possibly well thanks very much hi Alison really really loved that thank you um I was just commenting on um I think the five uh kind of key things or elements that you talked about around connecting and translating and probing and sharing and prompting um seemed to lobby a lot of similarities or synergies with some of the different models of leadership that we're starting to see um but I thought that might we could possibly conceive of champions and advocates as leaders in an organisation or in a project anyway so might not be too surprising I wondered if you had any thoughts on that or if that came up yeah yep yeah definitely there's lots of um crossover here with uh with the leadership literature and and even individualised consideration the thing that I mentioned um really came up strongly in the transformational leadership types of models um but the part that I'm really interested in I think again would be worthy of further research and I think what the the point of difference of this research was too is the leading um from low power positions as well because the majority of these people were not in leadership positions it was really interesting to see um and it wasn't an aspect that I specifically focused on but I could have definitely cut the cake in a different way to really explore what that looked like um because I think this this could have potentially value for so many people who are not in middle management and who are not in senior level positions but to see how much power that these individuals were actually able to have in leading teams um even though they didn't have any formal power was something that would be really worth exploring so yeah there is um it's actually um hide I think it might be Cheryl Hyde who's got some great references on on low power positions and being able to lead so it's definitely an interest of mine to explore that further and I'm happy to talk more about you and lead some references so please email me if you want to take up that conversation yep Alison Karen was a great comment for you audio Karen do you want to um take yourself off mute and show your face just so that everyone else can see you as well or if you prefer not to that's okay too okay yep um for those you can't see the chat box um I can take myself off mute sorry I'm at least in hospital with a face full of stitches so I'm not going to take the photo you have to just stick with that it's really not a good look but yeah now I really really love the research um and the way that you've looked at it from the uh social psychological and organizational psychological um perspectives because most of the stuff I've looked at has focused on organizational systems and and leadership and and those kind of issues um so this I think this has got a really practical research to practice implication as you've done with this presentation but but the underlying stuff as well so thank you so much I'm really excited about it thanks Karen I think to that point I think um you know anyone working in an organization is acutely aware of the structural barriers and the resource constraints and those issues that constantly impede um you know in to to bring things to fruition but at at some point in time you have to take control over the things that you can control and pick up with the um with where you are actually able to you know facilitate change and move things forward and so it was kind of um after years of focusing you know I spent a lot of my life looking at the social determinants of health and just found you know very overwhelming and I completely acknowledge that they all exist but at some point in time you have to get out of bed in the morning and say what am I going to do today and bringing it right back to the interpersonal I think this essentially is also the same way that we can have most impact on those structural changes and resource constraints so I've actually come full circle and think yeah it's not it's not about ignoring those it's about looking at the micro to see how we can influence them yeah both yeah so they're not um dichotomous so that's great it adds to it and gives it a richer picture so thank you thanks thanks Karen um Ritu has left a question for you Allison Rania uh Ritu would you like to join in or yes hi Allison I'm calling from Delhi and yes I really enjoyed the presentation and the fable was like it's amazing what a way to link to in your entire research and congratulations um I am and I just to give you a context why I'm asking this question is I am in my final semester of masters in evaluation and my thesis is on evaluative thinking in communities so yeah so I was just trying to understand that did after after the the non-profits staff got engaged in evaluation and valued what it you know brought did they try to also engage the communities or were they more facilitated with the communities that it sharpened the the thinking of the communities and develop that critical thinking um thank you thank you for dialing in for so far away and um it yes thank you thank you for that question um the the questions that I asked were really focused on how they interacted with their colleagues and how they influenced their colleagues so there was a specific focus around what I did the the case studies I was able to look more how the organizations worked and what had happened as a result of their work to be able to see what some of the flow on effects were but it was still really focused on how those colleagues and evaluators who worked with those individuals saw their information reflected back so it wasn't a great deal of the study that focused on okay what are the flow on effects or the impacts of this you know resulting in there but it was really clear um that the way that these individuals worked and promoted critical thinking and evaluative thinking and reflection was really across everyone that they came into contact with so one of the participants um you know explained how she was working with her community reference group and it was about um using that community reference group as a means of testing her own assumptions and at the same time that meant they were included and brought into the process of um analyzing data together and making decisions about prioritizing things so it really um flowed on to every aspect that they did so while I got limited you know kind of data around how what the communities really thought of that I can see that these individuals applied those interpersonal skills they had across everything they did and and for them the evaluative thinking was really a mindset that they brought to everything that they did and that's why they really saw how useful it was in wanting to understand whether their work was making a difference or not and not just doing the same thing every day that these people weren't happy with just um repeating things without knowing if they were actually making a difference so they brought that evaluative thing that critical lens to everything they did and said are we doing things the right way and they they embedded that with what they were doing it was was like the logic of evaluation was kind of embedded into the way they thought about most things that they did and that flowed on then to the work that they were doing into the interactions with the colleagues um so I hope that helps answer your question oh this is very helpful because even in my work I work with five non-profit organizations and I'm seeing that yeah I like to cite your work if it's possible in my yeah of course yeah yeah and please don't hesitate to keep in contact so it would be fun yeah I'll get in touch with you thanks a lot Alison no worries I'm Alison I don't know if you've seen some further comments from Kehiwa and Cara and I just wonder if you want to kind of review those and perhaps give us an indication of what you're doing next oh yeah I wonder if they've got a sort of job for you next yeah oh yeah well that's exactly right I think that field guide for evaluation advocates may be the first point of call to be able to to help identify and and I think you know a lot of the people that book participants in this study didn't identify didn't put their own hand up as champions of evaluation as evaluation advocates they were often quite surprised that evaluators nominated them as such so I think that they're you know this is again a relationship between evaluators and evaluation advocates to really find that connection and it's it's a two-way process so potentially that field guide might be a stepping stone to one help under help evaluators understand when there's so many dot points in the literature that says find an evaluation champion and put them on your team and it's like okay that's a great suggestion well how do I find these evaluation champions and how do I recruit them onto the team and do I just drag them along or do I need to bring them moist fruit cake to make a pleasant environment to make sure that everyone's you know around to be able to do that so that that would really be the first step in in being able to do that but I I agree I think yes that the next step because I did a lot of self-reflection as part of this journey and in conjunction with Alicia McCoy and Leanne Kelly we did a mini self-reflection in a group environment of really reflecting on our roles as internal evaluators in non-profit organisations and thought about well where do people transition from being people who advocate for evaluation into entering the field of evaluation and at what point in the sand line in the sand do you step over from being someone who advocates to someone who's done three or four workshops and is enrolled in a formal training course and suddenly says oh I think I might start to call myself an internal evaluator now even though they might not have that in their job title and even though they might have never been given that label and never thought that they'd end up in that position that's some really interesting place in space there that I'm I'm really sure that there are a lot of people especially in the non-profit sector that have found themselves doing evaluation maybe not even having that evaluation in their title yeah but suddenly a full face with the world so there there's a definitely a lot to explore there and yeah Leanne Kelly and myself had continued on that journey to really look at internal evaluation and some of these interpersonal issues and potentially a publication arising from that next year so you know that we're definitely in that using this as a self-exploratory process as well as thinking about the implications for other internal evaluators and people that are non-evaluators and will never identify as evaluators and that's totally fine too yeah but yes there's lots of angles to look at this thanks can you just see that question that was posted there yeah yeah yeah so this is can you read the question for people who who aren't looking at the chat on the phone yeah I'm gonna let my champ out of the room for two seconds before he barks there we go that was his calling card um so the question says can you talk a bit about more why the participants held the view of not wanting to be constrained by a secular cultural dimension label how many participants held this view and was this about trying to encompass the diversity intersectionality complexities this is a great question and and one really that um I've thought very deeply about because um after asking about their strategies and after asking there was one specific question that really did include so what does how does your practice um work in what do you do in a culturally diverse context to to make this other habit and and um the the negative reaction to that question was quite overwhelming I don't know the exact um numbers but it was very strong from both Aboriginal and Torres Strait Islander participants and from individuals from white Australians as well because they they really um thought that just focusing on cultural diversity wasn't it was too narrow it didn't encapsulate all the ways of diversity that people had to had to consider um and that was the the main reason of like um they said yes we they didn't say that they didn't acknowledge that there was cultural diversity and communication issues but they added in all these extra things yes but we have to worry about um aptitude for technology and I have to worry about who's who's had a tertiary level education and who's only had certificate level education and so these layers of diversity were what that they were thinking about so it it wasn't um that they didn't acknowledge that exist and they they definitely issued there but it was just um too too narrow um but I think that's really interesting that this is the part that I think needs further research because I think when we do think about that in regards to the cultural safety framework um there is that you know there it might be one thing to think about the levels of diversity around who's got computer skills and who doesn't have computer skills but with cultural diversity we really are looking at some social inequalities and power imbalances that go beyond just incorporating other people's worldviews and so um I didn't uh it was only one question it was it was a uh a sub question of a series of a lot of other questions and so it really does need more explore exploration beyond my research but I think we're at the point in time now that um this is a really pertinent question and and definitely worth following up my hunch is that um people who put these evaluation advocates um always willing to draw on different frameworks to see see things from different perspectives and to incorporate these ways of thinking into their work would be ready and open and willing to open up that cultural safety framework and say how does this apply and how does who I reflect on this and how does it mean for my work and what does this mean I don't think anyone would think I'll you know just in that that box and not doing that but I think we're kind of um you know need more information um to to actually work out you know what what does that intersection look like between between those differences and if so many researchers and practitioners and organizations like the AES are calling this out what does that actually mean for people working together in an organization and how does it apply I don't think there's any easy answers other than we need um further exploration there but I'm um we I know we've only got one minute left but if anyone else is is wanting to add to that and to give their opinion then please feel free um to to pick up on that or to make an opinion or to put something in the chat box and I'll stop talking for the last minute um perhaps well people are formulating their their final comments I just want to say thank you Ellison for a wonderful presentation um I really enjoyed this conversation it sounds like most people most other people have as well thank you all for for contributing um I do just just a couple of sort of housekeeping for want of a better word things um because this is an AES Victoria AES webinar you can see based on the name that I've logged in is um we are really keen to hear thoughts on topics um so we we do have some planning for there will be another seminar for this year but we're also planning for next year so if you have topics that you'd really like us to cover in these sessions we really want to know um so please feel free to reach out um to the to the general AES email you can of course always email me as well um and other members of the Victorian committee um for those of you who may have been able to join the session without technically registering um you will need to perhaps contact Ellison or myself to get a copy of her slides because they won't be shared with you on that on that list so if you've made it here without technically registering just let us know and we can share those uh documents with you um oh retus just posted I don't know if you want to unmute yourself retu and share some details for the recording for those people who may be watching this back or not okay um thanks for sharing your your your seminar opportunity retu perhaps share that you asked me something I just said your your post will not be captured by the recording so if you wanted to say something about your webinar please feel free okay uh so I moderate an international online community of practice called gender and evaluation Ruth is a member and uh so we are we organize webinars we'll be happy to also collaborate with you and the latest one is on uh art and creativity in learning and evaluation it's a consultation please join and the the details are given below