 So welcome everyone to the Victorian AES Seminar for November. This is the last seminar for the year as put on by the Victorian Regional Network. And today's session is all about evaluator career pathways. The ambition behind today's seminar is to talk about all things careers, given the evaluation field has somewhat expanded I think in number and diversity of roles over recent years. And there are opportunities for new entrants mid-career and more senior evaluators that are actually quite diverse. And today's session is a chance to hear from a fairly broad panel about their journeys, and also some of the things that they've learned along the way, some of the barriers they've encountered, and some of the lessons that they've taken from those journeys. So it's not a full suite of stories because everyone has their own, but we will have an insight into a few people's journeys through tonight's session. So I'm Charlie Tullick from Policy Performance. I'd like to start tonight's session by acknowledging the traditional owners of the land on which we meet. And I'm on Boon War on Country in Victoria. And acknowledge all the countries that you're all meeting from as well today and pay respects to past, present and emerging leaders. All right, so the PowerPoint slides should be up. And I guess the first thing really to do there is to introduce today's panel because I'll be really driving today's session. I'm purely the facilitator. So the panelists today we have Leigh Ann Maloney from Clear Horizon, Ruth Astin from University of Melbourne, Farida Fleming from ASI, Ellie McDonald from the Department of Premier and Cabinet, Brad Asprey from ARTD and Squirrel Main maybe on the call at this point, or maybe calling in shortly from the Ian Potter Foundation. So we've got a bit of a mix of consultants, government, philanthropic sector and academia on the call. So the idea here is to get a broad picture of experiences and what the differences are between different sectors that you might find yourself working in as an evaluator. So the broad agenda is there's a few questions that we'll talk through as a group. So first of all, a quick introduction to panelists and then talking a little bit about some of the barriers and challenges in the field of evaluation. So some of the tough things that people have encountered during their career and some of the barriers that have been overcome in becoming an evaluator, I suppose. And also we'll look at a day in the life of an evaluator. It might be your current role or it might be a past role and the challenges are just a little bit different based on the setting that you work in. So just contextualizing the differences in things that you might run into and challenges, issues and opportunities and great things about different positions that you've been in as an evaluator. Lessons learned and reflections and then some Q&A. So feel free to jump in with Q&As as we go. We'll stick to a loose structure, but if there are really interesting questions and things you want to learn more about, then feel free to ask. So first thing, let's start meeting our panelists. So the first, well, there's my introduction. I should introduce myself. So I work for policy performance and in the past have worked in other consulting firms. So a large firm, KPMG, medium sized firm, ASIL-LN, or the LN Consulting Group. And as a, in a small evaluation team at HLB Manjad, which is actually an accounting firm. And also about four or five years in government in the central agency there. And have spent some time on evaluative study. So the masters of evaluation and before starting in government on the public policy masters as well. And then a Bachelor of Arts at the very start. So loosely that's a broad journey. Our high squirrel, welcome along. Just launching into some introductions. So yeah, so that's a quick background to me. We won't talk content at this point, but we'll just ask each panelist to introduce their journey briefly. And then we'll get into some of the media discussion. So first of all, Leanne Maloney. Thanks, Charlie. Yeah, I am. So as you can see up there, I'm a director of clearising the principal consultant. I've been a clearising for 10 years and prior to that, a different consultancy and doing similar things, but focusing very much on the sectors of natural resources and agriculture. And nowadays, my focus is very much, I suppose describe myself as a utilisation generalist. I practice evaluation, good practice monitoring and evaluation and bringing evaluative thinking to a very wide range of sectors and different types of programs and strategies and organisations and so forth. So not sort of stuck in one particular sector. You feel like that's not the right word. But yeah, not working just in one sector, although still do a fair bit in that sector, given the importance of the programming that's going on around disaster resilience and that sort of thing these days and climate change. Yeah, I think that's all for me for an introduction. Thank you. Yep, that's excellent. So yeah, life of a consultant, pretty broad in evaluation. So Ruth from University of Melbourne. Great, thanks, Charlie. Yeah, so as you can probably see from that list started in public health research with my background and then sort of fell into evaluation and I thought, wow, this can really make a lot of change here. I'm interested in this. So worked mostly in universities, doing evaluation in universities and health and education for about 8 to 10 years, but I've left it several points to try and experience different sectors. So worked at the OECD for a little bit, worked at a evaluation consulting firm in Melbourne, which is a pick of change and also in not-for-profit at NCRI. And I'm also an honorary fellow still there. So sort of playing in academia and of course consulting and not-for-profit sectors, mostly doing health and education related program evaluation. All right, fantastic. So yeah, broad suite of experience there and you're obviously so young still. So that's great. I mean, it just shows I guess evaluators can travel a lot of different sectors. So Farida. Thanks, Charlie. I guess I came to evaluation with it through an interest in working internationally. So in international development after doing my, and that I guess was driven out of an interest, you know, my family background, a bicultural family background and wanting to work internationally, working first as a teacher and then bringing that interest in education and international relations through to work as an evaluator in international development projects after working for some of the project management companies. And I guess that came out of an interest, you know, working in project management. What you find in international development is that often you're working in projects where the scope's already been developed and it seemed to me that there are only a couple of points in the project cycle in design or then in monitoring situation where you had a chance to connect in with the community and with program implementers to see how things were going and to see if things needed to be changed because oftentimes it might be years after design that you're dealing with implementation and implementation issues. So that that reflective cycle which is similar to teaching is what interested me and maybe want to pursue this field further. Thanks. Okay, fantastic. All right, so next we have Ellie McDonald. So working at the Department of Premier and Cabinet in Victoria. Thanks, Charlie. So my sort of introduction to evaluation came from a background in public policy and research. The first positions that I had or while I was studying, I was really interested in working in the not for profit sector in research and improving public policy. And I was first introduced to evaluation when I was at Oxfam. I was working as an intern and officer there looking at an M&E framework for women's leadership. And I found the concepts that I learned there really, really fascinating and I think really showed the importance of driving evidence-based programming or evidence-based policy and sort of saw a link in my studies as well. And entered into government in the Department of Health and Human Services in a really exciting team that was started up as one of the first sort of internal consultancies to build evaluation capacity in-house in government. And I was exposed to undertaking evaluations, but also capacity-building training and learned a lot about project management and stakeholder relationships in that team. But I suppose also my background, I studied international relations, so I was always really interested in working overseas. So that experience from government in Victoria took me over to Timor-Leste recently. And I was working in monitoring and evaluation in another, in actually the government there in the Secretary of State for Equality and Inclusion. So it was around looking at gender equality programs and developing M&E systems to understand the outcomes of those programs for women and communities. And more recently, I'm in Department of Primary and Cabinet. It's still focused on evaluation, but a lot of it, the focus is on outcomes and how do we support program teams to measure against high level outcomes frameworks for a whole of Victorian government. Okay, yeah. Again, quite a broad, broad experience there in a, you know, several years of an evolving career. And interesting on looking at the list that evaluation was already just a few jobs in and then all of a sudden evaluation came along and things have, I guess, narrowed a little bit in that space. So Brad Asprey is a director at ARTD Consultants and backed by popular demand. So Brad spoke at the November 2019 end of year seminar. So welcome back, Brad. Thanks. Thanks, Charlie. Thanks everyone for coming along. Listen, listen to us be very self-indulgent tonight about our, about our careers. So I think we've got one person left, but you'll see that the pathways into evaluation are all really diverse. And you'll hear most people often use the phrase accidental evaluator, which I think Jean King evaluation theorist first year. So I was probably doing evaluation before I realized I was doing it. And, you know, it was quite lucky to have had a number of mentors who were, who were one of the first people to start doing evaluation in Australia from, you know, from when it sort of started to get off the ground in the 1980s. I came into evaluation via background in criminal justice and criminology. So hence the, I didn't put this slide together. I don't know if I got that listed on my LinkedIn, Charlie Community Corrections Officer Department of Justice. That's a blast from the past. That was a fun job during the, during the Kennett era. Yeah, I could tell you a lot of stories about being a Community Corrections Officer. They're probably more interesting than being an evaluator, but it really saved that for another time. So, yeah, my, most of my work has been as an external evaluator and working up until two and a half years ago in an academic setting. So, a lot of what I've been doing is training and teaching people how to be evaluators while doing evaluations. I guess for my qualifications, yeah, it's pretty, pretty diverse. I've got the evaluation thing there. So that's good. I've got some professional training in evaluation and it says PhD in public health, but it was, it was really around evaluation methodology. So, I guess I'm pretty comfortable calling myself an evaluator, although I don't often say I'm an evaluator because people don't get what that is. Just the other day out on my street, there's a guy walking past wanted to have a chat and asked me what I did. And I said, I stupidly said I'm a program evaluator and that the conversation went strangely from then on in. So, yeah, we live in an interesting space in terms of being able to define our identity. So, we'll probably touch on that tonight. But thanks for the introduction, Charlie, and I think screw your next. Hi all. Yeah, I well and truly feel like a bit of a wild card here, but thanks for listening anyway. Yes, so my first evaluation job ever was a summer internship from uni. I was in the US Department of Education, sorry, New Zealand born American education and accent apologies in advance. And they had me evaluate the goals 2000 Education Act for all of Congress with absolutely no training experience or even sense of what the word evaluation was I think was 17. I did an abominable job, but you know, it was a good deep ending. And then that I decided politics in Washington DC were not for me so I switched had a double major in education became a primary school teacher was very happy with that. I was out in California wanted to train to be a principal this shows my age a bit this is back when applications were tick form tick the boxes was lucky enough got into Stanford but I'm Stanford perspective principles program and policy analysis and evaluation program alphabetically next to each other. And I found myself on day one in a lecture by David Federman. So there I was in the policy analysis and evaluation stream and I went and checked with the registrar and in fact that's what I had signed up for. So, so I had an amazing education as an evaluator, which was sort of maybe opposite different to everyone else but absolutely no intention of being one. I became a lead teacher did a bit of principal things worked at Outward Bound as a kayak instructor after that, picked up a PhD and beginning teacher support, really passionate about education reform. And then finished my contract in New Zealand at Outward Bound School and decided to move to Melbourne was broke unemployed this was after the Christ Church earthquake so basically an earthquake escape and I went and baked my PhD supervisor john had a for a job any job and got one working on a longitudinal study called you for kids, gave me a bit of experience and things like SPSS and all that but again not evaluation. And then from all of that a bit of Federman and a bit of the E for kids study was it was able to then I finished my contract with Unimail picked up another one or two and then got a research officer at Brotherhood. So, I don't know if our paths crossed at the same time or not. There you go. And then, while I was working at the Brotherhood guy named Alberto who was one of our funders just tapped on my shoulder and said hey, you want this random job and it's a long story short but I was taking in pro comedy and thought I'd say yes because that's one of the homework assignments they give you in improv say yes to everything say yes to all offers I was like yeah sure. Anyway, long story short I've been in Potter now for five years working as the research and evaluation manager and. You know, I've certainly been well immersed in evaluation and very glad that I had the academic grounding even though the practical course was quite windy to get there. Sweet. Thanks Charlie for a slide reminding me of what I've done. I'll second that. But they were drawn from LinkedIn so public knowledge and thanks to just being hit by a whole career on a page it's quite full on sometimes when you look at that. So that's just a quick introduction I suppose to the panel who is speaking tonight, but really to get into some of the more meaty content. We want to start to think about what are the some of the challenges that we face as evaluators and knowing the different sectors and context that we work in is probably just a really important feature and factor so I suppose the first question is really about barriers and challenges that you might face as an evaluator and some of the sort of testing times and things that you might need to overcome in in the doing projects or facilitating projects and thinking about whether this field is is for you. And we will then get into after that a bit of a day in the life like what are some of the things that you need to do and that's probably a tough one because a day in the life of an evaluator is a little bit different every day like nothing's no two days are the same for the people in our field. So on the barriers and challenges thing I suppose to get the ball rolling from my perspective as a an evaluation consultant. I think some of the barriers that I've faced have been around sort of quite large projects with quite small teams is probably one of the more challenging times that your faces and evaluator. And that's that that's contextualized across the different settings in which you work so for example if I just again I'll flip back to my slide but working in a small team at HLB man Judd there was really only four people in our team. So there's not many places you can go for help in that scenario. You're a little bit you've got your projects and you're a little bit stuck with them and if you get more than kind of good for the business but bad for the individual because it's it's really heavy workload. And then same sort of scenario at a medium sized firm but with a little bit of extra support and perhaps greater diversity of skills around you. So you may be only one of two or three evaluation people who are focused on that field but you have other capable consultants who can play a role in your evaluation project. So you form these little distinct little teams to do evaluation work. And at the bigger firm like a KPMG the real advantage there was you've just got this great scale so there's always some free reasonable capability to be able to help on your projects for their downside is they probably have never heard the word evaluation little and worked on an evaluation and now they're now part of your team. So you need to quickly upskill someone in what is an evaluation what what are we doing here. But as that sort of evaluation project manager or person in the middle of it. There are challenges either way like this challenges if you're in a tiny organization this challenges if you're in a big organization with generally more projects to manage and run. So I guess that's just one of the things that I kind of look back on over the 10 years. And now I'm a sole evaluator so soul practice so there's other issues there where there isn't the scale but and there's not the support but then there's opportunities to partner. So I think the position that you work from or work in is a real defining feature to me about what your workload is going to be and what your daily life as an evaluator will be as well. So that's just a little bit of a start to get the ball rolling. Do any panelists want to talk to the barriers and challenges question specifically. Maybe just raise your hand and I'll take comments from the group. Who wants to get the ball rolling on that one. What are some of the challenges you faced so Leanne and then squirrel just on mute the Leanne. Classic. Thanks Charlie. I was just listening to what you said and what you said in terms of where I've worked has been very similar. I guess one way of thinking about it from my perspective is the way we come into evaluation. And I think you know you think about a teacher or a lawyer or a doctor you know you do your undergraduate and sometimes for many many years and then you pop out and you're a lawyer or a doctor or a teacher and off you go and you start. You get better and better over time and more experience but you start kind of on day one doing what you plan to do and evaluations really different. So the last time I looked there wasn't an undergraduate course and I can't imagine someone popping out after doing a three year undergraduate straight out of school and having some any legitimacy or credibility as an evaluator because I think a lot of a lot of what you bring to evaluation is is your life experience and your maturity. And so we my experience and I think it's the experience of quite a few people has been learning about evaluation on the job and then you know sometimes even decades after working actually going and thinking about possibly getting a professional qualification evaluation or not because you just brought up a bag of experience over the years. And of course that's changing now but how we come into evaluation is not usually straight through an undergraduate. It's a very, very different experience and we usually start off in the sector learning about evaluation in the sector. And then possibly saying in that sector or not moving on and becoming an evaluator who is possibly sector agnostic which is pretty much, you know, who we are. So squirrel. I guess I'm, I think, I agree with Leigh and that that experience factors some of some something to consider and I strongly encourage anyone wavering on it. Go do something else for a bit like whatever you're calling is teaching environmental engineering, and then pick up and swing back into the evaluation and it'll give you just a little bit more street cred, which is a sad thing to say like, you know, does a public school go and be an environmental engineer to get street cred as a school teacher. No, but it's different. It's not yet a professionalized industry. It is a cottage industry with a lot of small businesses that is aiming towards professionalization and you know the deliberating through, you know, do we want level 123 or four, you know, to what degree of professionalization. Are we aiming for it's brilliant conversations to be have so it's not nascent, you know, there's something in this profession that is congealing. But at the same time, it's really tricky to just jump into it if you've done nothing else. And I would say I'm speaking from the other end and this is going to sound awful so I'm sorry, but in my job probably I read about an evaluation today. And I've done so for about five years so let's just ballpark a thousand, let's say I've had some days off. There are some really poor quality evaluations out there and one of the challenges I find it's really hard to tell someone that they've done a poor quality evaluation. When there's not necessarily standards of comparison so I can think like oh you know, out of the last 200 arts evaluations I've read this one doesn't go so well as the others. A lot of times it's people with less experience in the field, or, or, you know, are newer on so I would encourage people to get a mentor, or somebody to really overcome that that the rather than seeing it as a barrier. The question would be if you have less experience really try to align yourself with an evaluator that you believe in and see what kind of mentorship arrangement you could possibly get from them, because I think a lot of people might think they're doing good evaluations and they're actually, they're not. And it's really hard to say that to someone's face it's nice and easy enough vaguely here speaking abstractly. And that's the challenge that I face and so I'm very much in favor of the professionalization of the industry and the increase in standards so that quite frankly as a consumer. I'm getting good product to read at the end. Don't know if other folks want to speak more to that. I think there's been quite a few interesting points there and just to branch off in a slightly different direction I mean I think I found the challenges, you know, over a couple of years can sometimes come at that last period where where you're negotiating with clients around. So you might have had initial findings and agreed on those done presentations and likely and I take a utilization focus so it's got to be useful for people and engage them in processes all the way through that's my approach. But even so I think it's important to recognize the different stakes that people have an evaluation, and that sometimes so so squirrel just to take the other perspective sometimes, sometimes clients or program staff have their own professional reputation and careers at stake as part of an evaluation process. And so, you know, what's important to understand is therefore sometimes people's perspective about what's important, you know, if some people are more interested in what's working and what can be built on especially if their careers and a part of that and if others are focused more on what can be improved you know that can be an issue of contest you know. So I think here, I think the profession so I think just to distinguish between kind of professionalization and a credit credentialism you know like I think we're not yet a credential then in fact as hasn't decided to take that path but definitely trying to be professional but as you say squirrel you know you need that body of knowledge to be able to compare oneself and know where one stands. But still I think within the profession there are things that can help us like the competency standards you know, and I think one of the areas that there's competency standards helps us look at is, is this thing about understanding the context and understanding the power and discussion negotiations of power part of that. And that with that comes some difficulty sometimes and we have to work our way through that. And if that Brad I think I remember one time I listened to you a couple of years ago and I remember you saying look, it's important as an evaluated to think about one's position whether one is only client focused or thinks of oneself as, as you know, as professionally an evaluator and, and it's incumbent then to be to think about one's own integrity and position you know so I think I found that very helpful. So, so I'll throw that into the mix of challenges that that I found and how I've tried to negotiate my way through it. I think it's a second wave for me then maybe maybe you Ali. I mean preparing just I jotted down a few challenges which some relate to some of the things we've been talking about in terms of the concept of evaluation careers but I think the brief was to talk about barriers and challenges generally not necessarily about how you come into the field. I listed 1234567 random things that came off the top of my head that all have a really long story behind that about half a bottle of wine for each of them. So I'll just read them out and that it could, you know, just throw some ideas out there so when I first I came into evaluation via, you know, research often where I will be given funding to do something and we didn't have a client. So it was it was a research grant so when I first started doing evaluation and there was someone who was paying you to do the job then you become like a tradesman you're a plumber you know you're a you're a high gun in some senses and I found that quite confronting. And the first one was probably a bit of a baptism of fire it wasn't it was very challenging and it just made me start wondering is it all just politics and I'm still wondering that. The second thing that links to that and what Frida had mentioned was this question around utility and validity so out to meta standards out of out of another three being useful and being valid or being credible being accurate and being technically rigorous whatever that means in terms of your paradigm. And I've often kind of framed that as as attention and kind of sum it up sometimes as saying you know getting it right versus getting along like I don't expect to make friends all the time when I'm doing an evaluation. So I think that's important to keep in mind as well and gels with the politics side of it. You'll probably see an ongoing thing throughout some of my quotes the second challenge of the third challenge is I've noticed this increasing push towards the sound by glossy infographic evaluation. And maybe I'm seeing more of that now because I'm, you know, more interested in what competitors are doing because I'm working in a commercial space as opposed to an academic space. And I just I want to say to people that I hope against all possible hope but there's still a place for the thoughtful considered report that stands the test of time and can contribute to change over the longer term. I think in the post truth truth and thankfully post Trump world now we're still confronting problems around the devaluing of evidence and that I can see that more and more and evaluations that I'm doing. And, you know, we're, if anyone's kind of interested in this notion of, you know, society fragmented a little bit, then there's a recent article from someone on my huge fan of any house who's written an article on evaluation of fragmented society that's just published in the journal multidisciplinary evaluation. And so we're seeing the context in which we're doing evaluation changing quite a lot and that impacts on us when, you know, we try to follow, you know, good practice in terms of methods but we're competing against this sort of post truth anti science sometimes mentality. I think I've seen in the 20 years I've been doing evaluation increasing democratization of evaluation, and I'm a huge fan of that to a point. Because it does address that elitism of the past so in the 60s when people started doing evaluations was a very elitist profession and you have a lot of academics crawling out of ivory towers to evaluate war on poverty programs. But I'll just caution that a low everyone can do evaluation not everyone can do evaluation well so like school I am a fan of increasing the push towards professionalization, because anyone can put up a shingle and call themselves an evaluator. The sixth one was something I was very passionate about which is the lack of appreciation for the knowledge and theory base of the discipline. And so while I'm, I think Leanne's point about learning on the job is really important I think there also needs to be that feedback loop and it doesn't matter whether you start as a practitioner and because we are a very practice based profession, and then connect back with the theory and then go into the practice. It's the constant. It's like a yin and a yang and they should be interacting throughout your career the whole time. And it doesn't matter which point you come into it. And the last one the last challenge is, I've labeled it turf wars and status anxiety. The turf wars bit is it's becoming increasingly acutely aware to me that that we still suffer from the fact that there's a lot of other people who are doing stuff like what we do and getting jobs. Yet they know nothing about the field of evaluation that's because the people who are buying the products don't either. You know the rise of behavioral insights and data science and data analytics and it just worries me a little bit because we can't position ourselves as offering a unique thing above and beyond those professions which are complimentary but they're not evaluation. So it's just maybe that stimulates some other discussion from from you Ali or Bruce. Yeah, I don't know if you want to share something Ali I was just going to take on that I that was one of my challenges actually. Brad I realized how much of the work that I do when I was reflecting back on it today is actually about advocacy for the discipline. And I'm not sure if that's because I work in the head make space but it was true when I was in consulting as well like what is evaluation and then why are we not research. Are we research or are we evaluation so I think that was for me earlier when I began an evaluation that was particularly hard to do. It's easier as I've got an order and I must admit squirrel and Leanne your comments about the evaluator being seen as someone who can draw on life experience and have experience. I definitely I definitely felt that when I began doing an evaluation which I was always as part of a team but I think I don't know if there are many people on the call I've seen a lot of students names here. So some of you maybe maybe you and perhaps beginning your careers in evaluation. I think that that's still a challenge for a younger or early career evaluated to be able to communicate one skill. One disciplinary skill is very important because when I began a lot of people were much more interested in hearing about subject matter expertise like OK so what can you contribute here and perhaps not so interested in a critical thinking skills evaluative thinking skills but I have seen that change immensely in the last couple of years and much more appreciation for capacity building and and I don't know Brad if that's getting confused and other people coming into the field of evaluation more and more. But I do think it's made it a little bit easier to to communicate what we can bring to the table particularly as external evaluators. The other the other thing I just wanted to bring to the fore come in I'm supposed to be I think the academic evaluator panelist. So I just wanted to bring to the fore one one barrier that might be more appropriate to someone working in an academic setting. So all of the time when when we're asked to do an external evaluation we're called an academic evaluator and I remember asking a consultant what on earth do you mean by that. And as Brad was sort of intimating what the common definition that I've experienced is someone who has great command of theory and and kind of knowledge foundations of the discipline and what I would urge everybody and I'm sure other panelists would agree is that they don't have anything to do with academia anyone can can hold that that background. So there's just a perspective I wanted to share and it's something I try and not dispelled but communicate when I'm asked that question because I think it doesn't have to be as Brad sort of intimated the ivory tower theory doesn't have to really sit within that that setting I think there have been so many advances by evaluation practitioners and contributions to theory and I can see that growing and growing and growing. So that's something that I try and communicate regularly. But as somebody who does work in an academic setting often I am asked to comment on a theoretical integrity of an approach or validate someone's approach someone asked me to do that today can you validate my evaluation framework. So that's, that's a lot of the jobs that we get which can be challenging for a person new to the evaluation to have the confidence to be able to say, you know, this is a good approach or not and be able to back it up. So yeah that's that sounds a lot like, yeah, Brad's, Brad was mentioning about the validity of research designs and being the critique of that and yeah and consulting space it's a bit the same as the evaluator in a big organization. Every time there's a project that's called an evaluation, you're involved and you need to help shape this robust and rigorous and reliable consulting method which is, on the one hand, it's a great opportunity to have an involvement but does add a little bit of an extra burden in a trying to get the projects done and budget them appropriately rather than going down the line on every component that you'd love to have in every evaluation but may not be feasible. So yeah, then on the flip side coming up with good methods that will also win work and keep the business solvent. So, Ellie, any comments about your challenges and barriers and things that you've encountered throughout your time in evaluation. Thanks, Charlie. And I just wanted to touch on Ruth's point. I think when I was in working in DHHS, which was a few years ago now but there was a positive, positive and very strong focus on theory and bringing the theory into evaluation so that is I think slowly moving into government. It's definitely not perfect and probably not at the standards of Melbourne University but it's very much a focus I think it's seen as important within internal evaluation in government. But I suppose just to also respond to the conversation around early evaluators. I myself am an early evaluator and I think I wouldn't discourage people who are coming out of their studies and interested in this field to not move into the area. There's a lot of places within government. I think there's a huge investment from different secretaries across different departments as well as Daniel Andrews to really push evidence-based policy and evaluation. So within a lot of the line departments like DHHS, Department of Education, Department of Justice, I think all of them actually have an internal evaluation team with varying investments but I know Department of Education is really building their evaluation team and starting up next year. So I think from my perspective being in a team and being surrounded by really experienced technical experts was a really valuable experience for me learning the ropes and seeing as Farida mentioned there's a whole range of different components of evaluation from stakeholder engagement to project management to qualitative and quantitative data analysis and methodology and design. There's just so many different aspects of it I think and no one can be great at all of them and I think being in a really diverse and dynamic team can help you find your place in the field and learn the ropes and make sure that there is that quality because you've got that experience behind you from your team. But in terms of challenges I wanted to be I wanted to just touch on my experience in teamwork. These are two just really, really different challenges that I think are still relevant in Australia but in international development. Some of the really basic things like bringing stakeholders bringing the right stakeholders together can be a real challenge really identifying them but also in terms of logistics a lot of the stakeholders are five hours drive away or you know it almost a day sometimes away. So how do we engage with them where there's limited internet connection and email capacity. And I suppose that comes into the ability to do data collection and no no data is ever perfect so I guess evaluation you really want to find a range of different data sources but when that's not feasible. I think that's a real challenge in the international development sector to really find the right approach and also the right approach for for the community and and the audience that you're trying to tailor that evaluation to. And I think also within government within the Victorian government that's a challenge. When writing reports there's a range of different audiences that want to hear a different story and they've all got investment in this evaluation they're contributing in some way. So how do you deliver a report that will satisfy a range of different audiences. And lastly the last one I had down was just the constant turn and very quick cycle of political decision making and election commitments and investment and often in government while there's really great work being done within evaluation and and trying to feed that back through recommendations to support evidence based policy. A lot of political decisions are made before the evaluation is ready or just not within the time frame so I suppose there wasn't a lot of discussion during the recent AES conference online about the sort of me from all quicker evaluation. And there's sort of I suppose a few different ways to get around it but that also compromises the quality so yeah I think they're just a range of different challenges I've seen in my early career. Alright fantastic does that sparked any other thoughts in panellists any of the broad discussion there so early speakers who wanted to pick up on anything or if not we'll jump on to a day in a life of an evaluator. So yeah we were hoping that on this call there might be some people who are either thinking about moving into evaluation or thinking well I've got all these skills that I've developed through my personal interests and study background and past jobs and things. Like if I was to do evaluation where should I start should I be looking at government jobs should I be looking at philanthropic should I be looking at international development should I be trying to get into consulting like where do I where is the best place to for me and I think what we're hearing here is that we've all gone through kind of gone into areas of evaluation that that interest us but as much from our personal personal interests as our career interest so evaluations not only a job that we turn up to and do usually the sorts of jobs that we're interested in doing relate to stuff that we've done at some level before or at least we can kind of understand and apply our minds to as evaluators across the discipline. But yeah really the question about the day in a life does it change based on the different organizations you've been in does it is it fundamentally different to the day that you'll work in academia versus the day that you work in consulting maybe a question there for Brad. I'm interested in a day in team or less day versus a day in DPC it's probably drastically different like what what sort of how does a position or our job in evaluation change what we have to do each day and the way that we do it. I think a big ones whether you're internal or external I know that they're blurry boundaries with those roles but that I've been an internal evaluator fortunately only once. I find it much easier to criticize from the outside and criticize from the inside when your job's on the line so I think that can change things quite a lot. The setting can change things quite a lot to like I think yeah if you're working in remote locations and yeah it can change like Ali was describing. So I think international development evaluations really really challenging because you've got language travel security risk small budget small timeframes like the environment and the commissioning approach can change it quite a lot like how much like what's going on. So we're all like evaluation is really interesting because it's much harder than research because we've always got time budget data political and other constraints so we have to manage the best we can and do something that's good enough in that context so I think the setting does dramatically affect particularly the role I think. And I'd add to that I think seniority plays a difference in what you're doing over time so when I started I was focused on data collection you know and then I passed the data to someone else you know and then as I advanced then I was doing more of the analysis and then and then getting into developing designs or M&E frameworks doing evaluation capacity building which I really enjoy you know so I think that's part of difference as well time and what you're asked to do as a function of your your experience. It really resonated me for I know it's you know these these discrete tasks that you know someone can take but they're not expected to have or understand the methodology from beginning to end and if you're not really careful about that especially and I imagine it's not just in consulting either. When you're growing people in their roles they can get really really good at these discrete tasks but not have an understanding of a whole. And that's a really obviously really really important thing for people to step in and lead their own projects. I had a different experience even though that's been my that's been my experience of watching others come into the organization. My experience was different in that I was given the a nice little cookbook on how to do a performance story report and training in it. And then there was the strength I was running a bunch of trial and it was paying a bunch of consultancies to run a lot of performance story reports across Australia under the National Heritage Trust back in 2006 2007. And so I got this little manual and I'm really good at following instructions and like cool and so I did my evaluation and the client was happy with it and there's a lot of support attached to a lot of guidance. And I believed that was what evaluation was. Evaluation was I had no idea that what I done was one form of evaluation that are the forms and I took me about two years to realize that this thing that I thought I've got this now I've never been more confident in my life to do an evaluation than I was my first one. And that maturity and growing from doing these bits to actually doing the holes is one of the challenges I suppose as part of your natural progression as well. It's something to watch out for. And so squirrel I think it sounds like your day in the life is really quite different to many evaluators so reading a new evaluation a day is a little bit of a luxury probably for some. I've got the dream job man I'm just not that everyone else's job isn't great but I just like let me just say. I'm on such a good weekend I keep pinching myself one day it's going to end but until then. Okay, so we have no loosely 240 grantees. My role is, is, you know, threefold past present future but I, I both work with the current grantees but I also and, you know, with as an external evaluator as a consumer as bad cop as whatever you want to call it the monitoring beast, which sounds awful but it's led me to like canoeing in the Tarkine or working with you that risking cans. So, enough evaluation to co and let me back it up and explain it in the database about 80 of our grantees we decide might need help with evaluation and that's across our fields of arts environment, community well being education. And so what I get to do is meet one on one with those grantees and again that's the canoeing in the Tarkine youth at risking cans like it's just, it's fun, I used to get to travel now I get to zoom a lot of zooming. But it's cool because I've met most of the people in person already. I am not doing the evaluations but helping them. Maybe it's commission and evaluator, maybe it's making an evaluation plan, all the things that LA was talking about those different components qualitative quantitative evaluation planning stakeholder engagement a lot of times nonprofits forget about stakeholder engagement they're so busy trying to prove they tick the boxes. They actually forget to bring in those end game stakeholders so it's gentle reminders about that and match making them with evaluators who will be able to help them down the track. Um, yeah so it's a quick tender game for me. Unfortunately, it is a bit FIFO as in I literally fly in and fly out and have a carbon footprint that would make anybody cry, considering I work at a nonprofit that funds evaluation, I mean environment. I think it does a lot of good though because I've often thought about it. Um, I, there's, there's two or three things. I have a lot of notes down here but I also but the flip side is, I get to present to our board and think about our grant making and encourage us to do multi year grants or grants that are over 100,000 or whatever it is that's going to flip the switch so that we are doing better grant making. So I'm using the data from the grantees I need them to do good evaluation so that I can then feed that back to our board and make good decisions about what we're doing in, in the various sectors that we work in. Advocacy evaluations becoming a thing so having the up skill on that. The philanthropic sector is keen on evaluation but rather. It's honestly, this is going to be a job sector, people like a lot of philanthropics and foundations are going to start hiring evaluators. Um, so and so I am meeting with my peers so it used to be me alone in a room talking to myself I did that about twice and then one more. So now there's about seven other people with my role, and it's becoming more and more popular so it's, it's kind of cool because that means I don't have to talk to myself anymore. But it's kind of daunting because people will ask me questions and suddenly I need to have answers. And that's where I rely a lot on people like I'm not afraid to be like Charlie red help me and clear horizons what are you guys doing because so and so is asking and I don't want to, I don't want to misrepresent. Yeah, I think there's, there's a bit of that, which is pretty fun. I do have to read the evaluations. I do have to give feedback. I do have to have really careful conversations with people when evaluations aren't good enough. That's probably the most challenging part of my job. I once read an evaluation where somebody plagiarized data from another cohort of you so it's about at rescue. I ripped the data from another study rounded it to the nearest five, ran the analysis on that and put it in as their own, not even the kids, I mean these are at risk kids and this is where I get passionate like do that with the kids from Jelon grammar but don't you dare do that with that risk use and when I see evaluators doing bad evaluations or mucking up or even worse in this case, deliberate dishonesty in an evaluation. And it's involving, you know, people with disadvantage, I get furious. Yeah, so that's part of my job is trying to contain fury and convey it well. That's probably been more than my four minutes but that's a good idea. So the gatekeeper of good evaluation in the philanthropic sector so this is what we need to look out for and setting your own standards or setting standards which I think is really pushing, pushing higher and higher but also having fun, not losing touch so having those meeting with the grantees for a day and seeing what they really need and what's happening so that I'm not becoming some weird ivory tower person. So like I can see like Brad, I'm not calling you a weird ivory tower person Brad. It's as a bad subjective clause, but a lot of them are saying you know, my government stakeholder wants a two page infographic or you know my in game stakeholders so by all means, I encourage you to do the research that Brad's talking about and have that rigor behind your methodology, but be able to code switch and if you don't have infographic skills I recommend trying to get a rudimentary level of it so that you can have a clear and well pitched presentation, if and when needed, and being able to code switch between the two, and I'll be quiet now. It sounded like a segue to Brad again, but yeah I do and I also hear from Ellie about what life was like in team or less day and walking in as an evaluator. I'm not sure whether you're the only one or there are others, but Brad did you want to add before we No I don't think we've got time, but I wrote a diary entry for today but we'll save that for another. I don't write the diary entries this is my first chance to do it so I won't read it out but basically it was I had deal with screaming child in the background deal with poor data that was just part of the entry for today. Right that's the day in the life. Yeah deal with screaming child deal with poor data. In that order presumably. Yeah. Ellie. Yeah thanks Charlie and trying to think of the day in the life of an evaluator for team or less day. I think I think the within the government the government of team or less day has very very limited resources and one thing I realized going there was that the focus was very much more on monitoring and setting up monitoring systems and monitoring finding finding data that would be of quality to show a progress towards outcomes. From a monitoring perspective because there just wasn't the money to to bring in a lot of resources it was very costly to go out to municipalities. So we were trying to make the most of at field offices within government and to support them to develop data collection plans and survey tools to do on paper in paper form because of the limited internet access. So it was incredibly incredibly basic which I found really refreshing I'm again I'm a new evaluator so I came from Department of Health and Human Services which was focused a lot on different design methodologies different theories which really helped me and build my understanding but going back to basics and really starting from the beginning was something that I found I found really valuable and and I think you can you can definitely do a good job of still finding finding impact in programs by going out and talking to communities and collecting stories from women and different community members about what a government program has has how it sort of changed their their financial situation or the impact of their family for example. So I think the day in the life in Timor was very incredibly slow constantly problem solving constantly. It's very much more focused on relationship building and really understanding what would be meaningful to the people that you're working with. What is meaningful for them to understand the impact of this program so that they can maybe adapt or deliver recommendations to those higher up to increase funding or redesign a program but very yeah I suppose very basic one example is that to bring stakeholders together it was required to basically write write a letter in in their local language Titan print those out and find all the different names of the different ministries which change very, very often, and they were hand delivered that they first needed to be signed by my director general so it could take up to a week to get invitations out to bring stakeholders together. So very, very slow moving but I think incredibly rewarding to really see here the stories of how how government programs are impacting the community but also how they can be improved. I think all of this makes me think that I mean ultimately we are all the spokespeople for evaluation in our own little in our own little worlds that we are we are all doing evaluation capability building of some sort and advocating for evaluation I guess having done you know being involved in evaluations for years we can all see the value of a good evaluation we know that they're really insightful useful important and have that change focus and can create change in organizations a good evaluation can can do that so we can all see the positives but I suppose part of our all of our roles really no matter what our setting is is about advocating for evaluation. Ruth I don't think you had a chance to speak yet about the day in academia and your role. Yeah sure I suspected somewhat similar. I also Brad was looking at outlook insights I don't know if anyone's looked at that but you can get you know data about how you spend your day and my collaboration time relative to quiet workers quite alarming actually but regardless of that I think what might be of interest to people on on the session tonight is so academic doesn't matter what discipline you work in and really have to do four things you have to do research on your discipline so I have to find creative ways and a lot of my day is spent trying to do this fine creative ways of finding how I can do research on evaluations that we're currently doing because I don't believe we have yet got grants is who are doing giving up money for doing research on evaluation but we're working on that as part of advocacy. So that's that's a big part of what I do so do evaluation and do research on it teach in the master's program which a lot of people talk about and supervise evaluation students and then a subject that I think again I don't think it's unique to academia but is a leadership and it's a service to the discipline and leadership of the discipline so being able to talk tonight and and to work with the committee is a component of that but also part of Melbourne uni and most other universities you really have to create international linkages is a huge push to try and create linkages to clean evaluation because CPE with the only academic center of evaluation in Australasia. So we were trying to build up a larger and larger and larger group of people who do evaluation academia but also in consultancy who might be interested in doing research on evaluation. So that's a big part of my job as well. And it's also some project work. Does that fit into the research bucket or like the. Yep. Okay. Yeah, yep. So that's the hardest bit to communicate and I'm sure Brad has got many stories about this and we're all you might do to actually with your work but communicating that evaluation practice. Meets the standard of what might be considered doing research at a university. So yeah, majority majority of academic value as I believe in 30 that you might want to comment to most of us are funded on evaluation projects. So you kind of have to make the research work in that so 80% funded on soft money, which means you just continually looking for the next project and trying to find a common through line for your research which is quite challenging and at the same time trying to select projects that might support what you're interested in but also pay for your job. Certainly appreciate a pressure that's a little bit unique to that setting that others may not not not for much longer with all the job cuts in the university sector. Everyone will have to have to do what I value the people who work in it. I mean there's a lot of academics who do stuff that they don't call evaluation but it's the same. You know, like, you know, they're working on applied work, you know, they're getting commercial research category, you know, category three and four grants which are, you know, have a strong availability of focus. They might not call it that but yeah, it is a juggle. Bruce being very modest is one of the hardest practicing evaluators as well. So, like, just, I think probably potentially currently doing more projects than I am while teaching and all that kind of stuff. So, yeah, being in it. That's why the term academic evaluator doesn't necessarily mean a lot because it's not like you're, you know, just sitting there with a, you know, patches on and a pipe wondering the library reading interesting random books. Yeah, about evaluation. Yeah, exactly. Really quite applied. All right, excellent. So the last block of questions and we can take some Q&As as well. So start thinking of any questions that are on your mind but really that the last element is around lessons and reflections about career options and development. So I think for me and what I'm just seeing as an evaluator is there are more roles in evaluation now that I guess they've ever been, which I think is a great thing for all of us on the call and our profession that there is variety like there's choice. There's some degree of choice and there are new roles being created. And so it's quite a good time I think to be, you know, thinking of different options and seeing what's out there and maybe thinking of new challenges or whatever it might be. But I suppose the observation from my perspective is that there are a whole range of choices and it was a relatively narrow field I think when I got into it about 10 years ago but seeing all of the government agencies really driving new positions a lot of not for profits. Establishing internal evaluation positions. We're hearing that the philanthropic sector is going that way. I mean part of this might be market driven where, you know, we're sick of paying the whatever fee for the consultants to come in and tell us stuff that we already knew. We can do this ourselves and capture that knowledge and build capability at the same time. So I think that sort of shift towards internal evaluation positions is something that I'm really noticing quite a lot at the moment. So I guess that's a good thing so for anyone thinking about is this the field for me I'd say give it a go and the skills that you'll learn in evaluation you can apply to different roles, regardless so you'll learn good good planning skills project planning skills you'll learn consultation skills how to speak to almost anyone how to collect and interpret and understand qualitative and quantitative datasets how to write compelling reports. So the skills that you'll develop as an evaluator are transferable. And a lot of you can take that anywhere I guess you can take that into into other sectors would be one of my messages so you'll feel like there's a lot to learn because there's a lot of components that could be thrown into any given evaluation role, but you will take from that things that you can apply in other settings. Any other comments there or observations about reflections about options and development and where people might go in the evaluation world. Random trend and that's a random but a trend I'm noticing. Like with everything else in the black lives matters movement but just push towards equity evaluation, and I think just in general diversity. I would encourage folks to not like to think of yourself and how you bring a diverse perspective, but to also think of the whole field so you might be thinking oh gosh what should I do when you know when you land your first job how can I do this but look around you and who else can you help. Who else what other person is starting with you in that cohort that where you can be helping build their skills and capacity because they're bringing a diverse perspective but maybe they're not maybe English isn't their first language so they're not typing it as fluently as you are or maybe mathematical statistics skills aren't up to yours you know you don't have English is the first language but you have excellent computation and modeling skills. Help everybody it's a village it takes a village so I really want to encourage you all as a cohort like in drinks afterwards and as you're going through whatever training program or whatever. God awful working hazing period you have at price water health Coopers or whatever else you do like. Like, you know work with each other but don't just think of yourself and am I good enough and how can I fit and how late can I work tonight but how can I help the sector the diversity the equity and build the capabilities all around because as a sector we need to move up but I also think we need to make sure that there are more diverse perspectives and it's not just diverse perspectives is tokenism on a particular project but there's a real wealth of knowledge and when you look at. When you look at traditional academics and you can take utilitarianism and you can take you know you, whether it's a featherman or pardon or whomever else you're looking at, it's a lot of older white men. So what else can we bring to that tree of knowledge and grow it so that in 100 years time, there's more. The forest has biodiversity and thanks. Yeah, just build on what Skrill says there about I think, you know, as an evaluator you also have to kind of determine your own principles and what you're working for and I think, you know, it, for me it's not a technical thing it's a, you know, it has to have a commitment to it to I think that's one of the things that makes our work wonderful is that if you have a commitment to social justice then you can work on programs that are important to you and that you can hopefully make a difference to you know so I think that's really important to know where you stand and what you're doing it for you know and the other thing is, you know the point you make a squirrel is it takes a village I think there also is a village you know so I think we have to recognize that we're not in this alone. I think society is great you know the journals are great don't just be a member of the Australian professionals association join the American one you know get the journals, go to the professional learning, read the theorists you know all of this just helps us and and to the girls point as well look at other people's work you know see what's out there see where you're good and where you really need to improve you know just take the coffee with people go to have a chat because it helps all of us you know it really makes this all better and yeah yeah. Now that's a good good point about getting to the seminars I remember I think when I before I was really an evaluator I got dragged along to an AES seminar and the speaker was this this academic guy who spoke about all these theorists from the past and talked in terms of all of their views. It was Brad doing the talking and it was completely over my head. And I thought this is a field I will never understand I mean this is so technical technical complicated and historic and views coming from everywhere. How could you possibly understand this field I thought this is not this is not for me but I guess it's a slow build and like. I guess if I had a chat with Brad now I'd probably get three quarters of what are you saying but about the theorists in the background but it's a little bit of a building blocks approach where it all starts to make more sense. The more the more that you do I suppose so I kind of encourage that stick out at kind of thing so show the grit learn just little bits as you go some on the job some theory some reading maybe some studies and seminars. Checking out websites etc at all at all helps. I've got just practical advice that I wish to some extent I wish I followed it to another extent I'm glad I didn't because I like how I approach evaluation which is very pragmatic and pluralist but I've seen people who have done well in their career and they've developed a niche. I'm thinking particularly for those it doesn't matter whether you're in academia or consulting but they've. I'm not talking necessarily about inventing an evaluation approach like David Fenerman and being you know some old white guru on a tree who talks about empowerment evaluation and hasn't really had a disadvantage background which is quite ironic but you know the the the idea of you know specializing in something given that we're a bit of a jack of all trades master of none can sometimes be good for your career. So rather than trying to be all things and other disciplines specialized we just haven't really got those you know within psychology there's clinical there's community there's you know within engineering there's civil there's mechanical there's we don't really have that distinctness yet we have people who rally around the evaluation and we don't we don't have these distinct things so you know I could foresee situations where people would become specialists in particular sectors and there are people who do that and so I do a lot of mental health evaluation and they start picking up and snowballing a lot of jobs in that space. There are those who might get well known for using a particular approach. And so they become very well known for being, you know, realist evaluators or empowerment evaluators or something something else. So that can be effective. I'm sure that all of you know we're talking about people who might we don't know who's in the room will hear from people soon but I think the hardest thing is just. You know, getting the opportunity. I've, I think nearly every in the last 20 years nearly every job I've applied for I haven't gotten and all the jobs I've had they've kind of fell into my hands. So it's really tricky because yeah I've just I've written thousands like not thousands but hundreds and hundreds of. You know, applications over my career even though I've had long stints at different places, but it's been often just the right place right time so it's it's hard to plan for that serendipity. Absolutely. Does any of the other panelists have a comment on options development opportunities. Yeah, I wanted to branch off what Brad was saying I think, as well as specializing in different evaluation methodologies. One, one area that can be useful is to sort of specialize in a particular subject matter area. Often, even though I think our skills are to be able to transfer knowledge and learn subject matter expert, sorry subject matter areas. Quite quickly, it can be valuable to, for example, I've sort of focused a lot on gender equality and family violence and out of home care and child protection and those sorts of areas are all interlinked but there is a lot of work in the gender equality area and I think a lot of different areas have a gender component so that is a valuable area and I think there's a lot of investment in that area in government, but other areas as well such as climate change and yeah, any area really that could be that can kind of maybe distinguish you a little bit more, as well as I would suggest engaging in a whole lot of different evaluations so putting up your hand to engage in a really quantitative focused evaluation that's not my, my background is more qualitative but I learned a huge amount having to do quantitative evaluations in DHHS and that really helped me in my work in Teemo. I would also suggest maybe moving across sectors so working in the not for profit understanding that sector as well as government and maybe the private sector I think that can really help you develop a big picture of different audience needs as well as different ways the sectors do approach evaluation and what else did I have? I think one thing I suppose also goes to Brad's point around the challenge of getting work but then you're sort of tapped on the shoulder or you get work that sort of falls in your lap through maybe networking and I think that there is such a value in networking and not in sort of a way that is sort of sleazy or over the top or because I know that term is sort of a bit strange but but I think developing networks and really helps you learn from others about what they're doing and what the challenges have been but also can really connect you to people that you might see again five years down the track and there might be a job going with them and because you sort of can get that they know your background or they've seen work that you've done it can be really valuable in in your career I think. I see you nodding there Ruth. Yeah, I was just going to say I totally agree and if I could offer one very practical suggestion. When you look at evaluation jobs, be open in your searching for different roles. So like, often I've seen lots and lots and lots of position descriptions, which to me is unlike all that evaluation but it's called an audit officer or a program manager or I've seen so many different role titles for what could be an evaluation position that looks like one. So I know that's very practical suggestion but but when you're searching for jobs, sometimes there can be lots and lots and more out there, but they're just not called evaluation. So, so try and be targeted and open to different role names as well. And on the flip side look out for the jobs that are advertisers evaluation that aren't really. All right, we happy to take any questions from participants or listeners. I'm sure there's things that have sprung to mind in the just fairly broad ranging discussion about the stories from a few different people and there may be sectors that haven't been represented today that we should maybe reflect on a little bit but are there any any questions from the floor feel free to unmute yourself and ask the panel or the chat box we can do that. There's a chat box as well we don't have any questions coming through yet but feel free to share any questions. And in the meantime, Leanne did you have any comments about opportunities or areas that evaluators can develop their practice I mean is it. I suppose this is a comment from me but a lot of young evaluators or new evaluators go to clear horizon as a great place to learn the basics of evaluation and be engaged in a whole range of projects and then on to other things. In some cases, there are any questions on that pathway. We try not to think of it as a retention problem and think of it as we've grown some people and now they're going to go out and do good work wherever they land. I think a really good opportunity and I remember you mentioned at the beginning that second months are great, I think, because, yeah, especially if you haven't had much experience, for example, in a program managing a program, or being inside of government but you'll see on the outside consulting to government on evaluations, it's good to have some of that experience. So, you know, we've had, you know, people at clear horizon, you know, from, obviously not at a junior level but at a senior or principal level, being suspended into, you know, Vigo or DHHS to do particular things for particular, you know, founded things for a period of time. That's a really good, I think, mix of opportunity to think about the. Yep. Excellent. So, no, here's a question from Aaron. I'd like to keep the door open between sociology, critical social science in the academy and workers and evaluator of social services. Is this a thing? So the door between sociology, critical social science in academia and workers and evaluator of social services. Absolutely, but one day one side of the other is going to just suck you right in. But you can start that way. Sure. Any other questions? Here we go. So from France, how do we get a foot in the door? I've done part-time, mostly voluntary, available for five plus years and have really, have really broad experience but not very deep. I've applied at government, not for profits, big and small, consultancies, and I'm at a loss to know how to get a foot in the door. How do you get that first start in evaluation? It's a good question. Any panellists have any ideas? I apologize if I was one of the presenters here, but sometimes it's, yeah, because it's the funding space is very volatile. I think it is sometimes luck. Like, I know I'd always encourage people to send their CVs around, but sometimes it's just not the right time to have someone on because you haven't got the work. Yeah, that's a tough one because I think I've been lucky a few times. I think when I first started at the Centre for Programme Evaluation, I was finishing up with Patricia Rogers at RMIT and there was someone who was retiring, Neil Day, at the Centre for Programme Evaluation. This is around 2005 or 2006. And, yeah, it was just walk out one door and in the other, but that was just pure luck. Luck, and plus I've done a lot of hard work for the guy who set up the Centre for Programme Evaluation. So, you know, I was a shoe in for the job, but, yeah, it was just timing. And I think, yeah, if you're approaching people, so I think you're doing all the right things. And I hope something happens for you soon. Yeah, it's a tough field to break into, particularly in Australia. It's that networking and relationships we talked about earlier. But I would say if you can, if you have a good relationship with any of them, have them take a frank look at your CV and just say, what's the balance that's needed to push me over the edge or be a better candidate. So maybe you've got a really strong statistics background and they say, well, we just need to see a bit more evidence of stakeholder engagement. And so in your next volunteering, have a focus volunteering experience where you're trying to get stakeholder engagement, you know, or statistics or whatever is missing to balance out a bit more. That would be my suggestion. You're welcome to flip me your CV and I can look at it and be brutal make you cry at night and tell you what you should have in it. More than happy to do that. But yeah, I think sometimes you don't get enough feedback, particularly with government or larger not for profits. You just get told that, you know, we didn't quite have the budget to hire lots of people. Thank you very much. So if you're doing volunteer experience and if you're doing all that networking and you're meeting with a person of fortnight and having a coffee with them. Then I would say, see how see where you can upscale to bring more balance to your CV. And good luck. It's not easy to be like six months to get my first gig here. It's just, it was a painful six months. It's important to contextualize it within the economy in general. So the way I read it most graduates don't get a job in their field that they've trained in for over five years, you know, so so it's not just us. It's, it's just how things are going and and I'm really surprised that a lot of people are having to volunteer, you know, and they before or in 10 they call it before they get, you know, paid position. So some of that just has to do with where things are broadly, you know. But the other thing I guess I'd say too is, you know, France, I know you're doing this now right like you're you're doing consultancy right now so your foot is in the door. You are through the door in fact and, you know, so just keep on doing it keep on working. The other thing is I think that in delivering this evaluation 101 training through the AES so many people just find themselves falling in sideways into evaluation as a field that they didn't really know existed until they were doing it. So maybe that maybe a tip there is work in an area related to evaluation or somewhat in that area and then try to move sideways. So within government there's always a huge amount of job shifting and the secondments and filling roles sort of sideways and in other departments and other areas so I guess part of getting the foot in the door is getting the foot in the door somewhere near evaluation or somewhere in that area where you can then transfer your skills to an area that you're more interested in. So it's a little bit of a longer road but probably one word thinking about Ruth. Yeah, I was just I was going to add on to what you're you commented Charlie like I've had a few periods of time where project finish up so the job's gone. So what are you going to do next. And of course networking is so important being the right place in the right time as Brad said it really really important and understanding the economy. And what I've also found works is figuring out what people really don't like doing in the evaluation cycle and offering skills. So I had a period of time where I just keep doing systematic literature reviews just literature review after literature review because no one wanted to do that so that was fine. So that's what I was doing for a little while or you know transcription and all that kind of I mean that was earlier but yeah figuring out and I'm sure you've done this but figuring out what people don't like to do and then kind of developing an expertise and then suddenly people say oh this person keeps doing this thing we all have to do it. So let's keep asking them to do all this work and then eventually the other things will align the market will be better you'll be in the right place at the right time and a job will come it and you're ready for it. But that might be something else to consider. Excellent. Ellie you've do you have a comment on that. What you were saying Charlie I think it's a good point to look at other areas for example research positions in government or in the not for profit agency. Often those things can lead to evaluation roles or if you have some experience in project management or policy again government. Is it is a good place to look for that and then you could often easily transfer over to an evaluation unit it's really easy to transfer even across departments. I know government it can be quite competitive through the application the formal application processes. There are a range of different kind of recruitment firms that recruit through government I think haze and Hudson and maybe a few others. And that can be a way to get your foot in the door I think they offer small contracts like three month contracts or six month contracts but once you have your foot in the door and are able to show your skills often. A lot of opportunities can come up that way. Yeah yeah and I think the other thing around volunteering it is it is something that as as the younger generation is required which is unfortunate in some ways because it is quite a challenge financially but being maybe strategic about the particular areas which I'm sure you're probably doing but the particular organizations that you do volunteer in to see maybe whether there are opportunities that might come up for employment and sometimes that's not in the NGOs because they are limited funding that's what I found anyway. But yeah that's just a few ideas. Right well we are coming to the close of the session I do have two two important announcements first one is there's a the AS is running a group mentoring pilot where they will link link you up with a with a senior evaluator. I think a research fellow of the AS and they're doing group mentoring so groups of I think five or six mentees and having someone that you can speak to regularly facilitated through the AS. So if you're an AS member it's possibly open just to members but there is a mentoring pilot that you can be part of the information will be on the website and the application closes on the 22nd of November so there's a few more days left to apply for the first I think it's worth noting to Charlie that the mentors are super experienced super high caliber like really amazing people they've got lined up for that mentoring program. Yeah, excellent and thanks for sharing the link there Brad. The second thing is just to acknowledge that Leanne Maloney who's been the Victorian regional convener for a fairly long time is stepping down from that role and moving into state so therefore can't sit on the Victorian committee anymore but just like to thank her on behalf of the Victorian committee for I think 10 or 12 years of service to the committee and almost half of that I believe, maybe longer as the Victorian regional convener. So thanks Leanne for all your work putting together the seminar program and leading the whole Victorian operation. You've done a great job. Yeah, good job. Thanks Charlie snuck that one in. It's a good time to announce who the incoming regional convener is, which will be put forward to the board to approve this weekend, which is Eleanor Williams and she's currently the COVID Evaluation Director at the HHS and has been at the HHS for a while and set up the evaluation unit there which has grown over time just instrumental in a very active and setting up the Victorian public sector and the Australian public sector evaluators network so you're in super good hands. Very lucky in Victoria. So thanks everyone for dialing in we're going to stay on the line if anyone's keen to just continue the dialogue or just social networking whatever it might be unstructured time. Hang about and we'll end the year as we would normally do face to face with a bit of a catch up and drink if some people want to go into breakout sessions we can facilitate that as well so just let me know but thanks everyone for coming along to today's session and thank you very much the panelists for contributing your time ideas and thoughts and journeys as evaluators. Thank you.