 I'd like to open the seminar by doing the acknowledgement of country so I'd like to acknowledge the traditional owners of the lands upon which we are all streaming into this seminar from this evening and so for me here in Victoria and based in Melbourne I'd like to acknowledge the Wurundjeri peoples of the Kulin nation and pay my respects to Elders past and present as well as acknowledging future and emerging cultural leaders. I'd also like to extend that respect to any Indigenous identifying person that may be joining us this evening or those that that may also be viewing this recording back at a later time. Honestly I'm pretty thrilled tonight to to be introducing our speaker Dr Matea Rorder who is doing her presentation this evening on towards developing evaluation criteria a conversation about chosen values excuse me. So Matea is a hugely experienced evaluator. I know she's probably known to many of you on the seminar this evening. I believe she's had 20 years experience in evaluation which is which is pretty incredible and is currently a senior consultant at Ellen and Clark. She's focused a lot on social sector evaluations in New Zealand, Australia and the Pacific Islands and of course developing defensible criteria for public sector evaluations was the focus of her PhD research and I know she's going to be drawing on that this evening. Matea has also been involved with the AES for a number of years which I did want to acknowledge as well Matea and in 2010 she actually won the best evaluation study award with colleagues from it value research. So I'm really quite thrilled that that she's agreed to be our first seminar speaker of 2022 and I want to hand over to you Matea to to start us off and just also noting to people please feel free to post comments and questions in the chat as we go. Matea is happy to also take those questions as she as she goes through her presentation so feel free to use the chat to record your comments. Thank you Matea. Thanks Ruth and kia ora koutou to everybody I'm on well I'm in New Zealand I'm in Whanganui Tara which is Wellington and today the government announced the New Zealand government announced that the borders have opened to Australian residents so I thought it was appropriate that I put a beautiful picture of New Zealand up there so hopefully encouraging you to come I don't know it seems like an unreal thing to be traveling again but very welcome so tonight so tonight it's 7 30 in New Zealand I'm going to be talking about an aspect of evaluation that is central to our practice and it's a part of evaluation that I think is still not that well understood um so I'm hoping that together we'll have a conversation about it please there are no dumb questions or all questions comments and thoughts are welcome so I encourage you to um put your ideas down in chat and um and we'll just stop the presentation and see where we get to um as well as values not being very well understood I think there are still not that many tools to help practitioners through the development of criteria so I will be sharing um sharing some a tool that I developed as part of my phd um I will start by um talking about some basic evaluation concepts which may well be known to most of you I don't know um and then we'll talk about the tool the tool in practice and then moving beyond the tool to um developing criteria so my part of the presentation of sorry my part the tool is the first step in developing criteria and and just a little bit about that word um or that term chosen values that's from Jennifer Green and she made a comment once that evaluative judgments are anchored in chosen values so if I leave you with one message tonight it's that you cannot do evaluation without um attending to values you know it's not enough to just collect data you need to think about the values um against which you're going to be anchoring your evidence or and um coming up with an evaluative judgment so hopefully this makes more sense um so this is the logic of evaluation this underpins every evaluation whether you are doing a personal evaluation about you know your holiday or whether you're doing an evaluation or a professional evaluation of a policy or a program or whatever it is that you do in your in your day job so there are four parts to the the logic the first one is that criteria define what good ought to look like and we sometimes refer to criteria as what matters um or chosen values so different words all mean the same thing it's the thing that is most important um in a program that you need to consider when you're developing your evaluation plan and thinking about what evidence you're going to be collecting at the second part is about levels of performance or standards so it's a rate rating so if you think about um for example a holiday um and your and your chosen values might be that you go somewhere where it where it's warm um so temperature is really important and you might have um a level of performance which is that it at a minimum it needs to be 30 degrees but you don't want it to be too hot so 50 degrees is um not great but neither is 15 degrees so ideally what you're looking for is a comfortable but warm temperature and then what you do is you collect your evidence so your evidence is what is happening in practice um so you gather data um and in this case you might go on a holiday um and blast it's raining and it's 15 degrees so you're comparing your evidence against what you value which is a in this case you know 30 degree temperature sunshine um and you and and you're so you're comparing your evidence against your values to reach in a value to judgment um now I just want to stop there and ask if if that's sort of clear to everybody um and something that you have come across before anybody got any questions about that okay okay all right okay so um so just thinking about you know even a basic activity like going to the supermarket and when you're thinking about what to buy inherently or implicitly you will have criteria in your head about what it is that you're looking for and that's based on your personal values the challenge um when we're doing a professional evaluation is to think about um um the values that you're going to put in place against which you're going to make an evaluative judgment so for a program so you know the stakes are higher you need to think um a little bit more systematically and carefully about whose values and what values you're going to include so this is a tool um that um well that is called a values identification matrix and what it's intended to do is surface um what's important or what matters um and it rather than just taking say the client's values at face value it's a tool for systematically working through all the different perspectives that might be at play for a particular context um and there are two parts to it so the down in the column on the left hand side you can see um interest groups so what you're needing to do is identify um all the interest all the people or entities because it could be a forest or a sea um so all the entities that have a key vested interest in the initiative um that you're going to evaluate um so it could be the funder it could be the providers it could be and it should include those most affected by the program and then along um in the rest of the matrix are different perspectives um and I've listed four key kind of perspectives that are based in theory around value so the first one is around consequences so what's most important is the outcome in a sense so you know that that concept of best bang for buck or what matters is say you get um get certain outcomes from your program is considered really important um and then the second one is about obligations or duty so things like contractual obligations um that that interest group might have to support that initiative the third one I think is quite familiar to many of us um and that's about rights and equity and fairness so what's important is that you abide by certain um um rights that you enact them um and for a particular interest group that these things are being considered and then the last one comes out of a out of that ethic of care so it's about caring and relationship so what's important is that you're taking um into account um things like collaboration or relationships or taking care of other interest groups um so uh what do I want to say about that um yeah so the the values identification matrix is intended to be a practical tool um and you can use it retrospectively um or you can use it as part of your evaluation scoping um and the idea is to you know if you've already got a set of criteria you can use it to map against the matrix and then think you know okay well we seem to have a lot about outcomes or we seem to have a lot in terms of one particular interest group and what they value but what about the other groups um so it's a it's a an opportunity for you really to um check that you have all the relevant um values all the relevant perspectives on what matters for this particular context and it may be that um one particular perspective um is not relevant to a particular interest group and that's okay at least you've checked it so so it's it's just a tool um to check that you you are taking care of all the values in a particular program that you're going to be evaluating and we've got any questions there about that tool i'm going to i'm next i'm going to show you an example of the tool um and an evaluation so i'll just wait in case there is a comment by gerry um i'm not sure if if you may perhaps want to elaborate on that gerry i know it was a comment rather than the question but um um would you would you like to him to elaborate or them to elaborate yeah well you could read the comment please feel free to interrupt me gerry if you want to to clarify so the the comment was gerry was saying they've been taught by a values must be absolute or relative the evidence must be based on maybe based on excuse me observation logic and slash or authority and c values are often multiple and conflicting um and he was just they were just saying that they're listening to explore how they were taught him and could use your framework in their practice so that was i thought an interesting observation that you may want to respond to menta um i'm going to respond to the the one about them about values being often conflicting um so one of the things that i have found this tool useful for is a to see eyeball where the values are conflicting how you deal with that um we could come back to in another in another perhaps later later on but um so it it's really important to understand that um yep not everybody will have the same values but it by the same token you may identify that different interest groups do have similar values as well um they might um understand them in a different way and if i go to the next slide i hope people can see this but you'll see um and i if i move my cursor do you see that yeah okay so um so this is so just by example in this um this is an evaluation of a funding um project that happened in new zealand that i was part of evaluating it um last year and um three billion dollars was pumped into local communities um with the idea that it would spark some regional development and most of these local communities and regions had been starved of any government central government funding for about 40 years um so the idea was to um get projects underway um and these were large and small projects and it also had a very strong focus on marie so indigenous communities and um if you look at um just this column here you'll see that um what was important for regional government and for funded applicants sometimes funded applicants were also regional government so i pulled them into one uh interest group and for them what was important is that priority projects were funded and were getting underway that the money actually went to the region and the projects were starting to get built um so projects like cycle paths and irrigation projects and roundabouts and then you'll see for the indigenous communities it was similar priority projects were funded and underway so um but then you'll also see across here that there are some different values so for example um in this last column around relationships I just have to move you out of the way um it's nuanced so what was important for regional government is that central government was having a relationship with regional government for indigenous communities they wanted central government to be working well and in ways that fit with their own cultural values and for government they were interested in collaboration as well but it was primarily around collaboration across different government agencies so they wanted a connectedness to happen between different agencies so the idea is that um you surface what's important for each of the interest groups and the interest interest groups um in this case um I've I've just grouped them into three because that would that was all that would fit onto this page um and if we if we go through it um you'll see the nuance the different so the consequences is all about outcomes you know strong economic performance um projects getting so outputs projects getting funded and underway um youth being trained whereas the obligations column is about projects being completed on time and within budget so meeting contractual obligations and that you know the decision decision making was allocated in line with the objectives of the initiative and that the contracting process was simple and accessible and in terms of fairness and equity for the regional government um they what they wanted was a transparent process for deciding regional projects so they didn't want a completely top-down process they wanted to be able to make their own decisions about which projects would be prior to prioritized and they also wanted it to be fair for everybody in the community to have access to the funding if they met the criteria um for indigenous communities um and that they were interested in um the rights of natural resources so in New Zealand we have some natural resources like rivers that have the rights of a person so they wanted those rights to be upheld and they didn't want projects to reinforce existing inequities so for example with some irrigation projects the irrigation would be in place but there'd be quite a high cost to accessing the irrigation that meant that rich farmers could access it but small indigenous communities were not able to access the same irrigation and then in terms of care and relationships so mostly this was about relationships um and one sense you could say that the health and well-being of natural resources is about also about care and I don't think it really matters where you put stuff it's more about the different perspectives being a prompt for you to think now what matters here for this group um so any questions about that okay we did have two questions we we didn't quite cover off on the last break yeah sorry no no that's that's all right Ian um we missed your hand being raised I don't know if you had a question or comment you wanted to share uh yes I did thank you very much for for noticing that uh I was hoping that this table was going to answer my question that's why I put my hand back down but what I'm interested in and I've got a bit of a financial background so I know that finance and cash drives so many things along with electoral votes I've never seen either of those two aspects actually spelled out in these sorts of tables before and I wonder how they fit so an example would be let's say I had I think the figure was three billion right so the figure was three billion and it'd be interesting to know how much went to indigenous communities and how much that is duplicated in the regional government areas as well which would become a problem to to deal with um so I understand all of the theory of this but when push comes to shove nine times out of ten it's the almighty dollar that ends up ruling that ruling the answers and and sort of a platitudes towards those who who have less inequity by point 001 percent than they had before to to meet the political taskmasters I've ragged on but any comment along that sort of I'll be interested to see how the three billion in this sort of project worked out you might be able to affirm me to uh to a report or something but thank you for that thanks Ian and well there is a report I didn't include it in the list of references but I'm happy to share it later I guess my response is that well the report did unpack how much went to Māori projects as opposed to non Māori projects and we know that the money went into particular regions so we'd be able to see how much went into a region and how much of that went into and uh projects that were tagged as particular indigenous projects I mean obviously there are regional projects that mainstream in a sense and might attract indigenous youth into training or employment and they wouldn't be picked up in those stats but it's uh the toll is really a way of making sure that those most impacted by an initiative have what's of what's important for them included in the evaluation so so the way that you would use this having surfaced what's important for different stakeholders including government is that you would then write your questions um and you're and then having written your questions so one of your questions might be if we look go to the next page for example okay so this is actually from the report and we didn't write the questions the questions were given to us but we've got a question around how well the design and approach has been actioned to maximize the changes needed to achieve the desired impact um and you'll see that the criteria from the previous page so if I go back to this I can't go back um but you'll remember that there was collaboration and pre-application and decision making and contracting and client management so we looked at those things but within those criteria we also had um a particular focus on say indigenous values as well and for this evaluation so the way that we understood collaboration was based on what was valued by the different stakeholders and then we looked at how that played out for those different stakeholders um for this evaluation we actually had a bar so if it didn't work very well for our indigenous population then a rating the rating could not be higher than consolidating um so this is going beyond um what the values identification matrix will give you but I just wanted to show you how we use this um when we had done our analysis so you know we collected our data these were our criteria um we understood the criteria in a particular way based on what was valued and um that included the values of government they of course they were very happy to share their policy documents and as key stakeholders um perspectives on what was valued but we also asked those communities um in the regions and indigenous populations um so um so this is this is what this is how it played out um now I don't know if that answers your question great all right it does okay any other questions about we do have um your question from Frank here which I think you possibly partly answered in response to Ian but um they've asked does the values identity identification matrix have to be completed by stakeholders can it be done on their behalf and still be allowed great question so I use I wish I could go backwards oh there we go okay so you'll see here that I use the term interest groups and I differentiate between interest groups and stakeholders so what you're doing here is identifying who has um a vested interest in a program to get the information about what people value you can go to a number of different sources so you might have a needs assessment um you'll have most definitely probably have policy documents and contractual documents um you might talk to the people who are most impacted um you might draw on subject or cultural experts and then you might have literature from previous research or previous evaluations there might be other sources that you draw on so you're using those sources not just the stakeholders to identify what's of value so all of those sources will be useful in surfacing what's of value for a particular interest group so in many evaluations that I've done in my practice at the scoping phase what we've done is government stakeholders have told us what is important for them you know we'll do some key stakeholder interviews often there's hardly any time to ask anybody else um you might look at a bit of literature but the the idea is that um you know if you're going to do a systematic analysis of what matters against which you're going to be then collecting data and getting to your evaluative judgment then you need to think very carefully about who you talk with which sources are warranted you know obviously with your literature you might be thinking about um literature you know I'm working on an evaluation at the moment where we privilege indigenous literature um so indigenous authors over other public health authors from you know world war health organization or wherever um so in each context it will be really different but it's really careful really important that you warrant your sources you don't take them at face value um so and sorry that was a bit of a lecture sorry Francis does that answer the question or not ask me the question again and this is going to say yes and no apologies for the lecture it was wonderful not a lecture at all okay well I didn't really want to lecture I would much rather that this is a chat so if anybody else has a comment or a different perspective please speak up David um has just shared a comment in the chat box he's said I'm interested in how you came up with a list of four value categories I did a quick google and found over a hundred values so um if I told you where I got these categories from I thought I would skew you all away um so in philosophy I know wouldn't you in philosophy there are basically three ways that you can categorize values um there is something called deontology deontological perspective which is consequences so what matters is the outcome how you get there is of less um consequence yeah what's important is that you focus on just getting good outcomes um and that wasn't deontology sorry I got that wrong it's a consequentialist perspective the the obligations comes out of a perspective called deontology so that's that's another rule based way of thinking about what matters and what matters is that you always do the right thing that it's um that you abide by rules or regulations or contracts so you know um if we think about this example here it's that you get things done on time within budget um that you report to government at a particular time so all of those things come out of this deontological perspective and then we have um another perspective in which rights and equity and fairness are grouped which is that you know you it it's not rule based it's about um particular things that are important like human rights or animal rights or that equity or that your fear in a particular circumstance um and then the last one comes out of an ethic of care perspective um and that's had a lot of uh there's a lot of literature in recent years that's come out of the feminist from feminist writers which is about care and relationships and in New Zealand um with with indigenous communities you'll often see that that is what's most important so there is a a saying here that um what's most important what's most important it's the people the people the people so you get the relationships right and everything else falls into place don't worry about outcomes don't worry about rules and regulations don't worry about anything else what's most important is that you are in relationship with others and looking after others and there is a crossover between them so don't get too hung up about where those different perspectives sit it's most important that you just start to think about the fact that there more than one way to think about what's important so if we go to this slide you know if you if you were using the values identification matrix at the beginning of an evaluation you could have a series of questions around those three or four different perspectives and um they could inform your stakeholder interviews or what you look for in the literature around surfacing what's of value does that help david yes i'd like to i'd like to challenge you on the fact that there are many many many different ways i think they all boil down to which which i think is quite useful i mean who wants to think about thousands of different ways of what you value if you can categorize them into three or four ways um i think we can manage to wrangle them into something i wasn't suggesting you shouldn't have four categories i was curious about how you came to categorize them yeah we answered that really well thank you yeah so something that i didn't mention is that the values identification matrix is grounded in philosophical perspectives that go back you know a gazillion years um and also the tool um has empirically been tested so it's it's a tool it's only one but it does have a bit of theoretical grunt behind it so me and aris dottles aris aris whatever his name is plateau okay um any other questions or comments just one comment additional comments oh and now another question so maybe i'll go to marianthe's question um they've asked did you explicitly discuss the use of the values identification matrix with key stakeholders how was it received and what feedback did you get another good question no i used this matrix in retrospect so with this particular client as often happens i don't know certainly in new zealand we were given several key evaluation questions each of which had a dozen questions under them so sub questions which um if you looked at them um had implicit criteria in them um and then they also had criteria so this client client particularly wanted a strong evaluation framework evaluative framework with a value of reasoning um but they had the criteria in place so we we used it to check in this particular circumstance and i don't know that you even need to use it um explicitly with clients i mean it's it's really just about keeping these questions in mind thinking about who are the interest groups beyond government or whoever the thunder of the evaluation is um where am i going to find out really what matters for these groups and then however you you know however you decide to do it may be a matrix or it just might be a literature review or whatever an exercise in identifying what matters and then you know doing an exercise like this where you grow you pull them up into a set of criteria um and just you just you know when you're developing your criteria there's a couple of things that you need to think about one is that they are quite separate that they're not overlapping um and that they're they are defined at the same level so that you're um you're not looking at a sub category of a criterion so you can get yourself into a bit of bother if you do that so so it's it's just a checking of you know these are the key things that we need to be thinking about um in this case we already had evaluation questions around all of these aspects um but i'm working on an evaluation at the moment um and also it also has an uh indigenous context in Australia whoops we're we've identified criteria so one of the exercises that we did in our first round of data gathering was to ask um people indigenous communities what matters what matters in terms of um primary health care for them and that has surfaced um criteria that were not included in our original set of questions so there is some overlap but there were things there that matter to them that were not in the original program logic or the questions or the way that primary health care was being um articulated by the federal government so um we didn't in that case use values identification matrix um but what we did do was make sure that we had a specific piece of work around identifying what was important for those that were most affected by the initiative so however you do it what's most important is that you surface all the relevant criteria and just coming back to a question um that somebody had earlier on about what you do when you've got criteria that are really at odds with each other I mean there and feel free to add to the conversation but one way I've seen that done is to report on the different perspectives so you know if it was collaboration and a particular thing was important for a particular interest group you'd report you know and for this group it worked very effectively so that's your evaluative judgment but for another group who value something different um it was not effective so you might say for this group what was important was this um and when we looked at the evidence that wasn't happening so it's not effective for that group alternatively you might decide that one particular interest group is prioritised over another and then go through a process of prioritising how you reach your evaluative judgment so you might set up something similar to what we did where you set a bar so if it's not working for a particular group then it automatically brings the rating down this feels like a very one-sided conversation I don't have any more questions yet or comments yet but um well we've got a hand Jackie Matthea thank you so much for your presentation so far it's very thought provoking for me I'm curious I don't know if I'm going to be able to articulate this it's just come to me but is is there a sweet spot when it comes to how many criteria you develop I feel like I I could potentially you know develop hundreds which would become unworkable do you kind of have and I'd like to point about make sure that they're clearly defined and at the same level but in your experience have you found is there kind of a nice number or is it just too context dependent um I think you do want to try to group it up if you can um but it might depend on the context and the evaluation um the evaluation that I'm working on in the Indigenous Australian context we're just starting to develop our evaluation having done that first piece of work to identify what people value and um for each evaluation question we've probably got about eight to ten criteria criteria or um and we have grouped it up so what we've done there is use the key evaluation question as the main criteria and so take so made it more explicit so the the key evaluation question is something around you know um does this and you know to what extent does this program um work well for example for Aboriginal and Torres Strait Islander people and then that concept of working well has um say eight dimensions and then each of those dimensions is described based on the data that we've got from what people say they value so it's layered um so in fact we've only got one criterion which is that the program's working well but then there are sub criteria like or dimensions or components and then each of those is described so we're unpacking these kind of nebulous terms um and what's most important is that in the unpacking it's not the evaluator's interpretation of what that term means it's transparently coming from um a place or another place in this case mostly what people tell us is what's important um does anybody else want to add anything I just want to acknowledge my colleague Brendan I didn't realise he was going to be coming so Brendan um was a co-evaluator on this evaluation that that is up on the screen at the moment and he might want to add something about how we how we got to the evaluate of judgments I wonder if you hear you sort of talking you're speaking your your VIM truth and I love philosophy too I was thinking about um critical realism and how you're unpacking the mechanisms and structures that we can't see which which hold these values which often cultural but going back to your point about setting a bar for value because I think that's such a key thing often we in doing these evaluations you the most the most the smallest groups who often they um the um often don't get seen these evaluations because there's no way of moving their priorities up in the evaluation um so this idea of prioritising the most vulnerable but the smaller populations because they obviously you know there's not as much money spent on them so that is expensive in an evaluation versus a general population with huge numbers but only a marginal impact on them so it's this balancing of small populations but big effects versus a big population with small effects and that way of um using your your value matrix to try and bring those voices up to bring some balance into it is uh really it's really useful that's me yeah I mean that's the wonderful thing about working with colleagues who don't necessarily come from an evaluative perspective one of the challenges that we're going through with our indigenous evaluation in Australia at the moment is how to explain this approach to to colleagues that don't necessarily come from an evaluation background and things that come to mind and and challenges for me uh how do we make sure that it doesn't become a reductionist process when you you know because very often in real life everything is connected so when you break it down to its components what does it what do you lose and how do you make sure that you hold the whole um so that's one of the challenges that I have very keen to hear any perspectives on that you're in a couple of of extra comments in the chat box Matea um so perhaps I might just take the last two two questions there um Marianthi was just responding to to your comments Brendan saying um the intersection of evaluations like social ethics and perhaps advocacy so I think that was a comment rather than a question but Michael has just asked how do we manage the situation where what people say is important to them so where what people say is important to them is different from other sources such as existing literature or other sources of evidence you were sharing on your slide Matea um um I don't think it matters that you have different perspectives the whole purpose of this is to see that different perspectives exist and then you've got your force to um think about how in your evaluation you're going to hold on to those different perspectives um are you going to privilege certain perspectives um are you going to report on the different perspectives and then what the evidence says and what that might mean um uh anybody else want to add anything he means just posted your comment I don't know if you want to speak to it Amy sorry I'm dead thank you I can it's just that I think sometimes we the we treat the literature as an authority but fundamentally it might be biased as well depending on who did the study and and if it was an evaluation that only privileged funders and not the perspectives of the community so I think it's a way that's why I think the matrix is so valuable because it puts that against um you know what's actually true for this particular evaluated in this particular context which um may be different from the literature but it also might be that the what happened in the literature had some inherent bias that wasn't identified so I think in your evaluation what's important is that you think about the sources that inform what's of value so you know are they a subject expert in this particular context and what we have found with this public this primary health project in Australia is that what health experts overseas say is important is not necessarily what the indigenous population say so when you gather a whole lot of data across Australia on what different indigenous communities say is important and you analyze that that's a strong perspective and actually with that particular data there was even though they're very very different communities and we know that Aboriginal in Torres Strait Islander communities are very different what they were saying was pretty much identical so that's a strong warranted perspective on what's important for them in terms of primary health care um and we would privilege that data over any literature that we were reading and of course it's um we also looked at the literature and then again also privileged indigenous Australian authors to understand what they were saying and how it might be different there's um one question here from Gary that I think maybe extends this conversation a little bit um they've asked whose values matter most is there a risk that some teens or consciously perhaps question mark grant more matter to the view of a funding body so the team can be in the running for another evaluation job yeah that's an ethical question um so I don't know if we're going to be able to able to answer this but as part of my PhD I did a review of evaluations in Australia and New Zealand and there happened to be several evaluations of the same initiative in Australia um and it was very interesting to see um how some of those evaluation companies took and they were exactly the same criteria that the government different the government federal government or state governments used um however some one or two of the evaluation companies dug deeper into the criteria and the way that they understood value and also critiqued it so I think it's very nuanced it's a very vague way of saying that I do think that some evaluations are very superficial and basically give government what not not so much what they want but they stick to the prescribed questions and the criteria interpret them in a particular way whereas other evaluators will take it to the next level and think about it from different perspectives but I think we all know that and that's the that's the value of um I guess having multidisciplinary teams um but yeah it's a it's a challenging one Amy and Brendan have the hand Brendan is it a new hand or a previously raised hand it's an old hand sorry a legacy hand and Matei I'm curious about how so it sounds like in this one that you're talking about as an example might be a hidden answer to Jerry's question because you they gave you a set of criteria then you went and you asked indigenous communities what mattered to them and there were difference so how did you persuade the clients to accept those indigenous things instead of just using their own set we haven't finished the evaluation however the way we set up the evaluation is that we said that so there is an evaluation question about how well is is the initiative working for Aboriginal and Torres Strait Islander people and the first step was to understand what's of value for those people so it's it's set up set up clearly in the way that we have set out the evaluation and I guess there you know we we all know that there are ways to bring the client along the journey so we'll be doing that I don't you know I want to I hope there's no federal government people here but they're pretty clueless about what's of value because they don't have the opportunity often to go into those communities and ask them what's important they are stuck in Canberra and they will you know I think that they would well I'm probably digging a deep hole here but I think they probably agree that that happens a lot is that they just don't have the opportunity to go out and therefore they're looking to us for the answers because we had that opportunity to go out anybody from Canberra wish to provide an alternative or supporting response to that David's worked in Canberra I'm going to decline to answer on the grounds that it may incriminate me however I don't disagree with what you said um yes I have worked in Canberra I spent 20 odd years there my when I first went to Canberra um we had I was working in an evaluation team and we had a seminar where they were giving some results from an evaluation and there was a long debate about one of the findings which was that a number of women claimed that they were working part-time and that was the word that was used claimed that they were working part-time because they it took after about 20 minutes I asked them what did they mean by part-time work and they said something I can't remember the exact numbers but 15 to 20 15 to 30 hours a week and I pointed out that if I had a part-time job as a cleaner cleaning a house for four or five hours a week I'd see that as part-time work and they um they just couldn't grasp the concept until I said it put it like that I'm not sure they did grasp the concept ever so yes I do think there is a cultural divide between um the lives and the lived experience of many people in Canberra and those particularly I'm indigenous um in indigenous communities although I'm going to be kind and say that I think it's partly because they don't have that opportunity to go out and meet with people and that's certainly what we hear from communities is you know why don't you know these people from Canberra just come and visit us and and even the same with state and territory with some particularly state officials they don't often have the opportunity to go out and see what's happening on the ground either they probably would love to so I don't know what the big deal is but I think we could solve a lot of problems by getting people in relationship what's all about care and um collaboration oh I have one last slide and here I just want to say um a huge thank you to Amy who has nurtured my interest in criteria and co-authored as you can see all three of those articles so if you're interested in reading more um check out the blog which is the top one or the two articles if you can't access any of them just let me know I'm happy to send them to you