 Please make everyone sit. Welcome everybody. I am午 slightly.) I am a member of the USAWA committee. In the moment I'm going to provide today's acknowledgement country. As a committee we wanted to take a new approach for this part of our event. Felly, felly, mae gennym gael bod yn bwrt yn gweithio, Llywodraeth i ddodol o'ch cyffredinol i'r rhwng ac yn siaradau cymaint Passybl, ac mae'n mynd i bryd i gael'i llwg iawn i'w gymun. I'n mynd i gael i chi'n cuno'r hyn yn ymgolod, cofnodd, ac yn cefnodd, Felly, byddwn i'n ffordd o'ch eich cyfredinol, reolaeth o'r cyfrif yn dda. fi o'r amddangos i'r holl o glingio ar館ig o bobi'r cyfahrol sy'n oaf swyddi, a ydych chi'n golygu'r ddysgu sydd yn ei ddysgu'r cymaint, o ran yw'r cymaintau ac yn ychydig yng Nghaerdydd, y hirth yn y trawn ac yn y past, ac mae'r oedd y MaeAES yw ddysgu'r cymaintau o ran yw'r ddysgu'r cymaintau y ddysgu'r prifysgol yng Nghaerdydd i ni'n gwaith ym Mhwrdd, ym Mhwrdd a Llywodraeth i'r Ysgrifol yng Nghaerdydd i ni. Ac mae'r sefyllfa yn y rhan o'r bolluniau, I'm really proud to reflect this commitment by providing today's acknowledgement and by doing it in this way. As a personal reflection, I also acknowledge that I'm a Brit. A long time ago I came on holiday to Australia and I thoughtlessly climbed up Aes Rock and I called it Aes Rock but I've never heard of Uluru. Then I came to live here and I've lived here for nearly 20 years and I've since then certainly been on my own personal journey of understanding and recognition and it's been quite uncomfortable at times. I also work here at the commission and so I'm really proud to show you our beautiful acknowledgement to country. It's been written by our own Aboriginal and Torres Strait Islander staff members and it's quite new to us this is the first time I've used it. We acknowledge the first peoples of this nation, the traditional custodians of country, Aboriginal and Torres Strait Islander peoples. We value the unique and diverse cultures, languages, sacred knowledge systems and important connections to land, water and country. As we work to promote and protect the human rights of all Australians we walk with you on your journey for self-determination, for social justice, for respect and for recognition. We thank the elders both past and present for allowing us to continue to care and nurture this great land that we call home. Thank you and welcome. Thanks Jen, I think that was very touching and definitely very authentic so thank you and to someone who attended the evaluation conference as many of you would have this year in Sydney, one of the things that really struck me and was commented upon by many people was the really inclusive approach to Indigenous issues throughout the conference. Okay, so my name is Greg Masters, I'm also a member of the committee and have been responsible for doing some of the coordination of the free events and this is the last one for this year and can I encourage you to come along to the drinks that we'll follow at 5.30 this afternoon. We have had I think a really good year of free events on these Thursdays we've tried to put in practice the interactive request that people gave us to make them less talky and more involved and we'll be doing that today as well and one of the other suggestions we've taken on board is to, we're trialling filming today that does mean that people that are in the front here might get filmed peripherally so if any of you are uncomfortable with that you can get behind the camera line I'm saying that because we want to avoid the technicality of filling out forms and disclaimers and all that sort of stuff so we're doing it in a less formal way but if you are in any way uncomfortable it's going to be posted subject to the AES in Melbourne's approval on the AES website so it's not going to be broadcast more broadly than that. Just a quick overview of the program the committee has actually had a go at trying to program next year's activities so I'm going to go through these very quickly but to give you a tantalising glimpse of what's happening in February we'll be talking about evaluation in Buddhism which is a re-run or an extension of a paper that was very well received at the conference we'll be looking at a paper around complexity and how to evaluate complex programs in March April about evaluation capacity building and that will also, those three papers are all extensions of what was presented at the conference in May we're doing something I can't read oh I know what it is, it's the un-conference format you know, bring the evaluation challenges that you'll keep you awake at night ethical dilemmas in June indigenous evaluation issues in July economic evaluation in August the conference which you'll all be attending in Brisbane is on in September and we'll discuss in October some of the emerging trends from that conference in November we'll have, so 12 months from today we'll have evaluation of behaviour change so some of those topics need to be filled out but we've got an exciting program ahead and expect you all to turn up and all to become members because one of our goals is to have more members then you got it we're on the way if you're not a member I expect to see you become a member otherwise shame we're taking the trapping next time thank you very much thanks for the invitation to speak again so thanks for coming along to listen I'm talking today about a topic that popped up or didn't just pop up it was fairly prominent throughout the conference by the way I will make my slide presentation about it to see if I need to take notes or if I want to be thanked by this and it's a topic I've been working with and on over the past few years in a whole range of spaces and I thought it was time to give that interest from the conference to sort of talk about those issues and rubric and just to start with an example rubric has a couple of elements to it obviously it's got a scale that matches some change of progress it's an ordinal scale it's just different it's a progress in some sense that in itself is not a whole lot different between standard sort of a question for example what makes this a rubric is the addition of these statements that clearly describe the conditions that have to hold for something to give this range so it's the use of these clear descriptors for each point on the scale that make this a rubric so you've probably seen them just generally how many people have tried to construct rubrics in their regulation or other work and just by the sharp hand you can see how common they are coming in with my talk and the regular experiences and what you've learnt from them now I've used rubrics in a whole range of areas ranging from measuring symmetry maturity to behaviour change to assessing capability development capability training a whole number of different evaluation and evaluation like contexts and what I've noticed as I've tried to construct them myself with clients or reading the literature on what's out there is that there's some common underlying scale points that define the rubric scale but rarely do you see them all in one place so what I'm presenting to you is not something grand or invalible new it's just pulling together some common threads that appear across a range of rubrics in practice so before I get to that just some other examples just to illustrate the use people have arrived at rubrics through different disciplines in different practices different things some people might have heard of goal attainment scales which grew out of psychology and criminal justice and rehabilitation and mental health and they use these kinds of scales with descriptors attached to them for what each of these means similarly those of you who are calling for agency here the Australian Public Service Commission has rubrics for measuring capability development and there's a whole range of domains and some consistent underlying meaning to the points on this scale so there's just some other examples there's a whole heap of them that we can come up with so as I said what I want to draw out is are there some consistent scale points on these rubrics that you find that maybe provide us the basis for a universal scale you may notice in the title that because that might be something we can discuss in my stretching things to do here but we can have a discussion about that now obviously one of the common scale points you see is what I've called fully realised the outcome is there in its full form whatever that outcome is whatever that you want to see interestingly the one that rubric doesn't really have that and I find that's a big limitation when you rubrics have used if you were doing an evacuation is good good enough for the end of the year to be excellent to be good if nothing else I think for evacuation purposes a rubric that has the outcome defined in its fully realised state is a starting point to do the rest but it's also to have the utility making evacuation adjustments so you can think about what that would mean obviously at the other end of the sky my other common point is it's just not there it's absent it's kind of not happening whatever it is that's a fairly easy one to define interestingly you might think points that find the type of model of the scar but when I've worked with rubrics I occasionally find points that exist beyond those for example something might not be happening the outcome is not present not just because of benign reasons but because there's something actively hindering coming about so that cannot those kinds of things as oppositional forces can be either internal like a motivation like a tangibility or it can be external if I did that it would kill my career systems don't exist to support me doing that so those kinds of it's just not only not happening there are barriers to it feel free to jump in if you've got questions or comments or seen things like this as well at the other end of the scale something this I got me an insight when I was doing some work for the Commonwealth Department of Environment and Energy on measuring regulatory maturity when we were talking about where they want to be down the track they said in 10 years time we don't want to just be turning up to international conventions and learning what other countries are doing we want to be setting the stand for regulatory practice in this area so that I just want to realise the outcome they want to be leaving it or innovating they're not exactly the same things that you get a sense of they're doing something new or establishing best practice so they've got interesting points on the scale you're sometimes seeing that that could be very meaningful in some evaluation context in some context you may not actually want this but the manager is saying if we actually got to that we're wasting resources we're happy just getting here we don't want to get there we better use our resources so it's not always progress if we define it in its context but it's good to know and obviously in between is the meeting part of the scale which is you're on the way at the very least the common points on the rubrics that I've seen are started but really early days or we're making progress but there's still some work to do obviously you can expand between hearing into more points on the scale of that but they're the two I see most commonly as a minimum now what defines movement along that part of the scale usually I've found have usually resolved along two dimensions one is the breadth or the reach of the program or whatever it is you're doing if you want to raise capability among this group of people but we've only really got this far yet and we've still got to do it for this group you haven't reached your whole target audience yet the other dimension that can decide how far along this scale you've moved is quality so we've reached everyone but we've done it really badly in an evaluation context that can often be also determined if you've developed a nice outcomes hierarchy and outcomes chain say we need to raise awareness first and then with awareness we can increase knowledge and then after knowledge we'll get some skills happening so those points in your outcomes hierarchy or outcomes chain can define the points of the scale through which you move that's the other way of doing it that's just to give you an example and I really want to stop for any comments first just one thing about what are we describing or what are we rating the outcome right it works well but from opposed how are we describing the outcome when we say opposed it looks like it's something opposed to reaching the outcome rather than describing the outcome itself you're right it's not the negative it's not going in the wrong direction it's something that is an active barrier to it it is strictly speaking a mathematician it's not the same thing but for a valuation purpose it's often found that especially I'll talk about again from a decision making point of view knowing this is important the opposed also be a realisation that the outcome is not necessarily a good direction there are other directions that are better it could be this gives you a measure like any other measure what you then do with it how you interpret it how you judge when you place on it is up to you as a context but certainly gives you a start to thinking about that so just an example if you're interested in bringing about a certain kind of behaviour change I know we'll get to that in a year's time we're going to talk about this in more depth but just later you can sort of see the kind of statement you might come up with this is from a direct example that I've developed for sustainability what you might the terminology might use for the labels if you want to measure behaviour change if you've got any questions or comments on any of those they're likely to be different so over to you on your timings you have a good paper someone on your table turns it over what I'm watching to do we want to develop a rubric collectively here that's going to measure this aspect of evaluation, practice and culture I actually have done this for the Western Australian public sector commission before I did a capability building workshop I developed a rubric with them and we applied it and it allowed me to focus the kind of training and capability development I did on the basis of I want to see what you come up with each on your table so I'll take this one for example I want this table to come up with a statement for leading or innovating as their point on the scale for measuring this quality so each of you has a different point on the scale so start by talking among yourselves get to know each other write something down and then when you're ready to show to the rest of us go to this metameter link with this code and type it in because it will then come up on screen for us to be able to talk that out there you go we'll take 10 minutes to see how we go it's strange I hear today this is a straight for you it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine ...dan yn y gallu'r anod. Ac ond, mae'n ddiwethaf ar rydyn ni gan o'n ddweud y gwyntaf gyda'r chweineidio cyfennol o'r ysgol. Yn y peth yn eu colliwn i gwbl swyddiad, yn yr anod o'r Llywunodau Rhywbeth, gofyn yn gweithio'r gŵr, oherwydd mae'n ddewisio ar gyfer y ddysgu staffol i'r maen nhw, yn y bwysig o llawer o gwylltiau'r model gweithl, oherwydd mae'n gweithlo'r modd bobl llai'r organ. If that was all happening you can go in and say, in my organisation, this statement describes the situation we can say that the outcome of whatever that outcome is using evaluation program design is cwydli will realise. A remove your understanding. Anyway, can you leave a comment or don't go to a pick it up? I will not hear that. I'm saying not a wrong statement because I got it right. However, other people are Conclude in if they were designing in that point on the scale we realized that. Woo. You would have whoo. Yes. And the discussion about the unit of analysis a point I was going to make maya that can be a very tricky one. You are talking about individual. dyfodol, fe fyddai gwahanol fel yw'r anhulu'n rwy'n rhan i ddweud, mewn i'r unig ysgol yw achos. Ymgyrchol maen nhw a'r wlad am gweithio canir ac ysgol, lle be oeddedd yw'r ysgol yn ni o'r unig, ac ar y pethau gwahanol. Ysgol o'r rhan i'r unig. Felly mae'n gweld ei alwch yn ddeg, ac mae'r dal yn oed yn dyma. Mae ydw i'n mynd i ddysgol am dyfodol am ddere anhulu'n ddere, a ydw i'n mynd i ddere o'r ysgol. Roedd yma eich… Ymgyrch yn ddullent y Moedd Grimiau i'r Ymwneud o'r bobl a'i ddweud. Yma eich 2bll�teriaeth. Gw pellwch yn ddim yn ddefnyddio gy prisoner i'w gw своod. Gw chillio'r amgylchedd i fath o'r mewn yn gwneud. Mae'r gwneud o'r amgylchedd i'r ond yn gallu gwneud. Mae'r gwneud o'r gwneud o'r ymgylchedd i'w gweld sy'n ddullion. Mae'r gwneud o'r wneud o'r ymwyaf yn dweud, ond rydw i'r ceisio'r ysgol iawn, rydw i'r bwysig gyrfa yllun? Dwy o'r ddweud o'r ysgol iawn, dwi'n gwybod y pethau ymlaen, ymlaen yw'r ysgol iawn,au'r gweithio'r ysgol iawn. Rydw i'n gwneud ymlaen arall. Rydw i'n mynd i'r rhwng o'r ffordd. Rydw i'n gweithio'r bwysig eich rheiniadau'n mynd i'r rhan. Rydw i'n mynd i'n gwneud ymlaen i'r rhaglen o'r rhan. The suggestion to it is that it is a collaborative process because it sometimes takes a bit of time to get things but I think it is a good idea to do something beyond what is minimally required for the outcome to save the success. Ion y cyfan o bryd wedi bod ychydig i'w ddweud yn ei wneud atgymu'r newydd? idinol o'r Cymru? Yn 3. Felly mae hynny ym yn arfermwyr. Mwneud o'r ffordd o'r proses cymaint i Cupysg Bryddi, mae'n llwyll dadle o'r rôl. ym Mhwyl S1rwm yng ngwybodaeth ym Mhwyl Rheidiw, newydd ym Ym Mhwyfgolwyr Mayfwrdd, gan hwyl iawn i'r ychydig i'r mhwyl yng Nghymru, astud am fawr o phae Maslaithaeth yn eu schylystaeth, yr Ym Mhwyl Rheidiw wedi gwneud allwy ni yn ym Mhwyl Rheidiw. Ym Mhwyl Rheidiw yr Ym Mhwyl Rheidiw, ac mae'n bellig yn eu hanechyd yn ei hytrifalu ym Mhwyl. De�iwch chi'n hyfrifwyr ar ysgilio'r���g. Mae'r prysyn, mae'n dderfynnu i ddweud, ac mae'n ddiwedd yn rhywbeth ffordd i'r lluniau hynny, ond mae'n cael ei ddechrau i'r rhannu. Felly efallai y rhai i ddychrau, roedden nhw'n gwell-nhynghwm, ond rydyn ni'n gweithio i'r yr hunrhyngau yn y bydd y rhannu. Rydyn ni'n dduld iddyn ni wedi eu sgw戴. Rydyn ni yn ymweld i sut ni wedi dduldol, mae yna'r rhannu i ddweud. and encouraging an inconsistent knowledge, culture and practice of evaluation. Evaluation is sometimes being used in program design, but maybe limited with lack of awareness, lack of resource or lack of formal process. So yeah, again, we've developed this one sort of down a little further and it's identified some of the areas which you need to think of if you're going to get through the gaps Any comments, thoughts, especially for those of you who developed it, or what was in your minds? Last one, making progress. There is an awareness of evaluation components, the benefits of evaluation and how they piece together. It's only awareness at this point, which is interesting. Some more all of the evaluation components are used in program design, but the use may be thought felly mae'r llwyfodd yma. Mae Fyfyrdd yn fawr, bydd yn rhan o'r perthynag, mae'r cymdeithasol yn ei ffordd, mae'r ymddangos yn eu ddweud yn annog. Mae'r ymddangos wedi'u gweld yn olygu'r ysgol. Ydw i'r ystod yn cyfrwng ysgol, mae'r bwysig yw sut ddwy'r hyfforddiad, mae'r bwysig yw ffordd, mae'r bwysig yw'r bwysig, a'r bwysig yw'r bwysig yw'r bwysig, it means however what is inconsistent tobleguorter is only done in half the number of few or sentence, tries to identify just where and how well and what is it… That's an exercise where I don't know what you think so I'm going to be working on the discussion now as exercising, get You to start with that so what are your thoughts or comments nor the actual details at all. Rwy'n fawr o'r sahen yng Nghymru. Rwy'n fawr o'r hyn o'r ysgol, o'r fawr ddweudio'r ysgol yng nghymfawr, yn y cerdd yma i'n ddechrau'r ystod yma. Ieithio'n gwneud y ddechrau'r ysgol yw i'n cymaint. Mae'r ysgol yn y gwerth o'r ddweudio'n ffraeg, o'r ddweudio'n ddweudio o'r ddweudio, o'r corwchio, a o'r ddweudio'r ddweudio o'r ddweudio'n eich ddweudio. Those should actually. When do you use one global scale for everything, or at what point do you break it up into multiple scales? Because I've seen some of that stuff that we've just talked about, even break it down and it'll have a separate routing for the quality of programming logic. Each thing you add in a statement might itself become the subject of an infra-judgment. Mae'r ailgo yma o'r ddaeth gyda myfyrdd o'r myfyrdd yma hynny'n i'r gwirio'r mynd, a'r meddwl i'r meddylidio drafodol i ddechrau sydd gyda myfyrdd yma, a wnaeth y gallai yma, a'r gwirio yma, a'r theib yn y byddodol i'r gofyn i wneud gyntaf? Mae'r pwysig hynny'n i'r myfyrdd yma e'n myfyrdd o'r gwirio. Mae fryf yn ôl eich cwrnod o wneud fynd a oedd i'r holl a ffyrdd o rhyw y'r gwirio, ychydig yn dweud o'r ffordd yn ymweld. Fy fyddai'n gweithio, ydych chi'n ddweud ystod ystod ystod ystod ystod, rydyn ni'n fyddai'n gweithio, a dyna'n ddweud, ond dyna'n ddweud. Yn y gallwch y ddweud, ydych chi'n ddweud ystod ystod ystod ystod ystod, ychydig yn ddweud ystod ystod ystod. Felly mae'r ddweud yn ddweud hynny, Mae'r ddifonwch yn ddifonwch, felly rydyn ni'n ddifonwch yn gyndeithiol i'r gwaith efo'r rhan o'r ffordd o'r cyflwng. Mae'r ddifonwch yn ddifonwch yn ddifonwch yn gwybod, mawr i'n ddifonwch â'r gwaith i'r ddifonwch. A'i ddifonwch, sy'n gweld cael ei fod yn goreithio'r ddifonwch, mae'r ddifonwch yn ddifonwch. Mae'r ddifonwch yn gwybod i'r wneud o'r ddifonwch yn gweithio'r ddifonwch. Ie ddod o'r ddweud o'r llwyddiad ymweld ychydig i ni i chi ddweud y dyfodol yn ymgrifwm, ac ydych chi'n rhai ddweud y ddweud o'r gwaith dim ond yn eu fathio. So, mae'r ddweud o'r ddweud i ddweud i chi'n rhai ddweud eich ddweud. Felly, rydyn ni'n rai'n gyfer y clywed? On that, do you ever get around that by offering another field that says other or explained why you chose that field? I got it! Gofyn, diwrnod! It just so happens! It wasn't a plan. I think there are limits to rubries however well you construct them. If you do construct them well, you can follow up and add really good intelligence and qualitative measures to them. If someone does tick a post, you can have a Q and Q that says, what is it about your situation that led you to that? felly i sut gwirionedd iawn y cysylltu, a müddio cyflwyddwr iawn i'r ddaluniaeth yn ei wneud. Oherwydd ta wedi bod angen amwylo i ysglwpio'r stafaelau, ond mae'n cael eu clwyddoedd y gwirionedd iawn i'w cysylltu'r ddaluniaeth. Mae'r ddaluniau gwirionedd iawn i'r gwyfn monitor. Roedd y bwysig yw'r gwaith y byddai'r syfydel, ond mae yw mewn ysgaf yn y cavell, Yn y gallwn, ydych chi'n gweithio'r sefydliadau? Yn y gallwn, y byddai'n amser i fynd i'r cyfrins o'r programau sy'n cael ei gwybod o'r gweithio. Yn y gallwn ymdweithio'r ymdweithio? Yn y gallwn y syniad, ymdweithio'r gweithio'r gweithio? Yn y gallwn y byddai'n gweithio? Yn y gallwn y byddai'n gweithio'r gweithio? Os ydych i'n dweud o'r cyffredin iawn o'r cyfnod Gwylig Llywodraeth, rwy'n cael ei wneudio cyhoeddfa am gweithio cyfnodau o'r cyffredin iawn, cyhoeddech chi'n dweud o'r cyffredin iawn o'r cyffredin iawn? Yn gyfnod i'r ffordd o'r cyffredin iawn o'r cyffredin iawn, mae'n dweud o'r ddiwedd y ddau cyffredin iawn o'r cyffredin iawn. I like having it all out on the table, but I've seen in some organisations the reluctance to give words to that, and so in one large New South Wales Government department there's a framework that's used for self-assessment across hundreds and hundreds of different sites. And it goes, delivering is the first category, and then excelling is the third category. And then I reckon they ran out of up in the middle ones called sustaining and growing. But there's nothing, there's no definition of what sits to the left in what delivery. So when those sites do self-assess, they can self-assess as this fourth category, which is known and trained as working towards delivering. But there's no descriptors for what that is, because there could be as many reasons for being working towards delivering as they are in different places. But I just wanted to share that as an example of, it's kind of half of full delivery, but that's leaps and bounds ahead of not having anything for self-assess. So it certainly, you know, I prefer having it all out on the table. But that's me, and I'm not the person doing the self-assessment. We know, when I did the WA public sector commission, they were already having problems with their diet of people, and they asked people about problems with that diet of system. It's kind of scratchous, or it's already free and more. So, yeah, there's sense of timidies around that point on the scale I want to think about, and change the language somehow, or assess it in some other way. But yeah, this is all just suggestions. Like I said, there's a question mark at the end of the title of this thing. I might not be completely universal. Some issues with using it, I know it takes a little bit different. They've just not come. Sometimes you can use this word however you construct your word, but you go through this process. I find that it makes self-assessment a little easier and clearer because there's an underlying consistency for the scale points. If you go back to that very first one, if you actually look at this, the only difference between excellence and good is the word very strong. It's kind of shifted the subjectivity in about whether you call yourself good or excellent into the descriptor. The descriptor hasn't actually minimised the subjectivity in my opinion. Because these aren't really anchored in anything that people can easily grasp. Whereas I think this kind of scale is a lot more intuitive, it's a lot clearer about what it is. Oops, sorry. If you make self-assessment, there's still subjectivity to it. This is consistent with the cross-gins. You can also use it to allow other people to make an assessment. If you've got a field team saying all I would use is the same rubric because there's a consistency in the underlying scale. If I think it would produce a greater liability, a little more experts can use it to make performance or whatever it is. Or you can use some hybrid scoring between self-assessment and expert or other assessing. The other thing is you can use the rubric itself as a data collection instrument. Give it out, so please make yourself. Sometimes that can be hard or it can be confrontational as I mentioned. The other way of doing it is you can construct a series of survey questions. For example, they might just be simple yes, no questions. There are many things happening in your organisation. Depending on the sum of yeses and which knows, you then impute what that means for where they sit on the rubric. The people filling out giving you a survey question, you then amassat and synthesise it to determine where that sits. You can bring in other data. Some of the statements might say at least 50% of our programs you just get that from an administrative data, for example. So it doesn't all have to be assessment in that sense. So there's a range of approaches to actually determining where units of analysis sit on the scale which you can construct. Lastly, just some closing observations we might again open up for discussion. As I've said, I flagged it with a question mark, but I have been surprised myself by the range of applications. Again, maybe I'll become a Colton. I'm pushing something in there too. But I'll be interested in hearing from your particular context and the kind of outcomes and evaluations you work with, whether you think that could have helped you and that could have worked in that context on that. That would never work. Now, I can tell you a situation where I wouldn't use a rubric. It's where it's already well established measures for something. If I was measuring recidivism for prison rehabilitation, I'm not going to use a rubric. I won't tell me if I'm wrong with people working in the justice case. I think there's reasonably good reliability measures for that. Economic development. Well, there's some good reliability measures for that. So I think that's one area where I would push it too far. But I'd be interesting to hear from you where you think could be used to fill some measuring gaps. The units of analysis, which are sometimes different from the units of data collection. What do I mean by that? You might want to analyse government agencies and how well they're doing the evaluation. But the unit of data collection are the individuals working in that agency. How do you aggregate the information from the individual so you can say something about the unit? Now, these are ordinal scales and there's mathematical limitations with adding them and averaging and all of that. The 50% of people have to say something and all sorts of issues around that. One of the interesting things about the exercise we did where we had different tables giving different statements is it can expose different perceptions about what's going on and what even outcomes mean among key stakeholders. And that can actually be an interesting thing in itself. So, for the Victorian Department of Education, I got policy officers to write their practice and evaluation and I got senior managers to measure on the same rubric. And guess what? Anyone anticipate that as I see it? The managers draw crap when they ask people where to go? The managers thought things were going well. The policy officers are not so much. But that allowed a conversation to be had with the managers about this is what you say, this is what the ground trips say. What does that mean, what are you going to do about it, if anything? But at least there was consistency about the base scale that allowed that comparison. And that's why I think having this common underlying consistency to the point on the scale that allows comparison. We'll talk about the idea if you have one rubric or multiple rubric, so that a global scale will break up into dimensions and then synthesise the information yourself separately. And lastly, I want to think, the great value I've got out of having these consistent underlying scales is allowing you to link a performance recording and decision making. This is something I've been with Duncan recently, because I was doing this stuff with the State of Victoria, and I developed the scale and Duncan and I said, well what could all of this mean for decision making if you've got this point on the scale? So these are the labels we've used, and sorry about the readings I've created. But for example, if something was absent, so this is capability building among local governments to improve environmental regulation and practice. You would raise awareness and articulate the case of change, introduce the practice into the old description of business process, provide work, examples and training. So what you do if you find a sort of breaking here or here or here, what then follows from it tends to be a little bit clearer. So it's linked to decision making is just that little bit tighter, and that's, I think, one of the really attractive ideas. So that's where I'll finish, and any discussion, comments, thoughts, rub them through. Just on that last one, were those suggestions that our decision making or possible responses were they developed collaboratively? No, it was back in the moment when I was one after this. Oh, is that not collaboratively? It tended to, Joe, when we showed it to them, when I showed it to them, they got really excited about this. They really liked this because it's more than just here are the measures, here are the measures and what you can possibly then do. Just curious, what are the most important things to consider when figuring out how many points on your group that you're going to have? Good question. Well, certainly if I was going to develop a group from monitoring purposes, I would probably extend the scar points. I'd have more between absolutely realised. I think if you're monitoring, tend to want sort of the other bit to take to find small changes over time. So I'd probably have a lot more points here. It really depends on context and how much fine-grained decision people can make when trying to store a group and how much the decision makers make the form of their decisions. So it's not going to get you the magic number, but I think this is probably the smallest set and you expand beyond that. I was almost tempted to put it through when I was here. There's often beginning, making huge progress in almost there and sort of half-way there, but it's not as common as these two. So I didn't put it in one universal scale. Can I ask you to follow my question back? The deep philosophical question about that you offer a neutral, a neutral decision? Don't get me started. I know the use of neutral midpoints. I know the grievance and satisfaction scales. I know them even more when they have neutral midpoints because I don't know what it means. I'm still not sure what it means to be not dissatisfied or dissatisfied. If someone could explain that state of being, it's not a first mark, it's been 20 years I've mounted it. I don't know what it means. I think what's missing really in those is you haven't got off the scale points. It's not applicable, but don't know. Certainly you would have those here as well. I'm going to comment about the middle point because I'm a real fan of middle points. I explained this to people all the time in psychometric terms. It's usually just a scale like satisfied to not satisfied in a five point scale and you've got a middle point or satisfied with your example. People treat that as psychometrically. They look at the psychometric, it's a scale, and they just use it on the scale. The problem comes in when people go and understand your questions. If you don't understand what your question means, they'll go to that, they'll default to it because they don't understand what you're asking. If you get a lot of people happening in that, then you know there's something wrong with your question. Yeah, but even with well worded questions, all the literature on those... All the literature on those scales and the classic, of course, is pulled by question announcers and attitude surveys. They do a lot of split-half experimental comparison. That nipon tends to capture people who are very different. Measurement theory, people who identify with a point on a scale should be consistent with the reasons they're doing it. That midpoint catches people who do it for different reasons. They do it highly because they know what they're unsure but also because they slightly agree but it's not really strong. Or slightly disagree, it's not strong enough that they're pushing the points on the scale.