 I just wanted to talk about the topic, evaluative thinking and reasoning. It's something that I've had on my mind for some time to try and unpack. It's a topic which I think is really about making implicit processes explicit. And there's not a lot that's been written about this in terms of how you make that process of moving from data to what we do with it and what we make from data explicit and open. I think there's a lot taken for granted in that process that somehow it's a miracle, happens, we collect data, we have information and suddenly we're able to make sort of, you know, very substantial findings, recommendations and learnings. The process of getting there, however, is not well understood. And that's what this presentation is going to try and do. It's a framework that I have developed that has made sense to me. I hope it makes sense to you. And certainly as we go through the questions at the end of the presentation, there's plenty of time for you to ask any further questions for elaboration. And also, as Ian said, the PowerPoints will be up on the AS website and due course because there's quite a bit on there that you might flick through a bit quickly, but hopefully if you're really keen, you can look that up at a later date. So on that note, I'm going to do the share screen. Here we go. OK, so this topic is really about unpacking evaluative thinking and reasoning processes in data synthesis and the identification of evaluative findings, lessons, recommendations about program performance. So really the real key concept here is how to unpack these processes of thinking and reasoning. And as I said in my introduction, the purpose of this presentation is my attempt to try and unpack the difference. I should also mention before I start that a lot of terms are used interchangeably in a lot of different contexts. And I've tried to reduce the terms to my definition, but it doesn't mean that in other contexts the terms won't be used differently or might be used differently. And I'll explain as we go through which terms are used in different contexts, but I've tried to stick to the same terminology so that things are clear. And you'll see I very much stick to the concept of thinking and reasoning as being two separate but interlinked processes. So when I started looking at this topic of evaluative thinking, there's a lot that's been written about what's called critical thinking. And this is a standard definition if you look at evaluative thinking of that process. This is a definition by Buckley, Archibald, Hargraves and Trocken 2015. And if you actually look at any definitions of evaluative thinking, this is one that comes up first. But I don't think it's a good definition, but I think I wanted to include it because it's a standard definition that you'll find on this particular topic. So I thought I'd better present it. And this particular definition of evaluative thinking says it's a process of critical thinking applied in the context of evaluation motivated by an attitude of inquisitiveness and a belief in the value of evidence that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking and informing decisions in preparation for action. So the reason I don't like the definition is it jumbles things together a bit and I'm trying to unpack. So I was looking for a definition that's a bit more unpacking like. And I found one, which is good, on this next slide. What I wanted to do was unpack the thinking from the reasoning. And I thought this definition was good because it talks about data and evidence and argumentation as being thinking and professional judgment as being the reasoning. And I actually follow this all the way through. That thinking is about data and evidence and argumentation. So it's about the logic of how you deal with data, whereas reasoning is the professional judgment that you make based on that data. All the way through this presentation and through the framework, you'll see that I separate out thinking from reasoning, but also try to show how they're linked, how the two mutually interdependent processes. And you really can't do evaluation without doing both, both logic and judgment. So this definition I thought captured better. It's a definition by Anna Vo and Martin Elkin, 2013. And I think it's a better definition in that it actually separates out the thinking from the reasoning. It talks about basically evaluative thinking or reasoning being a particular kind of critical thinking and problem solving approach that is germane to the evaluation field. Specifically, it's the process by which one marshals evaluative data and evidence to construct arguments that allow one to arrive at contextualized value judgments in a transparent fashion. And you can see I've underlined the marshals evaluative data and evidence from contextualized value judgments because to me that first one, the evaluative data and evidence is the thinking and the contextualized value of judgments is the reasoning. So I like this definition because it separates the two and talks about them as beings that pretty much interlinked. So when I was trying to think about, okay, why did I think it was really important to separate thinking and reasoning? And this is where I went on a really interesting journey of thinking about why I thought they were separate but interlinked processes. And then I did some research and found out the reason why I thought they were separate is that there's this notion of deductive and inductive logic. And deductive logic is the thinking and inductive logic is the reasoning. And I thought this was really good because I then realized why it was important to actually separate thinking from reasoning. So thinking, evaluative thinking is critical thinking and both those definitions affirm that that's the case. It's systematic inquiry. So it's very logical. It's analytical and it's deductive. And basically it's called theory testing. And I'll talk a bit more about that in a minute but it tests the theory. So we have a theory, we collect the information, we test the theory, we deduce a conclusion from that. And it's called top-down theory which I'll talk about more in a minute. Evaluate reasoning is bottom-up. So it's about sense-making from the data. It's more interpretive and it's more inductive and it's more about theory development. So this is why the two are so important together because when you're doing an evaluation, you're both testing the theory but you're also developing a theory at the same time because you're testing the theory around which the program has been based on. But through that exploration, you're often coming up with alternative theories and logics to the one that was actually developed for the program. And you're often recommending a different type of way of representing the program on the basis of the evaluation. So why don't you've got to test a theory? You've also got to be open to retesting or redeveloping that theory at the end of the process. So it can be both what's called bottom-down, top-down and bottom-up at the same time. I quite like this slide here because this made a lot of sense to me when I was trying to make sense of this notion of deductive and inductive logic that Aristotle was the one that developed deductive logic. He developed a formal system of top-down logic. And I won't go into it in great detail. If anyone studied Aristotle, you might be more versed with it than I am, but it's basically about sitting a hypothesis and testing the hypothesis and drawing a conclusion based on that. And Aristotle was really the one that developed this notion of deductive logic. However, if you look at Sherlock Holmes or any other detective for that matter, I read a lot of detective novels. So they all use what's called bottom-up logic. In other words, they go out, they look at what's happening, they collect the information, they start to draw together tentative hypotheses, and at the end of the process, that's when they develop the theory of who done it usually, right at the end, not at the beginning. So a good detective would never enter into detective work by putting their theory of who done it upfront. They would develop their theory of who done it after they've done their data collection and got their information and data to establish who they believed did it. Whereas Aristotle, in his top-down logic, would have a hypothesis about who he thought did it and then test that out. So this notion of deductive and inductive logic really helped me understand the difference between evaluative thinking and evaluative reasoning and gave me a much better way of actually conceptualizing it. I'm hoping this makes sense, but I thought Aristotle and Sherlock Holmes were good examples of the origins of these two different concepts of top-down and bottom-up logic. They're both logical. They just approach that logic in a different way, as I said, in terms of whether the hypothesis is at the front end or it's at the back end after the data's been collected. So in terms of top-down, bottom-up, so with deductive think, which is the evaluative thinking, it's top-down. You have the hypothesis upfront. You test the theory upfront. You generally stick with the facts. It's very logical. You basically follow a very... Well, deductive process, obviously. In that sense, it's quite compatible with quantitative positivist methodologies because it's very much about truth and will emerge if you follow a very logical deductive process. So in deductive thinking, as I said, you have your theory and proposition. You collect your data. You analyze and synthesize your data. You confirm your hypothesis, new draw your conclusion. In inductive reasoning, it's bottom-up. So the theory development is based on making the connections. In that sense, it's more compatible with the qualitative or the interpretive-vist type approaches. In the inductive approach, you're using data collection first to really understand what's going on. You analyze and synthesize the data and then you see patterns, connections, and generalizations emerging from the data. And from those patterns, then you can draw your theory, your findings, or your paradigm. And again, my argument in this is that evaluation needs both. It needs a really logical process, which is based on the facts and it tests the facts, well, the facts are tested against the propositions, but it also needs that inductive process of drawing the pattern's connections and generalizations. And so trying to understand where one ends and the other one begins is something I'm going to try and unpack a bit more. Because I think it follows a fairly linear pattern that you actually follow the deductive process in your data collection analysis, synthesis, and conclusions, and then you move to an inductive process when you get to findings, recommendations, and learnings. So that sort of continuum, I suppose, from data analysis to findings, learnings and recommendations, I'm going to unpack that a little bit more. I'm also going to provide a case, little case example to show how that's done. I think it's really useful to show, it's just hypothetical case, how that plays out in a hypothetical scenario. So talk about unpacking. This is the sort of little model here about all the emphasis we place in evaluation on collecting data. We often have program databases that collect and monitoring data. We have data reports, monitoring reports. We collect evaluation data. We have often evaluation data management systems. We produce monitoring evaluation reports. We have special study reports. But when we actually get to the reporting, there's this Xbox there, which is the actual unpacking I'm trying to do. What happens when you've got all that data from all these systems, from your monitoring systems, from evaluation systems, coming together? What do you do with it? And I've heard the term death by data quite a few times. And I think a lot of evaluators have experienced that at times when all the data comes tumbling in and we don't quite know what to do with it. And we don't know how to handle it. We don't know how to manage it. We don't know how to actually put it into any form that is actually logical and sequential. And often in really poor evaluation practice, the data becomes so poorly handled that whatever findings, conclusions and recommendations that draw and barely relate to the data that's collected. So that's what we don't want. We don't want to see that happening. We want to see the data that's collected, channeled and funneled into a process of evaluative thinking and reasoning that actually is logical, sequential and transparent. And a lot of the definitions I gave you talked about the importance of transparency, that we can go back and actually explain how we got to the end, how we got to those findings, recommendations and conclusions. We can actually go back and track that through the processes we've used and therefore justify it when those findings, conclusions or recommendations are challenged. We can actually justify how we got there. So unpacking is a really important part of I think our own professional development to be able to understand how we got to the point we got to. So let's go back to the unpacking. The unpacking of factor X. So as I said, we move from deductive to inductive. On this line on the right, you can see the deductive to inductive sort of movement paradigm. So when we're doing deductive evaluative thinking, we're looking at data analysis, data synthesis, data conclusions. The beginning of assessments and judgments start to become more inductive. And certainly by the time we get to findings, recommendations and lessons, we're very inductive. So we're moving along a continuum from dealing with the data in a very logical, top-down deductive manner to moving through an inductive process where we're actually making judgments. And the assessments and judgments, I've sort of got them off to the right because they sort of join together the deductive and the inductive. So we're basically sort of moving along, but when we finished our deductive process and we come to data conclusions, we need to make some assessments and judgments before we can move on to findings, recommendations and learnings. It's interesting, I'll talk a bit more in a minute about Michael Scriven's logic of evaluation, but he really talks about this process of assessments and judgments as being what evaluation is all about. That evaluation is the core of evaluation is this, the assessments and the judgments that we make. So I'm trying to actually unpack again how we got to that point, how we actually joined together the analysis, the synthesis, the conclusions with the findings, recommendations and learnings is really critical to our own ability to sort of talk about credible reporting and the fact that we've actually undertaken a credible process that we can justify. So let's have a look at the paradigm here that I've got. Now, I should actually probably even have prior to the evaluation questions to the program, theory and logic from which the evaluation questions are derived, but that's implicit that the evaluation questions are based on a theory and logic, but there is then a flow. So let's assume the evaluation questions are very much testing the theory and the logic of the program that's in there, there's a flow through. So from the evaluation questions, we develop a methodology about how we're going to answer those evaluation questions. We collect the data, we manage the data. That means we sort of put it through various databases or ways of organising the data. We analyse the data and by that, I mean we might analyse the data in silos. So we might analyse our surveys, analyse our focus groups, analyse our semi-structured interviews, etc. in silos. And then we synthesise that across those different data collection methods. And then we draw conclusions. This is the deductive process. And you can see I've got the yellow arrow there, trying to separate deductive from inductive or thinking from reasoning. So the thinking process is the logic of collecting your data, analysing it and synthesising it and drawing your conclusions. And in a minute I'll show you how why it's deductive is because we're actually analysing the data against some constructs. So we're testing some hypotheses, which I'll talk about in a minute. Then we move to the inductive, which is the assessments we make against the evaluation questions, going back to the theory and logic, including the assumptions, making those judgements. Which I talked about the assessments and the judgements. I talked about as being the sort of the E of evaluation. And then from there, the findings, the extent the program was appropriate, effective, efficient, sustainable, had impact was of quality and value, which then lead to the lessons and recommendations. So this is the flow through I'm talking about. The flow through from the evaluation questions right through to the findings, recommendations and lessons. If those recommendations are disputed, which they often are in evaluation work as we know, hopefully you can go back and track back how you got there through that process, through this process of being able to be transparent about how you move from data synthesis through to findings, lessons and recommendations, and how you move from conclusions to judgements. So again, as I said at the beginning, some of these terms are used interchangeably. Certainly, I'm probably gonna ask questions about why I've chosen these terms when they can be used in different ways, in different contexts. I've chosen the terms this way because they make some sense to me and unpack things in a way that I can explain it to others. But clearly you'll see these terms used slightly differently in other contexts. And people will often say all conclusions you can make at the end of findings as well, or you can make conclusions at the end of findings. These conclusions in particular I'm talking about are conclusions from your data synthesis process. So that's just how I've used the terms. As I said, I just wanted to mention here Scriven's logic of evaluation because he came up many years ago, obviously, with some of the components that I've been following through with, not exactly in his methods, but certainly some of the concepts. So his four steps of the logic of evaluation were about developing a criteria of merits and standards. If you're gonna follow a deductive process, a deductive process using evaluative thinking, you've gotta have something to measure against. You've gotta be able to measure performance against standards. Remember, the deductive process is fairly dispassionate, I think I'm trying to think of a word, dispassion in a sense that you're not making judgments. You're purely assessing what is or what isn't. So in order to do that, you need to have criteria of merit and standards or something that you're measuring against in a dispassionate way. In other words, you're just saying, did it happen or didn't it happen in the way that was expected? So those first two, the criteria of merit and standards and the measuring performance against standards and the synthesizing of the data is very much the evaluative thinking component or the deductive component. Then Scriven goes to talk about rendering judgments, which is as I said, the fundamental core of evaluation, which is the inductive evaluative thinking process. And he talks about those judgments being based on evidence and the application of values, beliefs, expectations to that data. It's quite interesting because some other theorists have said that there's more to it than just values, beliefs and expectations that the evaluator brings in professional, know-how, professional credibility. And others talk about the fact that it's not just the evaluator, it's the stakeholders as well that are making judgments. And we were undertaking a process of sense-making, which involves not just the evaluator, but a whole lot of other people involved in the program. But the Scriven notion that we are actually rendering judgments, whether it be through an evaluator applying values, beliefs and expectations or through professional know-how, experience or through stakeholders involved in a sense-making process, they're all options. But the basic premise is we're going to be rendering some judgments, no matter which way we do it. That's what evaluators do. So I'm going to now go through the processes of looking at how we move from evaluative thinking to reasoning and that continuum. And the first process is what we call data synthesis, which is the first process we use. And I just want to show you the slide because stuff will be in Shinkfield in 2007, basically said data synthesis brings together a range of different types of data, collected at different times with different means, it's a highly challenging activity. So even this part of the process is not easy to synthesis. This is bringing together all these methods that we've used in a mixed methods approach, the qual, the quant, the other data collection that you've undertaken and trying to make sense of the variety of data collection sources you've used. In an ideal situation, if you're lucky, the data will coalesce and confirm itself with each other. In other situations, it can be contradictions in the data with one form of data saying one thing and another saying another thing. And then you've got to really try and make sense of that. But the data synthesis process, hopefully can create enough synergy between the different data sources to come up with those conclusions. So as I said, I'm going to be using a bit of a case study to try and unpack how data synthesis works or evaluative thinking works and then look at evaluative reasoning, which leads to the recommendations, findings, conclusions that we draw through those processes. So this little program which I made up is called the Street Art Program for young people who are not attending school. And as I said, it's totally fictitious. So please don't critique the program. It's made up, totally made up. So the first step we're going to look at in applying the data synthesis to this program is what frameworks are we going to use? What are the standards? So looking at Scriven's notion of standards and criteria or criterion standards, that comes from the analytical frameworks called evaluation rubrics, the notion of criterion standards. But there's a lot of other ways we can set standards or criteria for assessing performance. There are other sorts of analytical frameworks that are not rubrics. There are obviously indicators and targets, which is probably something we're most commonly presented within our evaluative work. There are industry benchmarks. There are performance checklists. There are quality standards. There are accreditation standards. All these things provide us with some sort of checklist against which we can do that data synthesis that will lead us to the conclusions that we need to draw about the data. So in Scriven's model, and in model I've been presenting you with, you have these criteria or standards upfront and agreed and then you collect the information through a multi-method approach or a mixed methods approach. And then you actually have to assess that data against those criteria and standards. For the simplicity of this presentation, I'm just gonna use indicators and targets. You know, it's the simplest and it's probably why indicators and targets are so popular, because it's the simplest, most reductionist way of actually trying to develop a set of standards. They're not my favorite often, but if I tried to show you against a rubric or an industry benchmark or any of the others, you know, it would be a much longer seminar than we have time for. So I've just chosen indicators and targets to use as those benchmarks. So with data synthesis, you know, the question is how do you synthesize data? What conclusions would you draw from the data you've collected? And this involves deductive, evaluative thinking, sticking with the facts, testing the proposition on which the program was based. So in this little case study, I'm going to show you this slide on this street art program and I've used an indicator, which is participants from the target group of non-school attending young people. So this particular fictitious program would anticipate a mix of attending and non-school attending, but wanted a 75% target for the group to be non-attending young people, although it was allowing a mix. In the monitoring data we found out that actually 30% of the target group, only 30% of the target group were non-school attending and 70% were school attenders. So we actually didn't, as you can see, reach that target quite the opposite. We only got 30% of the target. The monitoring data found however, that of the people that did participate, there was a high degree of satisfaction from all participants, be they school attending or non-school attending. The monitoring data also found out that the school participants actually did reduce their court appearances and that the school attending participants increased their results in fine art subjects. Remember, this is a street art program, so they were able to take their street art skills back to their art class and do well in art. So the evaluation data that we collected through semi-structured interviews, focus groups, found that non-school attenders were difficult to locate and itinerate, that non-school attenders that they contacted did not see the program format or content as relevant, that young people who did participate found the program, all young people who participated found the program positive and the facilitators found that the people that participated engaged well and they got good results from everybody who was there. So we've got a whole cluster of data here and we've got an indicator and a target to test. So again, this is the dispassionate process of trying to synthesize the data in order to reach a conclusion. So the data synthesis is bringing together the monitoring data and the evaluation data and trying to meld it together, I suppose. And it says here that as a result of synthesis, the comment is the program engaged 30% rather than 75% of the target group. 70 targeted non-school attenders were difficult to locate, did not respond well to the format and content. The participants engaged well, were satisfied and positive about the program and the non-school attenders reduced court appearances and school intenders improve their academic results. So here we've pulled together all the data. Obviously in the real world, we'd probably have a lot more data to meld together but this is just an example. So the conclusion we draw and this is the evaluative thinking conclusion is the program did not reach the intended, sorry, the program did not reach the numbers of intended participants expected. The program was not successful in locating nor attracting the majority of out-of-school young people and those who participated had positive experiences resulting in good outcomes. So here you can see the dispassionate side of it. It's basically telling you what happened, what's what against those performance indicators. It's not making judgments. It's not saying why or why not. It's not giving causal explanations. It's not trying to understand why or explain why things happened. It's just telling you what. And but however, having done that, this is really important. We need to know what before we can ask why or why not. So this is the what and the what is really important and it's very logical. As you can see, it's a logical process. We're using the criteria or the standards that we've agreed to at the beginning of the program. We've collected the data, we've synthesized it and we've drawn a conclusion based on the data. Clearly, this is not where we're going to stop because we're going to go on to judgments that will lead to findings. But it's the first important part of that process of evaluative thinking. So we're thinking in a logical way. We're using a sort of way of reducing data to its elements to be able to understand how the program's performed against expectations. And this process, I think if you use this sort of diagram, this sort of diagram, this sort of table, is quite a good way to actually map out the thinking process in terms of making it explicit how you have arrived at this thinking process. And using a chart like this is often a good discipline to keep that as a record of how you've got to conclusions. I remember when I was studying evaluation with Ros Herworth, she was always really keen on us charting and recording our data and being really sort of clear what the data was, writing it down, putting a highlighter pen in those days. Which data we were using for what conclusions and having that as a record. OK, so findings are really what have you learned from the data. They're judgments about the meaning of the data. They're providing possible explanations for the patterns emerging in the data. And there's establishing calls of connections evident in the data. And this involves sense-making of the data. Again, this is what Scriven calls rendering judgment. And this could be with or without stakeholder involvement. Scriven, as I said, talked about rendering judgment being based on values, beliefs and expectations. Other theorists talked about professional know-how and experience. Other people talk about stakeholders and sense-making. And I said before, it doesn't really matter to me anyway, whether it's the evaluator rendering judgment based on values, beliefs and expectations or whether it's through professional know-how and experience or whether it's through stakeholder sense-making. The main issue to me is that it's transparent and able to be unpacked and explained as to how it happened or how it took place. But as I said, it's a very important process, the rendering judgment process. It's all about meaning and explanations and patterns. And again, if we look at Sherlock Holmes and he has collected the information, he wants to know, I've got all this information in front of me, patterns are beginning to emerge. I'm seeing some calls of connections here. And that really then leads to the good detective being able to find out who did it. And as I said, having been reading nothing but detective novels while we've been in lockdown, they all use very similar methodology of immersing themselves in the data until such time as the facts start to coalesce and they start to see patterns that lead to them to be suspicious of whoever did it. So, moving on to findings, recommendations and learnings. Again, I want to talk about how do we approach the sequence developments of finding recommendations and learnings. And this involves inductive evaluative reasoning, making credible, connective propositions that emerge from the data. In this process, as I said, I've split it up into findings, recommendations, learnings. I think more and more people or commissioners of evaluations are asking for all three for the findings, the recommendations and the learnings. The findings being, obviously, what is your conclusion from the data? The recommendations, what should we do about it? And the learnings, what does this mean for the future program or for other programs that come along that may learn from this particular program? I think it's a great thing to have learnings included. And I'll show you how that's done. Again, I've drawn up a little diagram that splits the findings recommendations from the lessons. And this is based on, remember the conclusions that we drew previously on this program, this street art program. So the finding, and I've got three sets of findings, I should say. I mean, obviously I could have many, many more, but I've just got three. The program was not able to recruit the expected numbers of out-of-school young people as the target set appeared to have been overly ambitious. So this is where the judgment comes in. So I've judged, or whoever's involved in my judgments, whether it be stakeholders or whatever, have judged that the targets were overambitious. And that's why we didn't reach them. Another judgment there could be possibly that the targets were achievable, but the program didn't try hard enough. But in this case, I've made a judgment that the targets were overambitious. So the recommendation is to reconsider the target numbers and reduce the number of out-of-school young people expected. The lessons are, pilot testing prior to launching a program is important to ensure it can attract intended target group members. And participation targets should only be developed after the program has been operational for 12 months to test what is achievable. Yeah, so you argue with these lessons, whether they're good ones or bad ones, but for the purposes of this exercise, that's how I think it looks. So let's have a look at another one. The finding. The program did not appear to have successful strategies for locating people out of school, nor appealing to them in terms of program format or content when they were located. So again, this is a judgment that the successful, the strategies were not successful. They could have done more and they certainly could have done more to appeal to the young people in the format and the content of the program. So again, I made a judgment here that they probably didn't do enough. I mean, the first finding was that the targets weren't realistic. The second one is they still could have done more than they did. So I have a sip. So the recommendation here is the program developed strategies for locating young people, possibly through alliances with youth workers and youth organisations. And the program consider ways to promote the program to attract the target group. And the lessons forging relationships with key referral networks, such as youth work agencies is important to achieve targets and publicity and promotional material should be tested and customised to attract the target group. Last one, while she's two more findings, so I said there were only three, but there's actually four, is that those who did participate had positive experiences and reported good outcomes, therefore demonstrating the value of the program, particularly in reducing court appearances for the out-of-school target group. And positive outcomes were achieved for school attenders and the positive outcomes achieved for school attenders were not part of the program design. So again, this is a judgment here that the program methodology and approach was effective. So the young people who did participate who were out-of-school did reduce their court appearances. So I'm judging here that the program was of value, that its approach was valuable for the people who did participate. It wasn't up to scratch in terms of attracting the target group, but certainly when they were attracted, the outcomes were good. However, there's also a judgment here that the outcome for the school attenders was not part of the program design. So here I'm sort of more of saying, well, that's good, fair, it's good that they increased their results in fine arts, but that's really not what the program was set up to do. So the recommendations, programs should maintain content integrity, given the degree of participant satisfaction and success in achieving desired outcomes for participants. So here we're saying the content was good and should be continued. However, if greater numbers of target group members participate, the format and content should be monitored to ensure ongoing applicability to new participants. So here we're saying, you know, it's worked well for that group, but if you get more of that group, something might shift. So maybe it only worked well because that group were in small numbers, but if you start increasing the numbers of young people who are non-school attenders, then you might get a different result. So the lesson here is maintaining content integrity is important, but the applicability of format and content should be reviewed when participant characteristics change significantly. So I just ran you through those just again to show the sequencing between the findings, recommendations and lessons. And if you track back to the previous slide with the conclusions that were from performance against the indicators and targets, and you put all that together in one table, you can see a really logical flow happening there. Moving from the thinking to the reasoning, moving from the data synthesis through to the conclusions leading to the findings, recommendations and lessons. And as I said previously, the transparency of all that is really important. To be able to, obviously it's not gonna be easy to map all this out necessarily on sort of, you know, sheets of paper that, because there's often too much data to do that, but you can distill it to its fundamental components and map that out. So you have a table that moves between the data synthesis, the conclusions, the findings, the recommendations and the lessons. And they call that a data display. Again, I'm taking that term from Ross Hurworth, but if the data display is displaying your data so that someone can actually pick it up and say, oh, I can see how you got there. And just being really aware, I think that you're moving from a top-down deductive process to a bottom-up inductive process is really important. So you're moving from leaps that involve very sort of much sticking to the facts, looking at the information you've collected against those indicators and targets to moving into judgments that you're making on the basis of your experience, your knowledge, your values, or stakeholder involvement or whatever processes you're using to do that. As I said, there are sort of three different bodies of theory around making judgments or rendering judgments with Scriven talking about it being evaluated as prerogative or responsibility primarily. Someone like Michael Quinn Patton may talk more about stakeholder engagement and sense-making and the fact that the judgments should be shared with the stakeholders. Sure, Scriven wouldn't agree with that. It would see data sort of possibly jeopardizing the independence of those processes. And others who say it's more than just the values, it's also your professional know-how and your experience. And your sort of values, I suppose, around independence and objectivity and how you manage data. So there are different theories about judgment and how it happens and how it should happen and what's behind it. And so it's sort of worth thinking about, and it might be on different projects who use different ways of making judgments. So on some projects, the judgments might be yours alone and other projects, it might be a stakeholder sense-making session and in other situations, yeah, you might involve different ways of doing that. At the end of the day, what we want to achieve is credible evaluation reporting that can stand the test of anyone's, I suppose, questions about how credible the report was. As we know, evaluations are very political process and evaluations get challenged, particularly if they're coming up with findings, recommendations, lessons that aren't what the client or commissioner expected. So credible reporting is a way of actually displaying the deductive inductive thinking and reasoning processes. So when you're writing an evaluation report, you want in the deductive thinking to be able to talk about what, how many performance against expectations, criteria, standards and targets, you wanna show transparent assessments and conclusions based on that. And then you wanna be able to move into the inductive space and talk about why, why not, possible causation explanations and be transparent about your judgments, your findings, your recommendations and your lessons. And again, to be able to really move between those is a really important part of the process and to be able to display that in evaluation reporting is really important. I think in terms of where these would arise in your methodology section, you clearly would show how you analyzed your data, what conclusions you drew from the data, how the program performed against expectations. You would show you conclusions against that and then you would move into that inductive space of talking about the judgments you made and the findings, recommendations and lessons. And, but also showed the sort of the, the continuum and how that, how that they were closely intersected. I think in, we've seen this, oh, I have anywhere in my practice, disjunctions between the deductive and the inductive process where the facts are presented, the information is presented and suddenly there's this big jump into findings, recommendations and lessons that don't appear to be necessarily connected to what came before in those data assessments and conclusions. And that's what we don't want to see. We don't want to see a disconnect between the two processes. They have to be linear and a continuum, not sort of disconnected. So I'm losing my voice. Okay, so why is this all important? Because, and I like this quote, it's an old quote from Schwartz and Main from 2005, but it really talks about the fact that we've got a rule and it's a long time ago, so 2005 we're talking about, you know, 20 years ago, 15 years ago. But even then they were concerned about evaluations and reports, uses tactical weapons in political and bureaucratic skirmishes. And they talked about a risk of credibility crisis regarding evaluative information and how to be misrepresented. And I think in this process of unpacking evaluative thinking and reasoning, we're trying to increase the credibility of our evaluations. Sorry, my voice is going to pack up. One more sip. So the credibility of evaluations is very much based on the ability to be transparent about the processes we used and to be able to display those processes in a really open way that can actually protect the credibility of the report and stop it from being used as a tactical weapon in political bureaucratic skirmishes by being able to justify how we got to where we got to. Also being able to talk about the fact that we used a very logical process and a very transparent reasoning process to get where we got to and that that's what evaluation is and that that's how it works. And it's really important we preserve that because that's the foundation as Michael Scriven would say, is an ability to make evaluative judgments and to stick by those judgments and for them not to get misrepresented or in any way used as a tactical weapon. It doesn't mean that you won't be challenged, it doesn't mean that it's... No matter how logical you've been and how much you're able to defend, much have come up with that it's still not going to be challenged, but hopefully the more credible the report is, the more robust it's going to be in being able to defend itself and you can point to where the data came from to get to where you got to. And one last point I want to make is I once did a... gone to a couple of consultancies for the Office of Evaluation and Audit, National Office of Evaluation and Audit and what they would do is they would actually number every paragraph of their data, of their, I suppose, of their findings or their information, their numbers and then when they actually drew their findings and conclusions they had to refer to the numbers in the report that those findings and conclusions were based on and that was tedious. It gave me a really good discipline way back then in how to actually make sure that you can actually track back your findings, recommendations to the data and you can actually show where in the data you derived them from. So that's basically it in terms of the presentation and I think I'm almost right on time, apart from a slight loss of voice towards the end, but I think the main takeaway message is to be more conscious of the importance of process of moving from evaluative thinking to evaluative reasoning in a logical sequential way to be aware when you're moving from thinking to reasoning, when you're moving from deductive to inductive logic, how you're moving between them and how they're sequenced so that you can link back the conclusions that you've done through data synthesis through to the findings, recommendations and learnings in the way that I did in those sample slides and I just want to make a point, those slides are not perfect, there are probably things in them that others would have expressed differently or phrased differently, but it was just my way of showing how you separate findings, recommendations and learnings from each other and also how you derive conclusions from data synthesis. There's probably going to be a request for the slides because there's a lot on them and as Ian said, they will be available in due course on the AS website if you want to go back and check those over. So on that note, I'll just stop the screen sharing and hopefully Ian will come back. There he is. I'm back and thanks very much, and for the great presentation, very thorough presentation and thanks very much for the challenge for getting us to be more reflective about the thinking and the reasoning processes and where they fit into the evaluated process. We're going to move into question and answer session in just a second. I'll give you a chance to offer to recover your voice and you can do a bit of talking. That's the whole purpose. But I just like, just before we do that, I'd like to remind everybody that if tonight's seminar has wedded your appetite regarding evaluated thinking and reasoning and we'll be delivering a workshop on this topic for the AS across two mornings, Monday the 12th of October and Monday the 19th of October. The workshop will involve many practical exercises in actually applying these concepts and you can see the AS website for further details. Now, to move to the questions, you've got a number of questions rolling in. So we'll see how we get through and thanks for the questions so far. And if you have any further questions, please, please type them in. The first question and comes from Sue Mudgett and I'm just grabbing it here. And it's, if you could explain more about the differences between recommendations and lessons. Okay. That didn't come through clearly from the presentation. I thought that might, you know, be in those sort of anyway. I'll explain it again, but recommendation is what the program can do differently itself and the lessons are what can be learned from that for future programs, other programs that are similar, programs that might be outside that organization. So the idea of lessons is that they might be applicable. For instance, this is a youth out of school youth program. Other out of school youth programs would want to know, you know, what the lessons were from another program that operated perhaps in another location that they could learn from before they do their program design. So, you know, in the ideal world, new programs will look at lessons from other programs before they design their own program and certainly the auspice organization will learn lessons for future programs of this type or other programs. So it's really about transferable lessons that can be applied more broadly and either within the same auspice or with two other organizations or other sectors. Thanks, Anne. And you've actually got another question asking about the differentiation between different terms. In this case, it comes from Michael Tynan. And Michael likes to know more about the difference between findings and lessons. And he actually questions about whether a lesson is also finding. What is the difference between... Well, it is a type of finding. Yeah, it probably is. But it's a finding that's applicable beyond the program. So the finding column is really just pertaining to that program. The lessons are, you know, bigger, more, you know, a wider remit beyond the program. But they are a finding in themselves, a finding for programs of this type. But the findings are very much particular to this particular program. The lessons are, as I said, broader remit to other programs or other sectors. You know, you can even have lessons. I mean, those are lessons there. Some of those are applicable to anything, really. I mean, the first one, pilot testing a program prior to launching. And I mean, that's applicable to any program. Or patient targets should be developed only after the program has been operational for 12 months to see what's achievable. That could be any program. So some of them can be almost applicable to any program, not even of that type. Whereas the findings would be very specific to that program. Okay. And it seems like you've got a few people interested in the terminology. Hailey Wainwright asked about the differentiation between a number of terms. But the one that we haven't covered so far is conclusions. Maybe you'd just like to go over that again. That's a difficult one because you can use conclusions. If you look at a dictionary definition of conclusion, it's the end summary. And that could be used at the end of your inductive process or your reasoning process as well as the end of the thinking process. I've used it at the end of the thinking process because I think when you've done your data synthesis and you've looked at your data against your benchmarks, you have to draw some conclusion about whether the program met expectations or didn't. So in this context, I've used a conclusion particularly looking at the data that you've collected and whether or not it met expectations against what the program was meant to achieve. And the conclusion is based on that end of the process rather than using the conclusion at the end of the findings. But you can also have a conclusion right at the end as well, like a final conclusion. So conclusion is a bit of a generic term, but I've only tried to separate conclusion because there has to be some assessment at the end of your data collection against the targets, the indicators and targets or the benchmarks that have been agreed. And there has to be some way of pulling that together. So I've called that a conclusion. Then there could be other terms that could use like a data synthesis summary or any term, but it's a way of pulling together the data synthesis against the performance indicators or the other benchmarks of performance, which is just a way of ending that part of the process off and then launching into the findings recommendations and learnings. Okay, the questions are still rolling in and this question comes from Mary Welsh and it's about the theory of change and where that fits into the schema. The question specifically is would you define developing a theory of change as a deductive process based on propositions and assumptions on which a program is based or is it an inductive process or is it both? That's an excellent question because it is both, isn't it? If you look at the top-down-bottom-up approach, it's both. So the top-down approach is you test your theory of logic in a top-down process. So your theory of logic is upfront and you're testing through the data whether or not the theory of logic were correct or whether the outcomes that were identified in the theory of logic were achieved in the time frames that were expected or whether the changes were evident in the way that was outlined. But then at the end of the process, after you've collected the information, you often go back and you review the theory of logic and reshape it and say, well, actually, it wasn't really that correct to begin with and we need to reframe it. And this is where theory of logic needs to be really dynamic and not locked in concrete. And I think most of us are aware that theories and logics evolve. So at the end of an evaluation process, hopefully, you'll also possibly come up with recommendations around the theory of logic and how it should be reshaped, reconfigured so that it really is at both ends. It informs the deductive process and it could be the result of an inductive process. Remember that the deductive process is theory testing and the inductive process is theory development. So really both are required. You need to test the theory and then as a result of that, you will develop a new theory or refine the theory you originally had. Yeah, so really both, yeah. Great. A question now from Josephine Haussler. And she asks, could you use an audit trail, physical and intellectual, to map out how judgments were made? In your experience, what are audit trails used in evaluation? Is it something that an evaluator might share with a commissioner in a report, such as in an appendix? That's an excellent question. As I said, when I did the audit trail for the jobs I was doing for the ANAO, I was really in the time, really frustrated by it because, you know, numbering every paragraph of the report and then having to connect the recommendations with the numbers, you know, I was really tedious to see why we were doing it. But in retrospect, it was a really good discipline because with the Auditor General, obviously with the ANAO, obviously they're very keen to make sure that they can justify their recommendations and stand by them. So that audit trail was really important to them. And I think, you know, I don't know, it's a hard one because I think at the time I found it a bit, because the way we write evaluation reports, like every paragraph stands on its owner's data and there's a bit of a conversation. It's a bit of a flow. It's not like it's compartmentalised very easily, which is why I didn't think numbering the paragraphs was really that good. But there are other ways to do it, other ways to have an audit trail that basically show your findings and where you got them from in the report. And I think, yeah, I think it's quite a good thing to do. And I think if you're in a particularly politically charged, politically charged context where you think you're going to have to justify your findings, I think it's a great thing to do to be able to put an appendix in the maps where you've got your findings from in the body of the report. So maybe you can have your findings in one column and the page numbers or the sections of the report that led to those findings. The mystery bit is how do you actually make explicit how you arrived at those findings when you're actually making an interpretation, you know? Like, for instance, in the example I gave you where I made an interpretation that the targets were too high for this program. That's an interpretation that I had made that I believed they had that targets were unrealistic. That getting 75% of a target group along was too ambitious, particularly if that target group were itinerant non-school attenders. That's a judgment I've made based on my experience, based on my knowledge of programs, based on possibly evaluations of other like programs. And, you know, so in that sense, you know, my own experience has come into that. Possibly my own values have come into that that I think a mixed group is better than a group that's loaded with non-school attenders. All these things are hard to make transparent. And they're the ones that get at the really sticky point when the commissioner says, well, why do you think those targets are too high? What makes you think that? Why have you written that as a finding? And I say, well, yeah, and that's very hard to make articulate. That's based on my experience. It's based on my knowledge. It's based on my understanding of programs. And it's also based on the fact that it's so underperforming that you only got 30 young people instead of 70 that it's unlikely you'd achieve 70. So these are all harder ones too in any sort of audit trial to really make explicit. And, you know, if you really pushed against the wall, it's quite hard to actually say, well, this is just my judgment. And that's, you know, and it's an unbiased judgment. I have no agendas here. I have no reason to push for one thing over another, but this is my judgment. It's also with findings that, you know, the evaluator's position, they don't have to be accepted by the client. If they disagree, yeah. And the questions keep rolling on. In this case, it's a question from Amanda. Amanda asked about judgments and the influences upon them. Actually, she says, to be explicit about judgments, what extent do you summarize the influences you relied upon to make the judgments? I recall that you talked about a number of possible factors which may come into the judgment, such as professional judgment or professional background. Well, professional know-how, they call it. Know-how, know-how, yeah. Now you define that, I do not know. Professional experience and know-how is one theorists, yeah, take us. I said Scriven was talking about values and what were his three? I forgot now to look it up. He had three that he believed were behind judgments, which were values, beliefs and expectations. You know, others, as I said, talk about professional know-how and experience. You know, I don't know, it's a really hard one because I think it's so implicit and so ingrained in your professional practice that if you're asked to really explain how you got there, it's not always easy, which is sometimes why it's not a bad thing to have stakeholders involved in those judgments. So it doesn't just rest on one person or one small group of people who are then pushed to explain how they got there. It's actually something that can be shared with a bigger group of stakeholders who can take some responsibility for those findings and give it a bit more, I suppose, what's the word for it. Back up that the findings were emerging from a group think, not just one person. I think there's a lot of pressure on the individual evaluator if they're doing this alone to justify those findings, even if you've done it in the most logical way possible without any bias, without any pressure going in a certain direction to be able to justify how you got there can be quite a lot of pressure. So maybe that shared experience and most of the time I've extracted findings, I've done it in a group, not always with a group of stakeholders, but usually at least with a group of other evaluators. And we might have a day or half a day set aside to look at the data and come up with the findings and we do it together and that creates a bit more checks and balances, but I think that it wasn't just one person's take on the situation. All right. Now, the next one is from Rebecca King and she's partly commenting on her experience, particularly in international development and asking a question alongside that. She says, often I've found that the process of moving from standards to the lessons can be quite circular, arriving at a lesson that is actually conformational of the indicator or standard set, particularly in international development where we run the same programs over many years. How would you deal with that and the assumption that lessons need to be surprising or new in order to be considered valuable? Yes. There's a bit of a pressure, isn't there, for lessons to be real aha experiences and sometimes they just really stay in the obvious, aren't they? Like pilot testing is really important before you run a program or you shouldn't set targets until you've run the program for a year or whatever they are, they're often fairly common sense things and I think you're right, sometimes the clients want some sort of really major aha experience that just hasn't emerged. There hasn't been any major startling thing and perhaps the principles of good program design which are really sort of, if you've evaluated enough programs you get a really good sense of what good program design is about and therefore often those lessons are about just fundamental good program design, things like in those lessons, communication materials should be customized to the audience to ensure that they're engaged, et cetera, et cetera. They're all just common sense, good program design lessons rather than major insights into how to engage at risk itinerary young people. I mean I said engage with youth work agencies, well that's a pretty common sense, isn't it? There's not exactly any sort of rocket science in that one but sometimes you have to state those common sense ones if they've been absent in the program. If this program hasn't engaged youth work agencies and that's why it's had problems engaging out of school kids then you have to state that as a lesson that this is what you should have done and it's what programs should do is make sure they engage with the referral sources that will give them the clients that they need to achieve their targets. So I don't know if it answers that really, well it just sort of says that yeah I agree that there are often expectations that are sort of real insights and yeah almost revelations that just may not be there and it may just be good program design principles that emerge as lessons if shown to be absent in that program. Okay, thank you Anne. Eunice Sotelo makes a comment for you to comment on about terminology. Eunice says that she's also come across actionable insight in inverted commas, in reports. It sounds like recommendation and lessons learned combined or do you think it's completely different? I've not heard of it. So that's a new one for me, actionable insights. It sounds like I probably need to find out a bit more about what they look like before I could comment on them but it sounds like the term, it sounds like it's a combination of a recommendation and a lesson but as I said, I think there's value in separating out the two because the recommendations are really pertaining to the program and the lessons should be broader and apply to other programs. So whether if they're merged, I wonder whether they have the same value in terms of the ability to sort of focus on what's good for this program versus what's good for other like programs or other programs that might be working in the same sector or space. But again, it's hard to comment without seeing it. It might be a really nice innovation but often people create things to be different and whether they have value or not we have to assess on their merits I suppose. And thanks again, Willa. And I have to announce now that that's the end of the questions. We've got no more questions at the moment. You've managed to get through them all very well. You've got a number of comments here thanking you for the presentation and also other ones commenting about making clarity where clarity in a process where sometimes blind leaps were previously involved in arriving at conclusions and findings, for example. So that's about all, Ann. That's all the questions. So I'd personally like to thank you very much for your presentation. Did another question just crop up, Ann? Sorry, I just saw one flash in front of me. Something about maybe actionable insight comes from Jane Davidson that commented. Yeah. I wonder if it's also avoiding making recommendations. It could be. And that's another thing is sometimes people don't want recommendations. They just want virtually the sort of findings without the recommendations. So that's something that some kinds avoid is you making recommendations and it always feels a bit odd when that happens but that you haven't finished the sequence. Yeah. I'm sorry. Can I cut you off? No, not to worry. You've got a number of comments thanking you for the presentation coming in but no more questions at the moment. I'd like to thank you very much on behalf of everybody for the great presentation. It was certainly very clear and thorough and I'm sure a lot of people are looking forward to seeing the PowerPoints when they do arrive on the AES website. And once again, we've got some people particularly looking forward to your workshop, your AES workshop on evaluative thinking and reasoning coming up in October. Yeah. And I should just mention if anyone wants to do the workshop you're going to have to do some hard work. So there's some exercises, some applied exercises to actually get you to experience doing this yourself. That's the purpose of the workshop is to move beyond a lecture to actually getting you to actually have a go at taking some data and moving through the sequence that I've just described. Okay. So thanks very much, Anne. I'd like to say good night to everybody and thank you very much for attending this AES seminar presented by the Victorian chapter of the AES and we'll see you next time for another seminar. Look forward to it. Bye, everybody. Bye-bye.