 I'd like to welcome Dr. Julian King. Dr. King is an expert, and many of you, I'm sure, will already know this, but an expert in economic evaluation and value for money is the director of Julian King and Associates Limited, a member of the Connect Group, and associate of Oxford Policy Management, as well as an honorary fellow at the University of Melbourne at the Assessment and Evaluation Research Center, as well as at Charles Darwin University. So, lots of affiliations there. And Julian has completed his PhD in the Value for Investment approach, which included a model for combining evaluative reasoning with economic evaluation. And as many of you know, his approach is used globally to evaluate complex and hard to measure programs and policy reforms. So thrilled to have you this evening, Dr. King. And I'd also like to welcome Andrew Gorkins, who's a partner at ARTD Consultants, a board member of the Australian Evaluation Society as well. Andrew works as a trust advisor across the public and nonprofit sectors. He's focused on understanding client needs and developing really innovative methods to generate credible evidence and useful insights. Here's a strong foundation and grounding of his approach in systems thinking, causal analysis and logic, which I'm sure he's gonna draw on tonight. And he has many decades of experience as well, conducting evaluations in public policy and is deeply passionate about the practical utility of evaluation and evaluative thinking for public policy in our complex and uncertain world. So I'm gonna stop talking and allow our great speakers to take us on a ride this evening. So just a reminder, if you do have questions or comments during their presentation, please feel free to use the chat box, but there will be time after the presentation for our speakers to take questions from you and to also hear comments as well. So over to you. Start whenever you're ready, Andrew and Julian. I think I'm off first. Oh, we've got the first face on the video screen. Always nice to see some humans there. Look, yeah, thank you very much for the kind words, Ruth. As Ruth said, my name is Andrew Hawkins. The focus I'm bringing today to our presentation is what I'm calling propositional evaluation, which is an approach that seeks to shift attention away from primarily research questions about what works to more evaluative questions about what makes doing that a good idea. And the process essentially is to make logic models more logical and useful by treating a programme as a proposition that a certain course of action will likely lead to a certain set of outcomes. Hi all, I'm Julian King. My focus is value for investment. So that's an approach to getting better answers to value for money questions using rubrics, mixed methods, stakeholder participation and a little bit of economics. And Andrew is going to be changing the slides tonight and it feels a bit mean, but I've got all sorts of animations and transitions and things in there. So he's going to be pushing the button a lot. So let's see how we go. Trying to read those non-verbal cues. Yeah, yeah. We think there's an idea that might be worth unpacking at the intersection of Andrew's interest, propositional evaluation and my interest, value for investment. So let's explore that. Next slide. We're both using theories of change, of course, in our work and we appreciate the value they can add and we can see some ways that we'd like to improve them. Andrew, do you want to kick this one off? Yeah, okay. All right. So theory of change in program theory, broad terms that define differently by different people, but generally I think refers to a description of the causal mechanisms within the program or the reasons and assumptions that justify a program. Theory of change of program theories, you know, I believe we're in advance on the experimental approach to evaluation that dominated the early years in the 50s and 60s and before that focused on where the programs worked or not. Towards now thinking about what makes them work. So theories of change and the leveraging of causal mechanisms in social policy and programs, I think was a great advance, but I think there's still work to be done. Of course, this idea about causal mechanisms in programs I think was most fully developed in principle at least by the realist evaluation group. But Julie and I both feel that there's some work to be done about theories of change. So as I sort of said to Ruth before and oh my God, I could talk about this all day and I'm really gonna try and keep it to under 60 seconds. So I'm gonna Julie and helpfully summarize my verbose stuff into these dot points and I'm trying to stick to those. So I think the first point is I think sometimes the theory of change and they're written by the narrative box in a different ways of writing them. I don't think there's an established canonical definition, but you know, you guys are all experienced so you'll have seen the variation. But I have a general feeling that some of them are a bit overly optimistic in what they set out in terms of what might be reasonable to expect or what a reasonable person might interpret from reading them that you think you're gonna achieve and may not adequately set out the actual key risks of program failure that really need to be monitored and evaluated to manage during delivery. So in that sense, I think they can often lack a bit of specificity about what it is that we're actually doing and why that might work. So I guess the general critique is I feel like theory's been somewhat overextended and by a corollary the logic rather underdeveloped in our approaches. So I think, you know, I'm a big fan of theory and the use of theory in program design and evaluation. I think a grasp of theory is crucial for a good design. I mean, if you violate scientific theories about the way things work in the world, your programs probably won't work. So it's really important that you have some grounding in substantive social science theories. But I don't think there is a very good at justifying a whole set of activities within the whole lot of program activities. Now, I think programs leverage theories but they're not themselves a type of theory or best understood in that way. I think the theory of change tends to discount or overshadow the crucial role of setting out the logic of a particular invention in a particular time and place. The reasons and assumptions to think it will work. And of course, any logic model could easily be wrong. No one's perfect. And we may well still see positive results from an experimental program. But in my view, if the proposition doesn't make sense on paper at the beginning, I don't think we should proceed. And finally, I think I just worth noting that critiques of program logic, obviously I'm focusing more on the logic side. I think there's a common critique of program logic that it's too linear. And I just wanted to say, I think the real mistake here is to consider program logic as displaying a chain of cause and effect which is often the way that they're described. I think if we change our paradigm and think of them more as a reason proposition for action that we don't need to fall into that trap of thinking it's an overly linear model of cause and effect as opposed to an argument structure. If we'd all been taught logic and reasoning which I don't think is part of the education in the 21st or 20th century anymore, we would probably maybe not make that mistake. So finally, I think the final thing I have a concern with is this overuse of the word theory and implication that what we're doing is scientific research tends to reinforce the idea that programs should be treated like hypotheses or tested with experimental design and would then tend to privilege the scientific methods of experimental design over other methods that may well be more valuable in different situations. So Julian, I hand to you. Okay, from my side, I'd like to improve theories of change by getting more explicit about value. Theories of change focus on how an intervention brings about change, but evaluations about more than just change, more than just about what changes and how it changes, it's about value. And when we're looking to answer a value question, I'm finding it helps to add some things to a theory of change to get explicit about what's being invested and what's the value it creates. So Andrew's work on propositional evaluation, my work on value for investment seems to converge on this idea of value propositions. And that's what we're gonna explore with you now. So the glib line that you can see on the bottom left there is this idea that maybe we should consider evaluators as more like engineers than physicists. We're helping people build interventions of value rather than acting to try to understand the fundamental nature of reality. As interesting as it may be, I would argue that's not our primary job. Our job is determining the value of things that we've built in the world. So let's get to making this a bit more concrete. Over to you, Julian. Very good. So here's an example. It's not a real program. It's just something I've made up at my desk one day, but I think it's pretty close to some real programs that are out there in the world. And the basic idea is a subsidy for people to buy e-bikes. So the subsidy makes it cheaper to buy an e-bike, so people buy more e-bikes. So we have ourselves the beginning of a theory of change, right? Let's build it up a little bit. So we've got an input, the subsidy, we've got an output, people are buying more e-bikes and we can add some outcomes. So for example, one thing I'd hope to see is that bike usage increases and we get more people engaging in physical activity. And once people really start to love their e-bikes, we might even find car usage decreases. That might sound far-fetched, but I know it's true for at least one person, me. And so perhaps we'll see some reduction out of that and traffic congestion, pollution, greenhouse gas emissions. We could theorise that more people, that we could theorise that we can measure and attribute the difference we're making, but I'm not gonna let that stop me putting in the programme. And we can add some impacts. So for example, New Zealand and Australia both have wellbeing frameworks. We can imagine achieving some population level impacts, maybe that contribute to a happier, healthier population. Again, I'm not sure we can measure and attribute those, but let's put them in our theory, let's acknowledge that that's the why that's where we're heading. Let's have a look down at the inputs level. We've talked about a subsidy, of course, but we also need a team to administer the subsidy and the team needs office space and computers and so on. And then we're gonna need fuel for those office space and for those inputs, so we're gonna need resources like money. So in New Zealand Treasury Speak, we call the subsidy non-departmental expenditure and we call the staff officers and computers and so on the admin. We call that departmental expenditure. I'm not sure whether Australian public finance uses the same terms, but at the end of the day, it's all just different buckets of taxpayers' money, right? But we're not talking about just the money. Money can't be the only resource here. Like what about natural resources? So if the subsidy has any effect, it's going to bring about some changes in natural resource use, like more lithium, more cobalt to make batteries, maybe less burning of fossil fuels. So there'll be some changes, some ups and downs in different resources. So here, look, we just have the beginnings of something we could call a theory of change, a little bit cartoonish, perhaps, a little bit glib, but a good setup to illustrate some ideas from propositional evaluations. So Andrew, back to you. Okay. So, all right. So I'm going to take Julian's super simple version of the program theory or theory of change or program logic. I could be any of those three. And I'm going to try and see how I can make it fit together into something that I would call a valid proposition, at least a program that I think makes sense on paper. So in my view, a valid program is one where the assumptions that we make about the world, because we have to, and there's an existing world for any intervention going into it, that the outputs of our activities that we directly control and do, in addition to those assumptions, will equal some outcomes. Then there's a bunch of external factors that are way outside our control that contribute to the extent to which will moderate the impacts that we ultimately have to the world. My sort of first sort of idea is that this is our sphere of interest. This is about these things outside of control, but it's why we're doing the program. This is our sphere of influence. And for all of those who have happy with self-help books and Warren Buffett and Stephen Covey and others, we'll see the language of the difference between your sphere of interest and your sphere of influence and not confusing the two. So let's put some skin on this. Now, of course, I'm not an expert in e-bikes I have had some work with subsidies and evaluations of subsidies and rebates and transfers and whatnot, but so see this as indicative rather than certain, but I'd go, okay, if I was going to run an e-bike subsidy program, I would need to, I think I'd be assuming that e-bike retailers and manufacturers or importers have actually a sufficient supply of bikes in the country to meet this newly created demand and prevent a spike in prices that might come from a low supply relative demand. We're gonna have a problem with our subsidy if there isn't any bikes to buy or not enough that the price shoots up. Also, we're gonna have to assume that there are actually enough people who will ride an e-bike not just buy one and put it in their shed, but people who will ride an e-bike for whom the cost of the e-bike is the only obstacle to them. That's what the subsidies meant to do is overcome the cost barrier not any other ability to ride a bicycle barrier, for example, that might lead to necessary for car usage. Okay, so then we get to more like the conditions and these will all be conditioned statements that I'll be talking about the conditions that we hope to directly achieve from our activities. So, you know, people who have not bought an e-bike due to cost are gonna have to know about this subsidy assuming it's a consumer directed subsidy. They're gonna have to perceive that the subsidy is easy to apply for and obtain. History of subsidy shows that there's a number of government subsidies out there people can't be bothered to apply for because it's too much paperwork relative to the money. So you'd wanna make sure that that was the case as Julian said, you'd wanna make sure that the people who administer the subsidy have sufficient resources to publicize it and administer the process such that these people know about it and have an intention to apply for it. I've put here in dotted box what I think is also probably a condition you'd love to be able to make was in place which is this are about the amount of the subsidy. Different people in different local economies may or may not need different subsidies to motivate them to buy an e-bike a subsidy and now I'm gonna show my New South Wales where it's here we go. Let me try one in Central Melbourne might need to be different to a subsidy in who wants to nominate a town that's somewhat less socioeconomic. I don't wanna name anyone. I'm gonna say I don't wanna name anyone. You might have been. And there you go. Different people in different parts might need a different subsidy but look it's probably unrealistic to think this program's gonna be so well specified that it's an individually tailored subsidy but who knows what's possible in the world of AI. So that's sort of what I'm saying and I'm going well if we did that let's just say well then target riders access the subsidy and purchase an e-bike that they otherwise would not purchase. And so I'm saying here's a proposition if these assumptions are true if we achieve these outcomes I think we can achieve this outcome. Then there's all the rest of the things that happen in the world that affect actual bike usage at the top of the weather people how much they use them they're outside of our control but will ultimately affect these higher order outcome or impacts that we're often trained to look at measure the impact on contributor we all know the history of contribution analysis we all know the history of the difficulty of attributing things. And I think we sort of still paying too much attention to that stuff above the dotted line when really our sphere of influence is look this is a program about getting to people to buy e-bikes how do we do that? Well if e-bikes is a bad idea well that should have been sorted out in the design phase us measuring the outcome in some non repeating way in a complex system is of arguable utility for me for decision making in the future it's certainly not science because it's not a high specific hypothesis I would argue to be tested. So I think the point is these stuff at the top of the program logic is our why this is why we are doing the e-bike program for all the reasons Julian mentioned this is what we are trying to achieve is that people who would ride them but won't buy them because of the cost do and it's quite a specific outcome and this is our how this is how we're going to do it another way of looking at saying these are conditions in the world that are contingent on a range of other things outside our control these are a range of this is a condition one or more outcomes that should be our activities need to be sufficient for this condition and then this is our how these are the conditions that we think are necessary in order for our particular intervention to be sufficient for this outcome it's not saying a subsidy is necessary to get people to ride e-bikes it's not a theory about how to get e-bikes it's one particular way of doing it and in order for our plan to make sense these we think these things are necessary and it will be sufficient for this other thing. So another way of putting this into more logical stuff and I'm not going to go into inductive and deductive logic and if you had a whole day on it and you were interested we might but you can sort of think of some of these as the minor premise the outcomes of the major premise and the outcome is the conclusion of an argument about why we should have an e-bike but remembering of course the premises and conclusions are probable not certain they're written in absolute terms but for whom and what under conditions and how and to what extent will vary in reality. Also there are many other assumptions that we can't make that we don't change to a different currency tomorrow and make the subsidy useless but you can't list every single assumption behind anything we do in life where you'd never go anywhere. So it's about the assumptions that we're not doing anything about what we're relying on but we're not 100% certain are actually the case and we might need to check a bit more. I've listed not mentioned anything about the particular reasons as to why activities will lead to these outputs or the theories that underpin that about subsidy use or the inputs or resources because it couldn't fit them on the screen and frankly it was already too much information. So yes. So in summary, what I tried to explain with that approach in that example that Julian came up with is an idea propositional valuation which tries to change the orientation from measuring outcomes and testing theories to whether a proposition is sound. And sound is a technical phrase in this world meaning ballad and well-grounded i.e. the argument makes sense on paper that if these premises were all true that this conclusion would follow with near certainty and you can use this at any point in its life cycle. I would argue particularly in the design phase as a form of perspective evaluation which is somewhat underdone in the valuation literature there is a literature on perspective evaluation but not a large one. I think it could be used for that. I also think it's a means of advancing a more deliberative democracy and empowering citizens to ask why certain programs are being done and why they make sense to spend money on them because it takes it out of the hands of experts or so-called experts who can say yes, this program will lead to a 6.73% increase based on some calculation or model that they purport to have and it helps people that argue but does it actually make sense? What are the actual risks that this thing would fail? Let's focus monitoring and evaluation on the risks of failure and like many other forms of evaluation, yes, also holding to people to account for what they were sufficient for or should have been sufficient for. What it does not do and this is why I was really interested in talking with Julian about this is I realized I didn't really address any different values and priorities that people might have. The only values that come in propositional evaluation are the ones implicit in the design of the program, i.e. the value that more eBike usage would be a good idea. But how much and to what extent each of the under conditions that underpin this program need to be in place haven't been specified and what would actually be good, bad or indifferent from different perspectives is not addressed. So I thought that's why there's this intersection with some Julian's ideas I was interested in extending. And of course this approach is a little risky in the sense that it tends to be outside the paradigm of social science research and therefore outside what a lot of people want to fund, which is science or measuring impact because I'm prefacing propositional reasoning over these concepts. So I appreciate that's maybe not going to work for everyone. Julian. Thanks, Andrew. So I'm going to talk about incorporating value in a theory of change. And as you might see, Andrew's ideas have started to influence the way I'm thinking about this. So a theory of change, of course, is a well-established evaluation tool. And its focus is on how a set of resources fuels a set of organizational actions that then result in various outcomes and impacts just at the button once, thanks, Andrew. Yeah, so that's the simplest I could make a representation of a theory of change, right? And it includes the causal mechanisms. It includes assumptions underpinning it and so on. But essentially, it's about change. And a theory of value creation is a new thing. It's part of my current efforts to add something new to evaluation. And its focus is on how value might be created. So let's take a little look at that. Next slide, thanks. Just to define some terms, so when I say value, I'm talking about the merit, worth, or significance that people and groups place on something. So people and groups, right? So the value doesn't reside within the impact. It actually resides within the people. We have to engage with people to understand value. Next one, values different from impact. We can have a small impact that turns out to have a high value, or we could have a large impact that nobody cares about. So they could still be correlated. If the impact has any value at all, then we probably expect that the greater the impact gets, the greater its value. But they're still separate and distinct things. Next one. Okay, what's a value proposition? Now, I've kind of moved from thinking of theories of value creation to thinking about value propositions, and that's really been influenced by what Andrew's been writing and talking about. And when I talk about a value proposition, I'm talking about a declaration of intent that communicates the benefit of something to its customers. So if you imagine going to Silicon Valley and ask some founders at developers about their startups, they can explain their value proposition to you in a matter of seconds. And I think social programs should have clear value propositions too. It might take a little bit longer to explain the value proposition of a social program because there's a bit more complexity, like perhaps customers paying for the program and the customers who are supposed to benefit from it are two different groups. So the question of value to whom and who decides is gonna be really important in our space. But I think nevertheless we can and should be defining value propositions. And that idea of a value proposition really resonates for me because when people start talking about value for money it has this way of making us perceive a program as a cost, like a cost that needs to be justified, a cost that needs to be minimized. And it's not that that's wrong, it's possibly those could be valid things to look at, but it's too narrow and it's the wrong focus. I think if we reframe a program as an investment and a value proposition, I think we can do something useful with that. You know, someone somewhere, a decision maker decided to allocate resources to a program based on a set of value propositions and a sort of promise or a potential about how the world's gonna be a better place if we have this program. And so that's why I prefer the term value for investment. But ultimately a value proposition is gonna be useful because it's a construct that we can define. And if we can define a program's value proposition then we can evaluate how well that value proposition is being met. And that's our space, right, as evaluators. We already have the methods and tools to define a value proposition and evaluate how well it's being met. So defining a value proposition is useful because it turns value for investment into a question that we evaluators already know how to answer. So let's take that abstract idea and have a crack at making a little bit concrete. So there's a set of questions that I can just work through if you wanna define a value proposition for anything really, you can say what resources are invested in it by whom, what values created and to whom is it valuable. How is that value created? Like through what mechanisms? And to think about a little bit like barriers and success factors, what factors influence whether a program creates a lot of value or only a little, what needs to go just right so we can create enough value to be worthwhile. So let's illustrate that by adding a few examples to this cartoonish theory of change. So what value, well, financial value to bike producers is a gimmie, right? They're gonna be selling more e-bikes so that's gonna make their day. There's social value to people who are purchasing bikes in the form of health and happiness. There's value to road users if they find it's easy to move about town and there's impact value that we can maybe express in terms of wellbeing domains like environmental value and social value. And we can think about what resources are going into it. So there's the opportunity cost of alternative uses of public funds to encourage cycling. This is an idea we're borrowing from economics. Opportunity cost is the foregone value of the next best thing, right? That we could have done with the resources. So for example, we're thinking what other ways could we have spent these public funds to encourage cycling? Is this the best way to do that? And similarly, there's an opportunity cost to consumers as well because they're still putting some of their own money towards purchasing these e-bikes. And then there's natural resources that get used to produce additional bikes and there's fossil fuel resources that may be saved if people are using their cars less and so on. So we're just putting up some obvious examples. And then on the next slide, we can start to think about how is the value created? We can start to think about some of the mechanisms that sit between the boxes and a bit of economic thinking can help there too. Like for example, we need enough of the subsidy to reach consumers. If you Google incidents of a subsidy or Google incidents of a tax, which is kind of the opposite, you're gonna find some diagrams that have supply and demand curves crossing each other and they show how a subsidy gets shared between producers and consumers. So the subsidy makes e-bikes cheaper for consumers. So consumers want more e-bikes. So the bike producers are gonna respond to that extra demand by putting up their prices. And so you end up somewhere in the middle where the bike ends up being cheaper for consumers, but it's also more profitable for producers. And exactly how that subsidy gets shared depends how eager the buyer's out to buy and how eager the seller's out to sell and it kind of meets in this middle somewhere. And that kind of eagerness to buy and sell economists call that elasticity of supply and demand. So it's just a way of illustrating how a little bit of economic thinking can help work through how a program creates value and what needs to work out just right for the value creation to happen. We need the subsidy to be set at the right level. So if it's too low, it won't incentivize enough people to buy e-bikes. And if it's too high, we're throwing money away. So we need to get that right. We need enough people to buy bikes who otherwise wouldn't have. So some people would have bought a bike anyway. And that's called a deadweight cost to the program. We really want to know how many extra people buy them who otherwise wouldn't have done that. What else do we care about? We care about having enough behavior change. If more people buy e-bikes and throw them at the back of the garages, right, then this has all been a waste of resources. So we need people using them and sustaining the use. What else? We need resources allocated to this. So we need enough resources to do the job. If we try and do this too cheaply, we're going to end up with long waiting lists, slow response times, and it'll all be too inefficient to own work. So value for money isn't just about being as cheap as possible, it's about allocating the right level of resources. We also need, what have we got next, Andrew? Is it efficient costs to administers? We need the admin side of things to be efficient. We want to minimize wastage, having the right size team, good workflows and that kind of thing. And we need the right checks and balances to prevent fraud. So let's just pause and take this picture in for a moment. There's a lot of things here, kind of all highlighted green that need to go just right for our subsidy program to create value. And now if someone asks us to evaluate whether this program is providing value for money, whether it's good use of resources, now we've identified some critical factors that we can observe and make judgments about so that we can answer that value for money question. It's those things between the boxes. All right, nearly there, one more slide, I think. Just a little bit more so what. So like Andrew and I think have both said it might be a bit of an overreach for the evaluation to try and measure the outcomes and impacts in the top half of this. But what we can do is a future facing what's called an ex-ante cost benefit analysis. And that's the way CBAs often get used to look at future scenarios and say under what conditions will the subsidy create more value than it consumes? We can model that and we don't have a crystal ball so we can't predict exactly how much value we're going to see but the model can help us explore how effective and in what ways the subsidy has to be effective in order to be effective enough to identify some performance thresholds if you like. So doing that sort of future facing CBA can help to answer that propositional evaluation question of is this a good idea? And that gives us something useful we can use later on when we're evaluating how well the subsidy's actually performing in reality. We can compare the actual performance of the program with the assumptions that we used and the results we got out of the future facing CBA so we can check whether we're getting enough subsidy reaching enough consumers, enough people buying bikes enough people using their bikes on a regular basis for the subsidy to be still a good idea still good use of resources. So in effect that the CBA helps us judge in advance how good is good enough so that when we develop rubrics or assessing performance in the bottom half of the theory of change, we know what level of performance is likely to give us a reasonable fair value so that's been a useful modeling exercise. Next. Okay, so in a nutshell, we're suggesting that if we can define a program's value proposition we'll be better placed to evaluate it and the value proposition has to be logical and it has to be explicit about value. Anything you'd add, Andrew? No, I think you've covered it. I think we're over to the group now to you've got some time for my favorite bit which is the question and answer. So Ruth, I don't know if you had any if you wanted to take over from here in terms of- Yeah, sure. And I was about to say no one's posted but Madeleine you just have. So I can see a question from Madeleine who's asking for good books on economic principles for evaluation. That's a really good question. There's a couple of books that I always recommend for economic methods of evaluation and I'm not sure if that's quite what you're asking but if it was somebody wanted to get a handle on how to conduct a cost-benefit analysis there's a couple of books that I recommend and one is by Levin and McEwen. That's, I think, 2011. They actually call it cost-effectiveness analysis as the title of the book but it covers lots of different economic methods of evaluation including cost-benefit as well. It's written by evaluators for evaluators which makes it a nice introductory text and it covers lots of examples from different sectors. And then there's also one by Michael Drummond Prof Drummond who spoke at our conference a few weeks ago has his little blue book that he and colleagues wrote for health economists called Methods for the Economic Evaluation of Healthcare Programs. It's an absolute seminal text and one I highly recommend. But I think Leanne very kind in saying Julian's blog as well. I can certainly pop up a link to my stuff, yeah. I didn't know, Ruth, how you do this normally but do you break people into small groups for discussion? Yeah, I'm very happy to do that. It's really depending on what you would like. We do have a decent number of people so that could work quite well if you'd like to do that. I find a way that often works is put people into small groups for however long you think, five to 10 minutes, they have a discussion and somebody from that group might want to come back with something that they discussed in that group or as Julian, you know, something that resonates, something that like clarified, something that like pondering and maybe that's a good way to get people warmed up. Talked about outcomes and value and where they overlap and I think at least in my own work and I think more broadly people often assume that an outcome is a value and Andrew raised the interesting point that it is a value but for who and then it may be valued to different amounts by different people considering who is going to value that outcome could be useful to determine if it's the right outcome or if it's the right program to achieve that outcome. Absolutely agree with that. Yeah, yeah. The only way we'll understand the value of something is to reach out to the relevant people and find out. I think the other thing that sort of came up was like obviously it's almost a bit of a straw man. Nobody lives in a vacuum. Nobody lives in an apolitical environment. Nobody is purely logical. We're all emotional. We all have bounded rationality. So, you know, it's not trying to say if this thing is not perfectly logical, push back and tell them that, you know, there's give and take but I think there's scope for us to be helpful with people to do more cost-effective evaluations in finding out which of the bits here that we need most attention. Because I know a lot of people are like sometimes I've got a colleague in turn and he goes, your orientation is very, what you call holistic, like, does the program work or not? Is it good or bad? And I go, I guess I have a different, my attitude is there's value in everything. And maybe this is just different human nature. It's about finding the value in it. And, you know, almost any program, probably almost anyone these days would have some value and it's about for whom and for how and for why and how can we maximize that? And then sometimes the question is, is this the best program? And hence, you know, Julian's, you know, ex ante CBA is going to be a much better method. Propositional evaluation will be totally mute on the idea of which is, if this is the best way of doing it or not, an ex ante CBA can compare a range of different things and go look which the propositional approaches is going, well, this thing that you've come up with, let's just, let's make sure it could actually work on its own merits and what would it actually achieve and where should we spend our evaluative resources to make this the best version of itself? But it's not time to answer ultimate questions. Hmm, great. I might just ask room two, which I think was David, Joe, Leanne and Madeleine. Did you, any of you have any comments you wanted to share or reflections from your group discussion? And I'm putting you on the spot. So, sorry. Oh, Leanne, you're on mute. My roommates were very active in the conversation, but they've gone quiet now. I was just acknowledging that she works in family violence and I don't want to talk on your behalf, Joe, but because you got your camera off, I will. And was just saying that, you know, there's a whole notion of talking about value for money and so forth is a little bit outside the experience, perhaps comfort zone of some people. Yeah, so I was just noting that we're really talking about, you know, you've moved away from value for money for value investment, which is much more holistic, but it's just what is the program value really? And then you've got to define for whom and all that sort of stuff, but it really is because all programs are invested in in some way. So we're really just talking about program value in general, which might be an easier entry point for some people who are not used to having to actually be explicit about that. Yeah, I think that's right. I also think there's the separate value in asking that question, is this a good use of resources? Because, and it almost feels for something like family violence, it feels offensive to ask a question about value for money, this is just got to do. But at the same time, if we're going to allocate some resources to addressing family violence, we do need to say, we've got one shot at using these resources in the best way possible. Is this a good resource use or are there other alternatives we should be thinking about? And it's framing it the way an economist would, but it's thinking more broadly than an economist would. Hopefully it's bringing in all our value of thinking to think about that question. Is this the best way to use this one shot bucket of money that we've got here to address a really important problem? That's a different way to address it. That's right, the same issue. And I think it's, yeah, the truth is we're operating in an environment where resource allocation decisions are made and we need to respond to that context. It's just how, it's just how the world works. There's also also a big opportunity here though, in climate, yes, it can feel offensive around value for money for really sensitive topics. But on the other hand, if I was looking, let's look at the proposition, what are you actually funding in, let's say it's a domestic and family violence space and not going to come up with something, what are you actually funding? Okay, what's that actually going to be sufficient for? Like it's way down here on this list of outcomes. And what I'm worried about is people that ask for funding to address domestic and family violence and it's got like world peace type sounding outcomes at the top, which may give the funder the false impression that we're really going to make a big impact on this. Look, and that might be necessary from a funding perspective. But another way of looking at it is going, see all this money you give us, it can achieve, it can really only achieve this. We're down here. Really want to deal with family violence. The investment is going to need to be a lot larger than what you'll actually get asked to do. Help build a business case. I'm just picking up on scenes question and I know that seeing you're in room three, so I was going to come to you next, but I don't know if you've seen this question, Julian, in the chat. Do you want to respond to that? So last year I gave a presentation and at the EES conference in Copenhagen with John Gaugani and many of you all know John because he's spoken at our conferences in Australia before. And so I'm going to be quoting John, which is why I'm mentioning that. He says there's a couple of inconvenient truths about valuing and one of those is that things don't have value, people place value on things. The second one is that different people value things differently and we need to understand that variation is information and we need to understand it. So yeah, in other words, values uncertain, it changes over time. One of the challenges for us, we're evaluators, we have to understand value, but we have to see it as this mercurial moving picture that it is. And all I'd add to that is there's a reason why we should embrace the fact that we aren't scientists. We're trying to study a world that is value-free and if you can make a claim about Adam's quarks, quasars, it's because they have stability and they don't change. Well, we're talking about things that constantly are changed. And I always said when I got into this field, why do I love doing a variation? Well, partly I think it's the most difficult thing a human could ever try and do is understand the value of uncertainty interventions into uncertain worlds and it's ultimately really important. But sometimes we fall under the shadow of, oh, we're not a proper science. But I think actually what we do is super-ordinates to science. We're gonna solve all the problems of the world, which I don't think any of us have the knowledge ability to do, but if we're gonna try, I think it's not through more science, but more of better evaluative judgments about what we should do with our resources. Michael Scribina is looking down so proudly on both of us, Andrew, with little Scribonites, the pair of us, so science needs us.