 Today is a free event organized by the New South Wales Committee of the Australian Evaluation Society and we're here to promote evaluation practice and capability. We're holding a number of free events during the course of the year and next month's session which we're planning will be the last event of the year and we're planning to get together as well on the 3rd of December a bit of information on that in a moment. So just for your information we are recording this session and the session will be available through the AES website so anything you miss the slides and the video of this session will be available through that. I'll introduce the speakers in a moment but while we're waiting for people to roll up, just a quick thing that we may get the speakers to come back to but just to get you thinking a little bit we want to make these sessions interactive and informative. In the chat window if you want to answer the following question, what is the thing that scares you the most about economic evaluation which is the subject of our discussion today? Just anything that while we're waiting for people to turn up, any late comers, it'd be interesting to have your thoughts there. Again we've got a lot of people in this session it was over subscribed so we won't be able to get back to each and every comment but it might give our speakers something to respond to but also provides us food for thought for potential future events. So feel free to while we're introducing the speakers to add your thoughts on what scares you the most about economic evaluation and that doesn't necessarily mean that everyone should feel scared about it. Some introductions about our speakers. We have two speakers today we're lucky it's a bit of an international session. Firstly we've got a failure cowl who's the director of the Center for evidence and evaluation in New South Wales Treasury, and she provides economic analysis and advice in the New South Wales government and promoting evidence based policy into practice. By working with agencies to improve the evidence based the Centre in Treasury that affiliate runs helps inform government decisions and promote evidence based decision making. So while you're all doing that I'll stop sharing my screen and hand over to affiliate, although one other thing I might just do before then, if you want to rename yourselves and identify who you are and where you're from or not. If you pass your cursor over your own video image in zoom you'll see three little buttons pop up on the top right. If you click those three little buttons you'll have a rename option. And if you want to make it clear who you are and where you're from please feel free to use that otherwise don't. And with that I'll hand over to Affilia. Thanks very much George and hello everyone it's great to see so many people diverse crowd here. And I guess it's a couple of years since I this time of the year in 2018 I think that I last had the opportunity to speak to this forum and strikes me the world has changed a great deal in that time. But if there is a silver lining to all of this then it's that we can have so many people you know participating from much further afield than we could before all of these changes. So I'm going to now share my screen hopefully you'll be able to see the presentation that some slides a few slides that I'd like to take you through. So, before we launch into the main topic and existing policy settings on evaluation I'd like to just provide an idea of the policy context which has largely driven why we think the way we do why the policy settings are what they are at the moment. And the context for the broader debate about what the future policy settings should entail. And then a quick overview of how we see economic evaluation what we want to see it entail and use of CBA in particular before talking a little bit about some of the challenges and some of the, some of the mitigating strategies to those solutions, some of the solutions to those challenges. So first of all just to the content. Sorry, we can't see your screen so you've been able to. Okay, so I will try and rectify that. Forges all. I had shared it. So video is working but not just going to hit the share screen button at the bottom. I'm just trying to get back to. Just trying to get back to full screen. Again. So, share screen. And apologies everybody technology is different. Can you see my screen now. Yeah, it's great. Yes. Fantastic. That wouldn't have done very well would it. Anyone could see the slide. Okay, thanks everyone for forbearance on this. So this is the broad agenda I just took us through. And here is the policy context that informs why we feel why we think the way we do and what our current policy settings are. So what we're illustrating here is that there's been a string of audit reports and various parliamentary inquiries and things like that over the years. We've listed three of them here. We won't go into all of those. They're rather heavy tones. But what they all have in common is and we've taken out a few of the quotes from them that sum it up that over the years there's been the observation that there are significant weaknesses in the resource validation process. And these of course are findings that are based on audits that were themselves prompted by the revelation of significant policy failures or infrastructure projects that the value which was questioned after the event. So, my guess is that many of you, if not all of you could cite examples of policy failure of the so-called white elephants investments and the scrutiny of the audit office parliamentary inquiries and media, and not all of which, of course, are to be attributed to poor evidence because, for example, poor delivery or unforeseen events such as COVID can completely change the basis on which you were forecasting things. But it is true to say that well-informed decision making does require the best evidence we can muster and deserves to be supported with that rather than pure anecdote or conjecture. So this background led to a few major policy developments which I'll very quickly skate over now. The first of which was the pilot of the Washington State Institute Policy Impact Assessment Tool in New South Wales. And this was done in starting 2013 to 15. And the reason we did it is it's a CBA model that takes program evaluations that have measured outcomes and economic evaluations and then exposes orders, policy responses to any given policy objective by their effectiveness, their effect size. And in this case, the priority was for Washington State juvenile arrest rates because of its clearing to a citizen rates which were going through the roof and more and more prisons were being built at a time when decades of low crime rates. So the incongruity of that led the legislature to legislate for the creation of the Washington State Institute and their first task was in this area. And what it shows is that the state rate of juvenile arrests departed significantly from where it had been tracking along with the national average. So then they went on to do the same exercise in other policy areas. And so that has been a great success. And New South Wales was facing a similar challenge at the time, the Water to Pilots model and it being regarded as the world's best practice. So that was the first big policy response. It was an internal exercise with costors but it did establish the efficacy of the model and its potential use here so that it was possible, it would be possible to get to something like that in the future if New South Wales decided to do that. The second big policy response, of course, most of you probably have heard a lot about is outcome budgeting. And this is a much broader area than the area that my team are operating, but it is an important context setting reform. So, of course, this has been a long journey to so the outcome budgeting announced in 2017 and 18 and we've moved through significant changes to the way agencies acquire and report on that funding that they're allocated. And from this point forward, there will be a much more, a much bigger focus on outcomes rather than inputs, which have driven and increment budgeting, which are really driven decision making until then. So this is the context within which I guess the rest of the presentation of the talk will concentrate. So essentially, that shift in mindset towards outcomes for people rather than spending a little bit of inputs is what this slide is showing with an example of how you define an input. And ultimately the outcomes worked example so I can leave people to prove that if they would like to but many of you are more than familiar with this. I'll move on to how that interacts outcome budgeting interacts with the scope of the work that the center for evidence evaluation is reading. Because this is a common question and there is some room for confusion as the programs in outcome budgeting are very, very high level. So many of you will be familiar with the state outcomes that have been articulated and then the broad programs that sit underneath those which are the operations of the agencies working towards those outcomes that they can be programs can be spread across cost as they're very, very high level. On the right in the dotted line is really the area that I'm talking about where we're operating and talking about today and so to for the avoidance of doubt we are referring to what many of you will regard as programs as initiatives here to avoid confusion with the higher level programs that outcome budgeting has brought about so those initiatives are the ones that we can apply the evaluation that I'll talk about now too. So a little bit on the role of the center for evidence evaluation how it operates. So overall our objective is to develop an evidence bank, you know for decision making that's putting it very simply. Within that, we have to set the standards for agencies to follow. So example, for example through the news as well as government guide to cost benefit analysis and other things which I'll reference a little later, and the of course program evaluation guidelines. Below here on the right, really is the part that I would draw most attention to go in our operations we spend the vast majority of our time working with clusters to build capacity to actually apply those standards in the operating environments that agencies are facing, because the guidelines that we've set, they have to apply to the whole public sector and they're necessarily very, very broad. So within that there's a lot of scope for working with clusters to help them apply those frameworks to the priority setting but also to identify evidence gaps and work with them to fill those where we can. And then lastly, our role is to then assess the proposals that come in for budget consideration. Anything above 50 million the team will be looking at to assess and advise the treasurer and the cabinet on the relative merit of proposals and we expect to see a CBA accompanying those that meets the standards that we've set in those guides. This slides just showing how the what the actual documents are many of you are familiar with these so I won't lay to this point but we've provided the reference numbers all publicly available. And this is an attempt to show how they more or less relate to each other. So suffice it to say that those are the three main guides that dobbling the, I guess what we call the investment framework. The appraisal that sits within a business case where a business case is required and then of course the evaluation and before that the monitoring if the monitoring way that's appropriate. So what are we talking about when we talk about economic evaluation so we mean CBA. So this is one of the key evaluation technique that we use. And many of you will be more than familiar with this but I guess the key the key point I want to make here because it's one one of the most common questions and points of I guess problems is that people worried that this doesn't take into account social and environmental factors. This is purely based on a rather stark assessment of economic value but this is really a main problem again because we don't mean financial analysis by this financial analysis is something that needs to be in the business case of course because you need to know how you're going to pay for things but you that is not what the CBA is the CBA exists entirely to address the absence of other impacts in a financial analysis. So social environmental impacts are definitely included. So we're looking to ensure that we can identify benefits that exceed costs in the context of the whole of New South Wales. So ex post CBA then is of course undertaken during or after the issue that was actually completed and very very similar to ex ante CBA. And we would expect this to be done with a program before a repeat funding. It needs to inform the Cabinet and the Treasury to say that the program or the project that has run its course or is part way through delivering benefits is actually delivering the same benefits as were expected in the appraisal. And if not, then provide the opportunity to to adapt and or cease those programs in favour of ones that do do work better at solving the problem that you've set out to solve. Of course, you can have ex post CBA during an initiative or following initiative completion. So here we're outlining the main differences between those. So during an initiative this informs you whether it comes likely to continue if they are being delivered. And then inform that adaptive management that I referred to before and following initiative completion. I suppose obviously it's it's assessing whether the benefits were expected that were expected are in fact delivered. But I think most importantly here it's to inform continuous improvements to achieve those positive outcomes that people need and expect and in fact paying for and we're promised by in the social contract they have with the government of the day that they've elected. So this is the part that links the the outcomes that the community needs to see with the expenditure decisions that governments making the point of accountability for us. Here are the steps of CBA and the differences between the main differences between the ex ante and the ex post. So I've folded the parts that for the focus of difference I suppose. And so obviously in the ex ante where you're stating original objectives. It's going to be different in the next post CBA where the objectives were already set. And rather than defining a base case you'll be defining a counterfactual and so on leading down to the main point of the of an ex post CBA to use those findings to inform future appraisals. Sorry, you can probably all see that too. So often we need to think about monitoring and evaluation as a way of preparing for ex post economic evaluation. We do also administer the benefits realisation framework which used to be administered by another cluster. We're trying to integrate those things now and minimise energy profession and to try to streamline them to make all these guides easier to use. So in probably the main the main point I wanted to come to is that having said all of that and having and being strong advocates of CBA as a way to help prioritise public spending. We recognise that there are a lot of challenges and there are probably a lot more than this with campaigns. A big one is capability. Data quality is a big issue and the absence of data is a massive issue in almost every situation that we go to. This though I think is not going to lead to the conclusion that we should just drop it and try for a different form of evidence or multi criteria analysis or something because I suppose in all of the years that we've been working in this area. The first time I heard that we didn't have data available and that's why we can't do proper evidence informed decision making. If we'd started this then we would be there now. So I think it's time just to get started and addressing those gaps and that's what we're doing in partnership with the clusters and we're coming out with a range of frameworks that sit underneath the CBA guide that are much more detailed to plug those evidence gaps and also issues of methodology and applying evidence to these kinds of deliberations and we're making great headway. So I know that in the questions there'll be plenty of references to the limitations but I suppose I'd put it to anyone that saw those as insurmountable that without making this effort it would be a bit like saying we can't do costings for a major metro because we don't know what the final costs will be. That doesn't turn into an argument that you shouldn't even attempt to do a costing on it. So I think it's the same on the benefit side and if you aren't looking at the benefits that sit outside of a financial analysis then you're really not looking at the full impact of a project when you're allowing negative externalities to go unchecked. And so I think that's really where the strength of it lies and why we're trying to increase rather than decrease the adherence to CBA. So I think we're at time so I'll stop there not to hog the platform and I'll attempt to share my screen. Thanks a failure that was very good and very succinct. Leave questions till after our next speaker and take them jointly just so that we can get through this and get the presentations to get to the more interactive part of the presentations of the session. For our next speaker like to welcome Julian King from New Zealand who's a public policy consultant and director of Julian King and associates many of you probably already know Julian and his work specializes in evaluation and value for money. He did his doctoral thesis in that area and uses integrates evaluation with mixed methods and economic methods is very interesting approach to this topic. He's a member of the connect group and honorary fellow of the center for program evaluation at the University of Melbourne. Great to be here to virtually to present. So Julian I'll pass over to you if you want to share your screen run the presentation. Thank you George and Kia ora koutos. Hi everyone. Great to be here and thanks for the invitation. Let me just see if I can get my screen. Can you see that? Yes. Yep. Great. Thank you now. I'm going to leap in with one interactive piece before we get started. And that is this poll on cost benefit analysis. So George is going to pop a poll up on your screen. I hope you can see that now. And I'm interested in hearing your view in general because I know we're all evaluators. We all want to say it depends. But I'm interested in how you feel about CBA right now as a general proposition and evaluation. So if you can see the poll just to put your answer in and we'll release the results once we've got most people their results in there. No one's voted yet. I think that's people still reading or if people got problems with the vote. Let me know. People are saying they voted. It's not showing up to me as having a vote. But I'll see what emerges out of this. Problems voting. Just let us know in the chat window. But it's certainly interesting. We've got 87% voted now. I'm not seeing it then. Okay. Let me know when it gets close to 100 and I'll end the poll. We're on 87% now and it's essentially stopped growing. I'm not sure how to share this with anyone who can't see it. I'll end the poll and you should be able to... Okay. Then I've got a button saying share results. Yeah. Can everybody see the results? I'll tell you what it's saying. The majority, 67% feel that cost benefit analysis is capable of telling us something important. A few people, 15% think that it's the gold standard approach and 19% think that it's not completely wrong but it doesn't feel quite right either. And I also gave the option to say it's dangerous and just plain wrong and actually nobody feels that way so that's a good thing. I would just acknowledge that for all four of these options, in my view, there will be instances where that answer might be correct. But I agree with the people personally. I agree with the people who... The majority who said cost benefit analysis is capable of telling us something important. That's certainly... Just have to figure out how to go backwards. That's certainly the view that I'm going to share with you today because economic evaluation is important. As evaluators, we should be using it more but maybe not on its own and I'll explain what I mean by that. And there is a way that we can integrate it with the wider world of evaluation, the evaluation toolkits and perspectives that you already bring and I'm interested in sharing with you what that could look like. So just by way of a practical example, this is my transport policy. Vote for me and I will bring in a cash for clunker scheme where you can get a subsidy by trading in your old car. You can ride out in a brand new e-bike. You can swap your smoky old car with the brakes that don't work and the tyres that need replacing and you can ride away in a brand new clean green e-bike or if you prefer an e-scooter or even public transport passes. And this is my policy, come vote for me. I think it's going to bring a number of benefits for us all. Fuel consumption will go down. Emissions will go down. Traffic congestion will go down. There will be fewer cars on the road. Health will increase in a number of ways including just that people are getting exercise on their bicycles but also that smog will be reduced in the city and that benefits everybody. Happiness will increase. I'm going now. And everybody knows that. And it will bring about this thing I've called Modeship. These are kind of second round effects or it will flow on effects where people will see other people on bikes and they'll think if those people are cycling maybe I can too and we'll build a critical mass of people who demand safe cycling paths and so more infrastructure will be built in the city for us to get around safely. More people will start cycling and so on and so on and so forth. And I think there might be some virtual circle and out of that I think there might be some equity benefits as well. More options for more people to get around in more ways. So what if we wanted to do a cost benefit analysis on something like that? I like to think of cost benefit analysis as being a bit like a blender but instead of a blender for super, a blender for values. We look at all the value created. All the social, the health, the economic benefits and so on and we can put a monetary value on all of those and we can look at the value consumed, the costs and we can put a monetary value on that too. We can put all of that in the blender in this formula that I've put down the bottom right hand side of the slide and that's going to give us an answer. It's going to give us one number at the end like a net present value that says this is the net value created taking timing of those costs and benefits into account as well which is why that formula looks a bit tricky. So that's what cost benefit analysis is doing. It's a blender for values. It's valuing everything in dollars. But as Ophelia said before, it's really important to note that cost benefit analysis isn't about money. It looks as if it's about money but it's really about welfare. What's going on conceptually is that we have this phenomenon that we're interested in called welfare. It's an area of research and development in economics is how we measure welfare, how we should conceive welfare. We don't need to go there and cost benefit analysis and cost benefit analysis what we do is use something called utility as a construct to represent welfare. It doesn't do it perfectly but it does it fairly well. It's based on a model of rational humans, the rational economic person. A person who has a group of people who have individual self-interested preferences and when those preferences are satisfied their welfare increases. So there's a little more to it but that's this idea of utility. Now when we value utility, we value changes, gains and losses in utility, we value in dollars. And we can do that by looking at people's willingness to pay for various different things as a way of measuring their utility. So that's just conceptually what's going on in the background when we're thinking about doing a cost benefit analysis. It looks like it's about money, really about welfare. And so when we're using money as a proxy for value there are various ways, various strategies that we can use to turn different values into monetary values. So sometimes we're lucky, sometimes the thing we're interested in is inherently monetary so we're not even really using money as a proxy. And so for example in our cost benefit analysis of my cash for clunkers scheme we're interested in one of the costs is the cost of the subsidy. And so that's just monetary. Similarly we'll be able to look at the actual spend on bikes and scooters and so on by people. That might be the subsidy plus what people put in privately if they want a slightly nicer flash of version than the subsidy will pay for. Then there will be items that are bought and sold in real markets and we can use market prices as a way of saying this is what a market full of willing buyers and willing sellers has sorted out for themselves the value of something like the price of fuel at the pump. Fuel consumption is going to go down. We can put a value on that by looking at what fuel actually costs right now out in the market. Similarly in other countries have created emissions markets as a way of dealing with carbon monoxide for example carbon dioxide emissions and that creates a price for carbon and so maybe we can use that price for carbon as a proxy in our model. Then there are going to be a few things that aren't bought and sold in real markets and undertured we can create pretend markets for those and so there are various ways, various methods for eliciting responses from people or getting people to taking them through a series of trade-offs where they can reveal their values and we can understand their willingness to pay for things like health and happiness. So there are a number of options like this available to us to turn dollars into a proxy or to turn value into dollars. I guess a fourth one not on the slide is that more and more social value banks are being built up where other people have already done the work and turned some intangible values into monetary values and we can with great care think about using those in the cost benefit studies that we do but always being mindful of shifts in context if for example those estimates came from another country another time, another context then we need to be mindful of that. So I feel he's covered a number of the strengths of CBA before and I'll just just by way of a quick recap it's a great strength that in cost benefit analysis we're measuring valuing benefits and costs in the same units in fact just the fact that we're looking at costs at all is a strength when we think about what we often do in evaluation where we often look at outcomes we don't as often look at costs and if we don't look at both sides then we don't have a full picture of what's going on we might have two programs that look fairly equivalent in terms of their outcomes but if we looked at their costs we might find their costs are quite different and so then we'd be learning something that would put a different spin on our findings and on what we might think was the preferable program so the fact that it's looking at costs at all is good having the benefits and costs in the same units is extra good because it means that they are much easier to reconcile it means we can just plug everything into a formula and that synthesis step at the end of the analysis is the simplest part of all and outcomes in their present value the costs and benefits are adjusted for timing this is something called discounting there are good conceptual reasons for saying that we prefer to have things now rather than later and that there's a social rate of time preference that can be used as a way of adjusting the value of costs and benefits according to when they occur over time that's something I'll leave you to Google if you're interested in finding out a little bit more it's rational, systematic and replicable and what I mean by that is it follows a set of rules or guidelines and so for example I feel you mentioned the Treasury guidelines out of New South Wales but there's a number of guidelines like that both out of academic texts and out of governmental texts all around the world and they're all basically the same all very, very similar we can pick up any of those guidelines follow those guidelines any one of us can can do a very similar looking analysis we're all going to approach the analysis in a very similar sort of way because those guidelines and the thinking about how a CBA should be done has been set out we will of course make some very important analytic decisions along the way and those decisions won't necessarily be the same from person to person in terms of what time horizon we're choosing to look at what discount rate we're choosing to look at which costs and which benefits are in or out of scope based on on what we think should be in or out of scope or what in fact we're capable of assigning dollar values to measuring and monetising so they won't necessarily be exactly the same but they will have followed the same process and so you can pick up my analysis or I can pick up your analysis and we can all see what's being done understand how that's been approached and it's a great strength. Looking at future value cost-benefit analysis is very very useful for this that's the X anti-analysis as Affilia was talking about on a couple of her slides for example we don't know how many people might take up my cash for caucus subsidy how many e-bikes are we going to have out on the streets we don't know but if we're doing a cost-benefit analysis looking at potential future value we can look at what's happened in similar programs like the overseas I believe Lithuania for example is actually implementing policy like this right now and so maybe they have some early data that they could share with us we can also get a panel of experts together and we can come up with some assumptions about what seems most likely what would be fairly optimistic what would be fairly pessimistic and therefore what the range of possible values would be for in terms of the number of people using using bicycles after one year after two years we can look at those assumptions into a model and we can assess not just what the future value is but what it might be under various different scenarios we can even run the model in reverse and say what would our target be what's the minimum number of bicycles we would have to get out there with people using them to replace smelly old cars in order for this policy to be worth the money and that's called doing a break-even analysis and I think it's one of the most useful things that we can do in looking at CBA because it's rational and systematic and replicable and it creates this this indicator called a net present value we can compare results across programs particularly similar programs but even in theory at least quite different programs so for example we could compare the NPV of a health program with the NPV of an education program I would counsel great great caution in doing so without having a really good understanding of what costs and benefits are in scope and out of scope whether it's happening in a similar context with a similar time horizon and so on but in theory these things are possible overall the strength of CBA is that it's providing us with an approximate answer to an important question and that important question is is society better off overall now that's of course not the only evaluation question that we might want to address but if this is one of our things that we're interested in we'd be well advised to look at doing a cost-benefit analysis as part of the evaluation work that we were doing the flip side of those strengths is that CBA isn't the whole evaluation all of the strengths of CBA the flip side of a strength is a limitation and there are various things that CBA can't do or struggles to do or it wasn't really designed to do in the first place so a decision to go CBA is a decision to go quant we may also be interested in qualitative evidence CBA is principally about efficiency looking at looking at questions of efficiency people have grappled with and experimented with ways of looking at equity you can do CBAs for different groups in society you can compare them but essentially this is a tool for looking at efficiency we might want to look at equity in other ways or at least have the option of doing so when we're doing a CBA we're using a model of utility to think about value there are other ways of thinking about value deontological values qualitative ways of looking at value and so on we might want to think about those CBA is a little bit of a black box method the big outcomes that come out the other end the how you get there the means isn't really part of a CBA but process might matter a lot in some other ways when we're doing an evaluation so we might want to look at that too I won't cover everything on the slide here but it's for you to come back to and have a look at CBA is aggregative it's based almost on an assumption that consensus is something that we should reach that is even desirable whereas we might want to actually deliberate on tensions and differences between different groups it's been criticised on the basis that it can act a little bit like a voting system that reinforces the status quo or favours the majority rule over the minority voice so there's we need to be mindful of that that might be quite appropriate in some context not on others there's the tangible intangible continuum where some things are easier to value in dollars than others so in my transport policy the fuel savings would be very easy to value in dollars happiness would be somewhat harder and so it's quite common for a cost-benefit analysis to leave out some things that are very hard to value monetarily but that might be very very important in terms of their inherent value within a programme that we're looking at and then there's the whole issue of complex programmes and complex systems and the evaluation field is developing more and more sophisticated ways of looking at that. One of the ways of dealing with complexity is the principle of parsimony which is what we apply typically in a CBA where we're saying let's make a model of the world that is as simple as we can possibly make it without making it totally invalid or it's just 80-20 rule just enough in there to make it an adequate representation of what we're looking at and that cuts out a lot of that complexity. Again there's no right or wrong here it's just that we might want to give ourselves more options than just doing the stuff on the blue side here on the slide. So my view out of all this is that CBA estimates something really important. We should be using it more than we are as an evaluation field but we shouldn't think of it as the whole evaluation. Instead we should think about mixing disciplines how can we bring economics together and do something a bit broader and similarly to mix methods so bring all the benefits that we can get from qualitative and quantitative evidence so that we can look at all of the evidence before us and make informed judgments. So how do we do that? If we're mixing disciplines and we're mixing methods we end up with some quantity evidence, some quality evidence, some things that have dollar values on, some things that don't. How can we bring all of that together and make one a value out of judgment about whether something is worth doing whether it's value for money. Well this is where evaluative reasoning comes in. So to me it reminds me a little bit of a prism working in reverse. You know where a prism you can shine a beam of light into a prism and a rainbow, a whole spectrum of colours will come out the other end. Well here we're running that backwards. We're using values as a lens as a prism and we can feed a whole spectrum of evidence into that and we can reach one pointy value out of conclusion out of that. And those values that we use as a lens, a very good way of doing that is to use rubrics. So here's an approach to using rubrics and an evaluation that brings together economics and other quant evidence. This is something I've published. It's something you can download. I'll provide at the end of the slides a QR code so that you can find the document on my website. This diagram comes from. This is just an eight step method just to give a sequence to how you would go about first designing and then conducting an evaluation of the type that I'm talking about. It starts by understanding the program, getting a good understanding of the context, the stakeholders, the theory of change and beyond the theory of change, the theory of how the program is supposed to actually multiply value, changing value if it works well. That sets us up well to develop criteria and standards that are aligned with the intended program design and our understanding of the context. Now criteria and standards, if you put them together into a matrix, that's called a rubric. So while I'm talking about it, steps two and three is making a rubric or several rubrics. Once we've got the content of those rubrics, and really only once we've got the content of the rubrics, we can see what evidence we need together to be able to address the criteria and standards that are in the rubrics. And in my experience, almost invariably, we find that anybody would look at that and accept that you can't do it all with numbers. You need quanta and qual. And quite probably there'll be a role for economic evaluation in there as part of that. Then when it comes to analyzing and synthesizing that evidence, the rubric comes back in at step seven where we use it as that prism as the set of lenses for bringing the evidence together so just by way of a slightly more concrete illustration of a rubric, this is what one looks like. The criteria or examples of criteria you see as column headings, equity, community buy-in and cost-benefit being the three examples I've put here, we could all think of more. And then some levels of excellence for excellent value for money down to poor value for money as our row headings. And then just some examples there of deliberately a little bit glib so they fit on a slide, but just to illustrate what this looks like as a way of bringing together all the good stuff we can get out of a cost-benefit analysis together with some things that don't really compute in cost-benefit land, like equity, like community buy-in and so on, and bring that all together and make one judgment. So now just to bring us back to this slide. Now we can have it all. We can have all the good stuff that CBA brings and we can go broader and then we can bring it all together and reach a unified conclusion. So I'm going to hand it back to you. Tell me in the Zoom chat window, either say one thing I learned or one thing I'm curious about or both if you like. Thanks, Julian. If you want to hand back the screen, share that. That'd be great. Really great. I appreciate both speakers keeping till time. I know we've already had some comments go into the chat window and Julian's just prompted for some specific things. So please put some questions. If there are questions directed to a specific speaker, if you just flag it, say, Ophelia, then the question of Julian, then your question. If it's for both speakers, just leave it open. We do have a lot of people in the room. So depending on the volume of questions and queries, as I said earlier, we may not get to all of them. But we'll certainly try and maybe offline. Our speakers might be able to respond in more detail to some of them. So feel free to ask some questions. I'll just flag one thing that's kind of why you're thinking about that. For those of you who are not members of the AES, please think about joining. You get some benefits like lower fees to the conference and lower fees for training, which almost pay for the cost of membership. There's a link there. If you are a member, please tell your colleagues, friends, that you can do that about these free events and possibly encourage them to join. George, perhaps while people are just typing up some comments, I might just share a couple more slides if I can. I'll stand back to you. So the first one is just a couple of resources. If you wanted to get nerdy about economic evaluation, these are the two books I'd recommend. The one on the right cost effectiveness analysis with the dark red cover by Lebanon McEwen was written by evaluators for evaluators. With us an education sector focus, but some broader examples too. It's a good place to get started. Very good first reader. The one on the left is really the go-to text for health economists. It goes into more detail and it's a highly recommended book. It's the book I pull off my shelf first when I want to check something out. Both of these books are the recommended texts in the master's paper that we teach on evaluation and value for money at the University of Melbourne. There's a couple of other books I'd recommend. Two, Cass Sunstein's book, The Cost Benefit Revolution. If you want something that argues very strongly in favour of CBA and why we should be using it, it argues that side very convincingly. And then one by Matthew Adler and Eric Posner called New Foundations of Cost Benefit Analysis. When I stop screen sharing I'll just hold that one up so that you can see the cover. It's the best book I've found that unpacks the theoretical foundations of CBA and really critiques them and says how solid a foundation are we on with this and how might we do things differently with CBA. And I'm such a nerd, I found that a really exciting book. One more, here's a QR code that'll take you to my website if you're interested in downloading the framework that I shared earlier or any other resources on the approach that I've talked about. Great, so I'll leave it to each of our speakers, Affilia and Julian, to start responding to some of the earlier comments about what people keep them up at night regarding economic evaluation and or any of the later comments we've just started receiving. Thanks George, Julian. I can jump in for a minute maybe and say a couple of things in response to some of the comments before handing back to you while you have a look at some of those questions. But basically I suppose one thing I'd like to add is that the idea that, I think there's an idea that in CBA, everything boils down to the benefit-to-cost ratio, a single number, which then basically decides whether something precedes or not. So it's really important to address that issue because I think the value of Julian's approaches that it brings in other factors and those are important but in the, and I don't think there's very much difference between what we're by saying sort of more of a way of getting there I guess, but in the CBA policy, the guide that we administer, it makes it very clear that in addition to the benefits and costs, you also, and it's in the steps in the slide that I put there as well, you need to do a distribution analysis to provide decision makers with information about how impacts will be, will land differently on different sections or geographical areas. And if you don't do that, you haven't got a compliance CBA in our view. So the decision maker needs to be able to look at a BCR for New South Wales as a reference group, which is necessarily the case because New South Wales Parliament is set up in the interests of New South Wales people. That's our reference group. So the BCR is calculated on that reference group, but within that we require equity considerations, distribution analysis and also sensitivity analysis, which is to say that if you may have got some of your forecasting wrong, how much, how badly out will you be if scenarios change a bit? So you need all of that. The decision maker should not be looking at, especially not an elected representative should not be overlooking the distributional analysis because that's incredibly important if different people are impacted in different ways and then can give rise to policy responses such as transfer payments for, say, if you're introducing a tollway but you don't want to penalise people who can't escape that road or are on low incomes. Then the distributional analysis is what provides you the basis for decisions about mitigating negative impact on some people which are inevitable from pretty much anything that will be winners and losers. So that's one thing. And the other thing the guide wants you to do requires you to do is you've got to talk to your stakeholders for that contextual and qualitative information. And if you haven't done that, we would take a bit of a dim view. How robust is this really? If this was made up in some ivory tower, how do we really know what help would play out in a community? So you've got to list those and demonstrate what the feedback was and how you collected people's views and that's all part of the methodology. So it is quite flexible, robust and broad-based and certainly doesn't define a decision. It's not a rule-bound thing that determines an actual decision in cabinet. It's one of many things that they'll hopefully take into account but I'd argue that it's a necessary input to any decision. It's certainly not sufficient on its own. It needs all other things to be valid. And I noticed some people have brought this into the comments. I think, is it Timor has mentioned some of these aspects, qualitative research and so on. So hopefully that addresses a little bit of some of the comments. Yeah, Apelia, that's really excellent and things I very much support, of course. It sounds like real strengths of the New South Wales Treasury approach. So that's great. And I think that if there's a contribution that my model might bring to that, it's adding the idea of using rubrics as a way of bringing diverse values and evidence together so that you can make a unifying judgment from the NPV, from the distributional assessment, from the qualitative interviews and so on, how to bring all those together and make a clear, transparent judgment about whether that represents an excellent, good average or investment. Georgia, there are particular questions that you want to point us to. There's quite a few there. Or Julian, is there another particular? They're all good, actually. They're all good. When some of them we can come back to people on later if we don't get to them now, but we could probably deal with a few now. Is there any common themes emerging? There's one interesting there about the use of willingness to pay as a way of creating learning values. And it's one that I've reflected on in the limits of those choice surveys. I'm just wondering if either of you... Oh, yes, strong views. Was the comment that there's some sort of doubt about the value of willingness to pay as surveys? Was that the comment I thought I saw it before? Yeah, well, I would agree. The best example I could think of is the U.S. polling before Trump was first elected and the shock and awe that followed. You can ask people what they would do, but what they really will do can be so very, very different from that that I wouldn't put a whole lot of stock in that without a whole lot of other qualitative or quantitative, if you can get it, information. The best thing you can do is reveal the preference. But obviously, in the ex ante situation, that's very, very hard to get. You haven't done it yet, so you don't know if they're going to turn up to a new art gallery or whatever. Would they really pay $1,000 each for one when it came down to it? But you can at least... You'd be better off probably using other evidence from previous projects than a willingness to pay. You could do a willingness to pay, but for the effort and expense that's involved in collecting individual data, sample space needs to be really quite big. There are significant issues with that as this audience would be well familiar with. But for all of the cost and the effort, it's probably... The majority of the cases, it's probably more worthwhile looking to other things that are already established, I would say. And I've got... Some of my colleagues are in here too, so if I've left anything out or missed anything, then jump in, Danielle. And Julie, I'm interested in your views on that too. So here's the book I mentioned earlier by Adler and Hosner, New Foundations for Cost Benefit Analysis. And in one of the chapters in there, I'm pretty sure they go into some of the construct validity challenges and also the measurement challenges and whatever approach you take to monetising value, whether that is through surveys or valuing games or borrowing market-based values or whatever, we're always imperfectly estimating an underlying phenomenon that we're trying to understand. And we just need to be mindful of that when we're putting all those numbers together to reach a conclusion that we should be less interested in the point estimates. In other words, we're not using a CBA to say that the actual value is $42.75. We're looking for something that says here's a range of possible values. It's probably not very much higher than this and it's probably not very much lower than that as a more useful and valid way of looking at both what goes in and what comes out of a CBA. Now, I'll just post a little link to some information about that reference in the chat for people. Invite to also contribute a failure. This is Danielle's work from Treasury also from the Centre for Evidence and Evaluation. So as you were pointing out, with willingness to pay, there is the potential, of course, to look at where there's realised benefits rather than necessarily doing a survey. So in the case of an exposed CBA, of course you're more likely to be able to find out what people have actually been willing to pay for a CBA assessment that you're undertaking or perhaps where you've predicted, say, a willingness to pay for an improved water quality and health benefits, then you should be able to start monitoring what those actual realised health benefit savings were. Good point. Thanks. There was a comment as well about how CBA sits with some other evaluation approaches such as social impact investment, I think it was, or social return on investment, sorry. Yeah. Well, respect your thoughts on VA. I've discussed that with Jeremy Nichols who pioneered the social return on investment set of methods and his view is that they're very different because CBA comes from an economic set of traditions whereas SROI has grown from an accounting set of traditions that you're looking to account for value. They're also explicit about valuing what matters so it's akin to a goal free evaluation if you've read Michael Scriven's work. It's rather than saying what benefits where what benefits we are hoping to see and looking for those and valuing those to the extent they're actually realised it's saying what do people actually value so it actually emphasises the method going and having conversations with people whereas if I follow a textbook CBA well under the New South Wales Treasury guidelines I would have to go and talk to people and that is excellent but under most cost benefit textbooks I could do the whole thing at my desktop if you just email me some data so it's a different sort of a perspective. Beyond that, when I was arguing the toss with Jeremy, beyond that I see them as being very much one and the same in the sense that they're using their valuing costs in terms and they're looking for the most valid ways they can to value those costs and benefits even if they're quite intangible they're bringing them together to say are the costs larger than the benefits they're making decisions about things like time horizons and discount rates to me they are much more similar than they are different I think that SROI and CBA have things to learn from one another probably and they both have things to learn from wider evaluation methods and some things I've seen when I look at actual SROI reports and the way they've been done they're often very weak in causal inference there's an assumption that our investment caused a set of changes rather than a rigorous assessment of whether that is the case and then if anything the SROI handbook goes through a set of steps for what John Gargani calls giving the outcomes a haircut where we start with a big estimate and we apply various percentages for things like dead weight attribution drop-off and various things to bring this down into a more conservative estimate in order to avoid over-claiming but it sort of sidesteps the causal attribution or contribution question altogether which I see as a big weakness in terms of the way SROI is currently done in practice but maybe certainly the potential is there to do it better in their way Yeah, I well said I'd agree with all of that and I think there are similarities and CBA can always learn and there are evidence gaps and methodological gaps that can be improved and that's the process of growing it rather than jumping from one technique to another and so the only thing I'd add really is from the perspective of an administrator you're also looking for consistency of approach over time it's even probably more important than which approach you pick, although I'd argue it's pretty important to pick CBA out of those two because you're trying to compare investments over time as well as across areas so if you're commonwealths in the state or jointly funding something you want the same approach so it's not going to be empirically perfect nobody really knows what the value of an avoided car accident is and we can argue it's the end of days as long as it's a transparent set of numbers then you can keep arguing about it or keep changing that's not the problem at all but what you want is consistency of approach over time and if everybody's using a different method and it definitely confuses the capability building side of the job as well because people might not know what similarly named things how they differ and how robust they are and the beauty of CBA is really that it really increases the transparency so you can see what assumptions are being made and by whom and what values are being attached to things and then argue about to get to the final CBA so anyone should be able to look at those and I would argue that they should be published but that's sort of the strengths and as I said from an administrator's perspective if you're chopping and changing between approaches from a whole of government perspective between a new hospital here or a new out of home care program for displaced children or whatever and you do have to make those calls you have a limited budget which determines which there will always be more projects than there are budget for and so you've got to work out how to prioritise those and a consistent approach is really really helpful for that I thoroughly agree with all of that the economic texts demand more rigor in terms of the way monetary valuations are assigned to intangible benefits whereas in an SROI I've seen some real leaps of faith I'm thinking of to use my cash for clunkers example how might I value the value of health and the sorts of things I've seen in some SROIs before would do something along the lines of well people pay for gym memberships because they value health therefore the value of health is the value of a gym membership and then that gets used as a proxy for value and I just think that stretches my imagination too far and I've seen a number of examples of that kind of thing that I just think that sorry I can't get my head around that is a valid way of thinking about the value of things there's just shifting a little from slightly away from the technical sides of doing it to the practical side of doing it I think there's been a couple of questions about what it takes to do this and the resource intensity when is this something that someone can do limited resources or is it going to obligate people to contract out a very expensive consultant is there somewhere in between where you can still do a reasonable job done it's horses for courses I've done I've done CBAs that have taken me half a day and I've done CBAs that have taken weeks and similarly combining CBAs with other things and putting them together can be more or less resource intensive depending on what you're doing and why you're doing it and how much evidence you have and how much time and budget you've got to do the best job you can do so it certainly doesn't need to be resource intensive if it needs to be done in a quick and efficient way it can still be quite robust a lot of the time I agree with that too it can be resource intensive especially for a really big billion dollar projects you need a certain amount of modelling of things to go on and technical work but what I'd say to that is if you're going to invest 20 billion dollars in something you want to spend a little bit of money figuring out what the options are what the alternatives are that you want to solve in other words you can't really afford not to because if even half of the state budget or any government's budget went to something that was less productive or well for maximising than something else then the whole of society has paid a very very big price and paid all our tax to do that as well and the other thing about capability is that you've got a consultant's question is a vexed one the capability has to grow it's a bit of a chicken and egg problem but if we keep hiring consultants then we don't develop the internalised capability to do this more quickly and a lot of repetitive work goes on because of that and if we shared our work more freely we would probably be able to make more use of the studies that have been done and so my team is trying to centralise in an evidence bank the data from previous CBAs so the next time you've got to do one in a really big hurry the kind of virus response was a very good example of that you want to be able to draw on what did work in a previous situation to support a particular section of the economy or business type or residential area or whatever and without those studies you have a precious little chance of flying quite blind so we put a lot of emphasis into that capability building and also developing more rapid techniques for CBA so they're much higher level but at least you're still organising your thoughts into what are the benefit categories what are the costs and benefits are likely to be and you can do that even doing that is better than just throwing a dart at a dartboard and hoping you've got the right answer and helps you when you've got 300 proposals coming in from classes and here's what we should do to help small business or whatever you've got a way of ordering them that's consistent none of which will be empirically scientifically perfect but should at least put things into an emerit order that will help you make a decision that you can that you can justify and that will more importantly lead to the outcomes that you're hoping for so the short answer I think is you can't afford not to do it and we can bring the costs down by being less maybe more transparent with these studies and sharing them more not just within a jurisdiction but across them So is that something that you see Treasury embarking on in the future to provide that kind of clearing house or centralized point where people can come to and see what's been done before is there a plan for developing something like that? Yeah well we're doing it now with the clusters as they're required to do the clusters being the agencies that come with proposals and we've worked with each and every one of them over the last couple of years to develop a way of sharing this that they can still manage their confidentiality requirements and the sensitivities around these things you'd need a degree of confidentiality to make a decision especially a contentious reform before all the hares start running in different directions and you can get some pretty perverse decisions when that happens so ultimately I think the Washington State approach is where we would want to get to be a long journey because they publish everything so if a program is ineffective and say why haven't we closed the gap with all of the spending that's gone on in Indigenous policy for instance we don't really know because we haven't really done that work so my argument would be get that work done and even if initially that's internal and clusters can share it they all have a part to play with the policies that are contributing to or not helping to solve those problems and if we share that at least internally initially we can move forward a lot more quickly over time I would think that there isn't much justification for not showing evaluations in business cases to the public because it's their money so let the debate happen there too eventually but until there's a degree of comfort around how we're going to manage some of these things and the inevitable requirements on agencies and to maintain confidence where it's needed especially where there's commercial confidence we haven't solved those problems yet but we are working through those and if we have all of this data then it'll help with that capability and the cost problem of evaluation because we should do less repeat work and so that's very much the aim just reviewing the comments through there's quite a few comments about intangibles and valuing intangibles and social change touched on a little bit I'm just wondering if either of you want to direct some specific points around those issues I think I've acknowledged it's a tricky area there are multiple approaches to that you have to be able to measure causally attribute and monetize in some way and intangible in order to be able to include it in the CBA but of course you can include range estimates rather than just point estimates which I think is a strength I think I cut across you there no I was going to say I definitely have a view but I'm very happy for you to put your view first would you like to say more on it no no I'm done it is one of the hardest areas and I mentioned before limitations and this is one of them right but again walking away from it just means those things aren't counted at all so if you leave it to an accounting approach then things that matter most to people like community safety, public health education how do you value those things in decision making if all you've got is a financial analysis so ironically the most important things get left out without an attempt to put a value on them as squeamish as it may make people to do so an example that we encountered when we ran the Washington state pilot a few years back was in the US they had come up with a value for avoided crime which includes avoided victim costs but also things like feelings of safety feeling like you can go out walk out of your house whenever you want so how do you put a value on that and this is where CBA has to be incredibly flexible because in the US the way they did that was they referred to the compensation payments or settlements that would happen in court as a result of damages or fear or emotional abuse and those numbers incredibly high it wouldn't be fairly arbitrary depending on well first of all you've got to go to court a lot of people can't do that and also how sensational crime is or something like that can influence what a jury will decide we needed a different way of valuing that so we we used a range of data including compensation payments and we that wasn't the only one but it was also a number of other studies that have been done before and so I'm sure the number that we ended up with in there wasn't the empirically accurate number but at least there was a number in there that we could say could be justified on the basis of these different inputs and then that meant that you could put those things in as an avoided cost if you were successful in reducing one incidence of crime whereas before that that avoided cost would not be there and so I think that's a neat example of I know other ones like benefits returns to education and things like that so how do you value spending billions on improving teacher quality or something like that you have to have a way of doing it otherwise you wouldn't spend other money on the basis of the financial assessment you would do it for other reasons like legislated minimum standards and things like that but when you want to know when something is really worth doing you've got to find values and the value of those things will always be contested as it should be and new evidence will emerge and things will change over time and that's important that that happens rather than fixating on I suppose the deficiencies the obvious limitations of this kind of work right thanks there's only a few more minutes if there's any other comments in the chat that you'd like to respond to now that says some about some specific areas like innovation carbon emissions what's been going on in New Zealand, Julian if there's any particular innovations coming out of New Zealand what do you want to talk about somebody asked there's some work called CBAX which is equivalent really it's New Zealand Treasury's equivalent efforts to what I think New South Wales Treasury is going through to try and get better consistency across government departments thinking about costs and benefits using a framework and a guideline and I guess the question was have I seen enough examples to have some insights into how that's worked in practice and I have seen a few examples in practice but I'm hesitant to generalise from the few examples I've seen but I do think it's an excellent initiative by the New Zealand Treasury and it's been going for a few years and it's and it's reach and it's effectiveness and I hope that they will reach out at some point and work with me on how to bring wider values and evidence into the equation then you can do with monetary valuations alone and is that available publicly? CBAX is something that you can google I think on the New Zealand Treasury website they'll have some written resources on it and I have a feeling there's an Excel spreadsheet I've seen with monetary valuations of different outcomes in there which may or may not be on their website but I think the effort is to make things available We've had a look at CBAX as well and I think putting the sort of framework into that approach is incredibly useful but it is I think my understanding is Julianne could be outdated here but that it's proprietary and for that reason we haven't really been able to look under the hood so it's a little bit difficult to know how some of the values come up with and that just with our approach with CBA where we want that to be transparent in the CBA itself and so that's for us a limitation on its use but the way that model works and so on I think is really useful having parameters in there that you can block out, I think it's an incredibly useful way to help ease that problem of capability and expense of doing this kind of work so it's a I guess it's a wait and see for us I agree with that you're probably already in touch with your counterparts in the New Zealand Treasury and everything I can do to help connect you to the people who are doing the work, I'd be happy to do that Yes, we have met with them but thank you, I'm sure we'll take the conversation further, I'd be interested in more of your thoughts on it Okay, cheers Question on getting the cost of carbon right is an interesting one, it's something I'm actually grappling with right now in a real evaluation of the climate change mitigation program and I don't know what the rights there are lots of different carbon markets, I found a report online that kind of collates all the different carbon prices from different places and tries to comment on what do we make of it I think in the end one has to put high and low carbon values in there and do the sensitivity analysis to see to what extent they affect the results does it alter our conclusions and I think it would add power to the conclusions of a CBA for example to be able to say even if we put in one of the relatively low carbon prices we still get a positive NPV if that's the case then that's a useful conclusion that can be got, similarly if it turns out that this investment only looks worthwhile if you use a relatively high carbon price then that's a useful thing to know too even without knowing what the right carbon price is to put on things we can still get insights by plugging different numbers into a CBA to see how it affects the results I agree with that and we wouldn't regard any CBAs complete unless it did tackle that question and put numbers in and then of course the eternal debate is what numbers do you use but the number zero clearly isn't right so anything that's not zero is better than zero but there's a lot of work to go in terms of how you determine that and how you determine it for jurisdiction so we welcome news that you're looking at that as well because it's certainly an unresolved issue here I've got a few more questions coming but I know we're getting close to the end if there's any particular questions you want to respond to I'll give you a chance for one last response to any of those that jump out at you otherwise as you see on the screen I think the links being posted in the chat window we really appreciate your feedback on today's session we're meant to meet a poll there for you to click on through your phone or your browser and enter that code and it would be great to get your feedback that's very helpful for us to play in the future also we do have an end of year virtual drinks done through Zoom planned for Thursday December 3 it's similar time I think the AES will circulate information about that and Lincoln registration procedures or whatever through the regular AES email system so stay tuned for that one and maybe put it in the diary now so you're free as the Christmas period gets a bit busy and I'd like to thank our speakers for a great session very informative from the questions it's been I think people are getting a lot out of it and certainly I've got some interesting discussions and thanks again to Julian and Felia and thank you all for coming along and we look forward to seeing you with the next one