 Thank you so much Keith and thank you for the opportunity to talk in this series like yourself. I really like to learn from other areas of application I think there's a huge amount to learn by doing that. So yes I am an economist although I'm probably unlike some economists my primary motivation is to identify how to make good social choices made on others behalf that's kind of what motivates me always has and that's why I ended up working in the field of health because that offered at the time at least the greatest opportunities to start to explore that and see how we could make that real and certainly the UK decision science and an explicit approach to difficult choices has been at the heart of decisions for two decades. The formation of NICE the National Institute for Health and Care Excellence in 1999 I was a founding member of that appraisal committee and sat on that committee for 13 years and that body NICE makes decisions about access to healthcare and they are mandatory decisions. So it's been a way in which to operate as an academic you kind of do your academic work but then you're also part of the decision making process and that's been incredibly useful and very important. So what about decision science and making decisions in healthcare? Well the kind of questions that I've kind of devoted much of my career to are well which health technologies mainly drugs but not always which health technologies what price should we pay and how much evidence do we need to support their widespread use. Now these sound interesting but they also sound kind of quite technical but let's be very clear about this these decisions are some of the most profound decisions you can have in social choice. What we're actually talking about when we make these choices is who will live a little bit longer and who is going to die a little bit sooner and those are very profound and they ought to be accountable to reason to evidence and all the evidence and also to widely held social values they also should be scrutinised to the greatest possible extent and the decisions that I was involved in on the nice appraisal committee certainly were so here's a few I'm not going to explain each one these are a few headlines from newspapers as a consequence of choices that we made on that committee. Charter for promiscuity is one of my favourites but there's there's many more. Now not only did we suffer the slings and arrows of outrageous editorial comment from the red tops this is what John Harris the then editor of the Journal of Medical Ethics had to say about the role of economics in these kind of choices and the role that Tony Collier and myself in particular were playing in having an influence on nice methods of appraisal that we were responsible for wickedness follow more likely both that we were ethically illiterate socially divisive and responsible for the perversion of science so so John Harris was quite upset and obviously I regarded this as fighting talk and it led to an exchange of about eight papers in his journal which I think was very useful actually to try and bottom out what role decision science and economics ought to play in these difficult choices let's examine a bit about whether John Harris had a point and the basic principles are why we need decision science and economics in these choices let's imagine for a moment that the primary purpose of healthcare is to improve people's health it's not the only purpose but it is a primary purpose and secondly let's imagine that everybody counts and that everybody counts equally probably not true but let's imagine that's true for a moment then the question becomes well how should we make those choices or the first thing we need is a measure of health if that's what we care about we need to be able to measure health and we need to measure that's going to capture the impact we're likely to have on survival and quality of life in a way that reflects people's preferences now we have those measures they're not perfect by any means but they do attempt to do that in a way that captures people's preferences when confronted with choices and that's a quality adjusted life year so imagine the green dot represents what the NHS has currently got available the x-axis is the quality's gained and let's imagine we go out we identify the evidence we synthesize it we estimate long-term effects using decision analytic modelling and let's imagine that for a particular technology for every patient treated we expect to gain two quality adjusted life years at the price the manufacturer is charging it's going to cost the NHS an additional £20,000 taken account of any cost savings or additional costs it's a net cost to £20,000 per patient treated one way to kind of express what's on the table here is to say you know what this new technology for every £10,000 we spend on it we gain one quality adjusted life year is £10,000 per quality worthwhile well to answer that question we need to know where that £20,000 is going to come from and what else we could have done with it in other words we need to understand the opportunity cost of this choice let's imagine for a moment that we believe the health opportunity cost to £20,000 per quality what we're saying is that £20,000 elsewhere in the healthcare system could have delivered one quality adjusted life year for other NHS patients what does that tell us what's on offer is £10,000 per quality our health opportunity cost is 20 what does that actually mean well what it means well what it means is that on average we expect to gain two qualities for every patient treated but we expect to lose one quality quality adjusted life year elsewhere in the NHS is a consequence of those costs in this case we gain two qualities we lose one this is good value to the NHS because we improve health outcomes overall we have a net health benefit of one quality what if the manufacturer decides to charge a little bit more for this product and they probably well let's imagine they now charge p-star and now it costs the NHS £40,000 per patient still the same technology we still expect to gain two qualities but that £40,000 is also going to displace two qualities so we gain two we lose two we're right on the cusp of saying yes or no in other words p-star is the maximum the the NHS can afford to pay for this new health technology what if the manufacturer charges a little bit more more than p-star now costs the NHS £60,000 it's £30,000 per quality for this new technology we can only afford 20 what does it mean well if we do approve then what do we expect well we gain two so it's an effective treatment but you know what those costs are just going to displace more health elsewhere in our healthcare system we're going displace three qualities for every patient treated if we approve at that price we're going to reduce health outcomes overall so that's why John Harris is wrong that's why you can't make ethical and coherent social choices without accounting for resources because you know what in healthcare healthcare system resources are somebody else's health the idea that economists know the price of everything and the value of nothing couldn't be further from the truth when anybody says that they're actually talking about accountants okay accountants know the price of everything but the value of nothing the job of economists is to understand value now to make this real we need to be able to estimate what's on the x-axis and the y-axis and nice has done a pretty grand job but doing that using actually having decision science at the heart of its methodological reference case to make these decisions what does nice need to know well we need to understand costs and benefits over an appropriate time horizon when they're going to differ between the alternatives for anything with a mortality effect that time horizon is the patient's lifetime if we've got a dynamic disease process it goes way beyond an individual's lifetime and if we've got long-lived investments it might be over many many decades that we need to be thinking about this time horizon way beyond the licensing trials where we have randomized controlled trial evidence we need to make sure that we're estimating things relevant to our particular target population in the UK and the UK context we're going to have to translate or think about how evidence translates between different contexts we need to compare all relevance alternatives not just those that have been included in the regulatory trials we need to use all available evidence not just the particular licensing trials but how those effects might be modified what the baseline risks are for example relevant to our population and we also need to characterize decision uncertainty as well so on the right is just a little picture of the kind of decision analytic models that are built to support this process within NICE so decision analytic modeling we assign distributions to the parameters we randomly sample in Monte Carlo simulation and estimate costs and effects of the alternatives we're concerned about and we use that framework to try and extrapolate in a sensible and plausible way beyond what we've currently observed and offer decision makers a range of scenarios and let them judge the reasonable plausibility of those alternative scenarios in coming to their decision so that's kind of really brief about a whole edifice of development of methods to do this in health including the very rapid development over the last two decades of a Bayesian network meta analysis multiple parameter evidence synthesis a whole range of things that have been done and in many ways it's been the engagement of policy makers that have pushed the methods forward in the field in which I work but of course doing all this is not enough as we saw on that earlier slide sure there's a big task to estimate what the additional health benefits and additional costs of a proposed investment in health might be but that's not enough we need to understand what are the health effects of the other things we could do or others could do if we made those additional resources required available to the health care system or alternatively if we're not expanding expenditure in health care but we're going to have to find the additional resources from existing commitments we need to understand the health effects of those things that either we would give up or other people in the health care system will give up and that's been what I've been working on for a good few years now estimating those health opportunity costs for the UK to inform these kinds of decisions and importantly pricing decisions for pharmaceuticals this is kind of a summary of where we are with that in terms of the published evidence so this is estimates of the health effects of changes in health care expenditure expressed in terms of the cost per quality how much does the NHS spend at the margin to generate one quality over these 10 waves of expenditure data we've got two series here the the blue dots represent our original work the red is relatively new work with a different approach to identification effectively here I won't go into the details there's a lot to say about it effectively we're using cross-sectional variation with instrumental variables two-stage least square to get these estimates a couple of other things to notice about this figure is the nice the thresholds that nice uses to make these decisions stated threshold is 20 to 30 000 pounds per quality what you can see is that all the evidence that we now have is that it's much lower than that that the NHS is really efficient at generating health at the margin we get an awful lot of health for relatively modest amounts of money in the NHS the dotted red line represents the department of health assessment based on this initial work the original work of how it assesses health opportunity costs in its impact assessments and our job in the next two months is to write a report for them to update that and I think you can see that that report is going to be saying actually 10 000 pounds per quality is going to be closer to the mark what does this tell us about policy well first of all although I was a founding member of the appraisal committee and devoted much of my life in an unpaid fashion to supporting nice I declared war on nice in 2015 precisely because they were making decisions using thresholds that didn't match up with what we know about what we give up so is nice in making decisions about healthcare doing more harm than good well unfortunately it is so the nice thresholds 20 to 30 actually nice never says no below 30 000 pounds per quality it goes up to 50 in some circumstances so for every 10 million pounds of additional resource that nice decisions commit and they are mandatory for the NHS if it proves a technology at 40 000 pounds per quality 10 million pounds we expect to gain 250 qualities what do we expect to lose well based on our original works just over 770 qualities elsewhere in other words the ratio of harm to benefit is about three to one what else can we say well we can start to use this to examine some other policy decisions if you remember this it seems a long time ago 2010 but this was one of the placards billboards used in that election campaign a commitment to fund all new cancer drugs that's a very big promise and it's going to require a very big checkbook to fulfill it and civil servants and others told ministers at the time you don't have a checkbook big enough to keep that promise and what they did was to create the cancer drugs fund which was supposed to be a temporary stopgap measure until proper price negotiation mechanisms were in place the price negotiation mechanisms never happened but what we are left with is a cancer drugs fund and these are the numbers in terms of spend budget not spend budget of the cancer drugs fund 13 to 2016 in every single one of these years that cancer drugs fund over spent by more than 100 million pounds this was money taken from the NHS to devote to new cancer drugs we have very little evidence of what the benefits were the drugs that were included very few of them had evidence of overall survival benefit if we take a very optimistic view of what the benefits were that's what you see in the in the in the third column of this table so in 2015 we spent or the budget was 340 million at best we got 5000 quality adjusted life years what did we lose elsewhere well over 26 000 quality adjusted life years in the NHS so this table and this analysis was used by the office of national statistics in their very not the office of national statistics sorry the national audit office in their very critical report the cancer drugs fund which led to its reform those reforms happened those reforms meant that it could no longer bust its budget but the cancer drugs fund is still taking 40 million pounds out of the NHS every year okay what do we know about this from other countries well I won't go through this but this idea of the centrality of opportunity cost in these kind of choices has been taken up around the world since our original work in 2015 we've got estimates from Australia Spain the Netherlands Sweden very recent work published in the United States looking at private health care systems and the impact on health of additional health care costs crowding people out of health insurance got Indonesia South Africa and very recent work just completed for the chinese provinces as well this is all using very similar approach to estimation than our own which is quite data intensive what about other health care systems where we just don't have that kind of data well we can try and get estimates using country level data and this is what it looks like for lower middle income countries expressed in terms of cost per dollar on the x-axis we've got under five mortality rates and as you can see it kind of looks how you'd expect that where we've got very high under five mortality rates very very small amounts of money can have a huge impact on a lot on health this time measured in terms of what's called disability adjusted life years averted a measure of disease burden now until very relatively recently the World Health Organization like nice had kind of threshold rules to decide whether to recommend things and their rules were one to three times GDP per capita per disability adjusted life year averted how does this evidence stack up against those kind of thresholds well these are the low income countries the dotted rising diagonal is what it would look like if it was one times GDP per capita in other words it's way lower than that way lower than that in other words very very very small amounts of additional resource can have a big health impact in these low income countries and this is what it looks like for middle income countries what you can see is that one time even one times GDP per capita is probably significantly too high for many middle income countries how can we use this usefully well in lots of different ways but i'm just going to illustrate one way which is we're now in a position to actually express the value either in terms of net health effects or in terms of dollars equivalent dollars of a range of different interventions and figure out who should can afford to pay for them and if we want everybody to have them then they're kind of subsidy or the kind of price negotiation tiered pricing mechanisms we're going to need to make that available so we're in a position to have a global demand curve for technologies and interventions across the world so this is for the HPV vaccination across gavi eligible countries the dotted line represents the market price of HPV vaccination that's available at that time 25 dollars and what you can see is that for at this time the World Health Organization made a blanket recommendation but what you can see is that for many countries that it's just not affordable at that price and implementing that recommendation would do considerable harm and by doing this you can start to inform those questions about when should you make a blanket recommendation if you're going to make a blanket recommendation what kind of subsidy or price rebates are you going to have to have to make that work and it also starts to indicate the kind of tiered pricing system we would need to have to make these kind of technologies widely available across lower middle income countries the other thing it also tells us and I haven't got slides about this but it's work we've been doing with the Gates Foundation we can use the same principles to start to look at technologies in development and that's how the Gates Foundation uses this work in assessing their portfolio of R&D thinking about what is the value and what is the value across lower middle income countries to start to prioritize the foundation's efforts but so that's kind of the first part but what about uncertainty nice requires a full characterization of uncertainty through probabilistic analysis of the decisionality models why why do we really care about the uncertainty why should we care about it well we should care about it even if the only thing we want are the expected values even if we just want expectations of health effects and costs most of our models almost all of them are non-linear we've got a non-linear relationship between parameter values and net health benefit and therefore unless we fully try to fully express the uncertainty we get the wrong answer in terms of expectation but there's another reason why we should care about the uncertainty and that is uh considering whether we have sufficient evidence and what the benefits of additional evidence are going to be and I think a clear separation between what do you want to do now given existing evidence from that question about do we need more evidence and in a sense the traditional rules of inference hypothesis testing or even Bayesian benchmark error probabilities basically confuse these two conceptually separate questions and as a consequence don't get a sensible answer to either of them so we need to use our understanding of uncertainty to start to get to grips with do we need more evidence to inform this choice in the future if so well what type of evidence do we need and how much of it do we need we then need to answer the question well if we do need more evidence and we know the type of evidence we need should we wait to approve a new intervention until we get it and that's going to depend on do we think we can do the research and get the evidence if we approve in medicine that's very unlikely if you approve a new drug you're not going to be able to do any random more randomized controlled trials and there's no incentive for manufacturers to do it so that's pretty much ruled out in medicine even if you could do it it might be costly to change your mind once you've approved a new intervention it might not be possible to change your mind at all or it might be possible but it's going to be really costly in both time and resource and finally if we approve we might be committing costs or opportunity costs that are going to be irrecoverable we're never going to be if we change our mind because the evidence shows that it wasn't a great idea we're never going to get those costs back and for all these reasons our understanding of the need for evidence starts to influence whether we are going to withhold approval until we acquire that evidence i'm going to try and illustrate the principles i think you might have had webinars on this as well but i'll just kind of do my version of it so when we asked the question if you remember what i said i said let's start with a principle the thing we care about is health when we ask the question is the evidence sufficient we're not asking the question can we reject the normal hypothesis we're asking the question would more evidence improve health that's that's the question we're asking and from the kind of decision analytic models that express uncertainty we can start to answer that question imagine this table represents the output from some decision analytic model and we've got five iterations from from that simulation process let's imagine there's just five and let's imagine these numbers represent net health benefit in terms of qualities for alternative a and alternative b let's imagine alternative a is current clinical practice and b is the new technology now when we do our simulation what do the results tell us well the results tell us that on average on expectation we get more net health benefit with the new technology b than we do with the existing current practice a so what's the best that we can do now given what we know well the best that we can do now is to choose b we approve b we expect on average to get 12 qualities per patient treated and that's a gain of one quality are we uncertain yeah sure we're uncertain and in fact in this case we're going to be wrong two out of five times the error probabilities point four now most people regard an error probability of point four has been really high but does that mean you want to choose a i don't think so i mean choosing a you're choosing the alternative which on the basis of expectation you would expect to be the worst and would have an even higher error probability so why why we bother him with this uncertainty we're the only reason why we're bothering to look at the uncertainties because we're trying to ask the question could we do better could we do better well we could do better if only we knew how that uncertainty was going to resolve if only we knew how the uncertainty was going to resolve we could make the best choice for every if you like realization of that uncertainty so for iteration theta two we wouldn't choose b we choose a and we get 16 rather than eight qualities for iteration four theta four we wouldn't choose b we choose a and we get 12 rather than 10 qualities the problem is right now we don't know which of those realizations which which of those possibilities is actually going to be realized so the value of resolving the uncertainty is the expectation over those possibilities in other words the expectation over those maximum values which is 14 qualities in other words we could do better if we resolve the uncertainty we'd expect to get 14 qualities compared to the 12 with current information so information's worth two qualities per patient treated twice as much as the value of providing access to the technology and so we can do this formally here you go this is evpi it's the difference between the expectation of the maximum net benefit and the maximum expected net benefit now of course we're never going to fully resolve this uncertainty so this is giving us an upper bound on the potential value of acquiring more evidence provides us with a sufficient condition of deciding to require more evidence to be to be collected just another way of saying the same thing but it's a kind of slightly different graphic this is from a recent paper which is doing value of information for an hiv self testing program in malawi it's the same idea but just a different way of looking at it is that on the on the left top panel we've got the relationship between a parameter value and net health benefit at the at the at the red star you can see the points at which our net benefits become positive if you like that's our trigger point and below we've got the distribution that we believe our parameters going to take now on the left we would expect with current information that actually this intervention is going to be cost effective and worthwhile we've got positive net health benefit but actually there's a chance we're wrong and that's given by the tail area the red tail area at the bottom and in the top panel it shows us the consequences in terms of loss of net health benefit as a consequence of that uncertainty on the right it's pretty much the same thing except this is the case where we given parameter values this technology does not look worthwhile so based of current information we would say no and get zero but there's a chance that we might be wrong and it might be worthwhile given by the tail area on the right and the potential to gain net health benefit if we could only know that that was the case so we can also use this to start to think about that question about well when have we got enough evidence and what's the value of making sure we implement what current research findings tell us is the best thing to do versus continuing to do research to try and figure out what the best thing to do is and this is a relatively it's not actually recent anymore but this was a paper based on work we did for PCORI in the United States which allocates the five billion of comparative effectiveness research allocated in the Obamacare affordability care act and this is just taking a very classic meta-analysis a real classic meta-analysis of the trials looking at thrombolysis following MI and looking at the effect of streptocinase so this is very historic it was one of the first cumulative meta-analysis ever conducted and what you can see is these these clinical trials have been lined up in time order and the and then we've got the cumulative meta-analysis on the right hand side and what you can see is that the red arrow points to European three in 1979 that was the first time that we had a clinical trial showing a statistically significant effect of streptocinase on mortality and the cumulative meta-analysis also showed a statistically significant result the question is was it sensible to wait until then before we made sure that everybody gets streptocinase well we can do that by looking at the value of information on this clinical endpoint of mortality and this is kind of what it looks like European one which was really early in the sequence I mean pretty much throughout the cumulative meta-analysis shows that streptocinase is a really good idea and the value of implementation per year is saving something between six and seven thousand lives per year possibly more at European one though we're very uncertain so the value of resolving that uncertainty and continuing to do trials was also very large indeed now the question I guess is posed if you were if we're at the position of European one the question is okay we could implement now but we won't be able to resolve the uncertainty and to gain that value of information which could actually avert more than six thousand deaths in the future how long is the research going to take if it's only going to take a year so we only need to delay implementation for a year we're going to get information that's going to be valuable in perpetuity for every patient with MI who might need thrombolysis so at that point absolutely it makes sense to continue to do research but once we get to European three the value of implementing our research findings at that point are over seven thousand deaths averted the only thing we gain by delaying implementation and continuing research is 27 deaths it's never going to be worthwhile so you can use kind of value of information and looking at these two alternatives to figure out when is it right to call this and say you know what we've got enough now's the time to say implement I'm conscious of the time so I'm not going to labour too much these next few points but just to point out that everything I've said so far is looking at mutually exclusive choices between a range of alternatives for a specific patient group with a particular indication but we can also apply the same principles at a system level and this is just a clip from a very recent paper which looked at the HIV program in Zambia across nine regions so that HIV program there's lots of a range of interventions mutually exclusive non-mutually exclusive so the question is what package what collection of interventions do you want to put in your HIV program given the resource constraints or for a range of resource constraints and given the uncertainty and how much better would that be if only we could resolve some of the key uncertainties surrounding the performance of the elements of this package and kind of that's what you're looking at here the the green line represents the best that can be done for a range of expenditures on HIV in per capita terms across nationally across those nine regions the red dotted line represents how that would perform if only we could resolve the uncertainty so you can see at point a we could get the same health effects as we get at point a by resolving the uncertainty but spending three dollars less per capita so point e in other words that's one way to look at it that's transforming the health benefits of information into their equivalent dollar value in terms of this budget the HIV budget and that's what you're looking on the on the bottom panel looking at the the value of information for the HIV program in dollar terms it's kind of telling you how much of your HIV expenditure might you wish to devote to resolving some of the key uncertainties in order to maximize impact on health a bit difficult to summarize a big piece of work in a few seconds but there you go it's just a theatrical problem we can do it at a system level as well just briefly I don't want to labour this I'm just going to click through these slides we can use the same framework to identify what type of evidence it is that we need what's the value of theta one compared to theta two we can use that same framework to start to examine sequences of research do we want to do research at all do we want to do research on theta one or theta two or both but also the sequence do we want to do theta one research before we do theta two or theta two research before we did want to do theta one and we can do that we can do that and and that it can be quite important in the sense that a very small very quick very cheap piece of research might provide results which means we don't have to do the really big expensive piece of research that's going to take years for example and that sequence might be much better than doing them simultaneously but of course everything I talked about so far is this kind of necessary condition up abound on the value of evidence what we really want to understand is what is the value of actual sample information for a particular research proposal and we can use exactly the same principles the value of sample information is just the difference between the expected value of decisions based on the predicted posteriors from the possible sample results given our price and a decision based on current evidence and the difference between that measure of benefit and the cost of sampling which includes resources but also time and opportunity cost gives us if you like the net benefit of sampling or the payoff the societal payoff from conducting actual research and with that metric we can start to answer a number of quite interesting questions we now have a sufficient condition we do want to do research if these net benefits are positive we can solve for the optimal sample size is the one that maximizes these these net benefits how do we allocate between arms of a trial for example which end points what length of follow-up what comparators to include what combination of studies well we evaluate all those possibilities and pick the one with the highest expected net benefit there you go don't have time to talk about it much but that's that's a particular example in influenza three studies we could do the clinical trial top right survey to understand quality of life or an epidemiological study to understand incidents better and we can identify the value of those things and also their optimal sample size as well I just want to finish up on this I hope it works it may or may not but of course there is a link between uncertainty the need for evidence and the price we should be willing to pay for a new technology on the x-axis is price on the y-axis is net health benefit and as pricing if we reject we just get zero right we get zero net health benefit as price increase when price is very low we get some net benefit from this technology as price increases that declines and ultimately becomes negative at that point marked a stroke are that approval rejects kind of boundary how do we add uncertainty well this blue line represents what we could have had if we could only immediately resolve all these uncertainties so that blue line represents the expected value with perfect information if you imagine that we could costlessly and instantaneously resolve all uncertainties but of course we can never be on that blue line ever research takes resources it takes time and there's only two ways to get research you can do the research whilst you approve that's awr approval with research or you can do the research whilst you withhold approval in other words only allow only approve the use within a research design now I fear I might have run out of time and misjudged it but what I'm going to say I'm going to get to the end quickly to preserve time for discussion so with these two ways of being able to generate research what does it tell us about the relationship between price and this approve delay require research decision well we want to be on the outer envelope of these four possible payoff functions in other words when price is very high we're uncertain but we're just going to reject and we don't need evidence because the price is so high the uncertainty is never going to get us to change our mind it's just not affordable slightly lower price actually we still would reject based on current evidence but it's possible that research might get us to change our mind so that's an only in research recommendation so restricted restricted access to the technology below that red line that's where we think on the balance of evidence this is worthwhile but if we're only just convinced then sure we're going to give unrestricted access but it's going to be conditional and conditional and doing additional research and if you cut the price low enough then we're simply going to approve and remove the need for you to conduct more research so in a sense there's a there's a menu options ranging from the far left unrestricted and conditional access to the far right which is totally restricted access and a straight reject and if the list price is given by that blue arrow then you can start to see the scale of discounts we're going to require from manufacturers to get her into into these different categories of conditional and restricted guidance just going to show you one more thing in many circumstances if we approve we can no longer do the research it's certainly true in medicine we're not going to be able to do randomized trials we need to remove that the awr line what happens when we do that what happens when we do that is it means that actually we're going to have to have an even bigger price discount before we approve you for use because by approving you for use we are giving up the benefits of acquiring more information about your product so I'll just finish on the final slide implications and prospects I think we've had we do have we have had an impact on on UK policy in terms of health and I think also to some extent in other high-income countries and in some low-income countries as well I think also at global bodies Gates Foundation in particular but also WHO WHO removed their one to three times GDP per capita there's current discussions around tiered prices going on I think the value of information stuff has had less impact than the kind of cost effectiveness and we need to understand health opportunity costs so far I think for me at least what motivates me is not the geekiness although I do like geekiness what motivates me is it's about accountability and I think decision science when applied to social choice can add to the accountability of current social arrangements at an international level exposing global inequalities at a national level exposing the fact that our NHS generates health for very small amounts of money and a local level in terms of choices that have been made on our behalf so I'll leave it there and apologize for going over what I how much time I plan to use on this. Thanks very much Carl that was very informative we have a few questions already coming up in the chat and I mean certainly some of them relate to you know my observation is that these kind of methods have hardly been begun to be used in environmental research and monitoring in terms of prioritizing research and investments so one might ask the question why I mean they've been widely used in health although I think you've had some struggles to get them adopted at some stages as well but we're a long way from from getting that in terms of environmental topics. I think it's fair to say that certainly struggled and continued to struggle to really get concepts of value of information embedded in decision making that that that is still a struggle the earlier I think we've had a lot of stress but the value of information remains a struggle. I mean Steve sort of asked the related question is you know how these approaches be employed to explore environmental gain and the cost of schemes to improve environmental status is giving example of the new post-Brexit agriculture environment we've introduced concepts of public good from public money you know what are the learning points that we can carry carry over into the environmental sector. For me like I tried to express at the start you know I'm fundamentally an economist and kind of fell into health almost by accident very early in my career and it became the best place to have opportunities to pursue kind of accountable quantitative work with a policy audience. For me it just trans it all translates it's just decision science right so it does translate you're not going to be using qualities but that's fine I mean you do this you know kind of cost-benefit framework as well. I suppose the things that I would I've just expressed my mistakes I think I made a mistake early in my career which was that I tried to sell a method because it was really neat rather than understand that before people buy anything they need a problem you need to be the solution to a problem and I didn't spend enough time on making sure people understood they had a problem for me to be able to sell a solution to them and I think the other thing I did was people often said well you know Carl yeah but what about this and you missed a bit and you missed a bit and you missed a bit and I devoted an awful lot of my career early on to making the methods more sophisticated and trying to capture all the things that people said were missing and I kind of missed the point that the reason why they didn't want to use these methods wasn't because I've missed a bit it's because they didn't want to be held accountable and and I diverted too much of my energy into imagining that I could make this so swanky that I'd have captured everything and of course you can't capture everything ever so I guess that you know if I had my time over again that's the lesson I would learn make sure people know they've got a problem before you try and sell them something and secondly when people say you missed a bit well is that why they don't like it hmm not so sure yeah I mean um one um Ron Costangio raises one issue which is I think makes it more complex for us the environmental sector because you know you use quality adjusted life years in most of your analysis is sort of which is one dimension and we're sort of dealing with um you know land sea air uh environmental health and there isn't sort of one um you know one metric we can use for just health one of the biggest arguments we get into is actually how to measure health I mean even in soil science which is the the there's the relevant several of us here you know we get to to end with arguments and debates about how do you define measure soil health just that's just one tiny component to the environment um so we're all faced with that kind of level of complexity so have you got any advice on that on that way in that area absolutely I think I think that's I think that's absolutely right I think even in health people are still arguing about how to measure health you know that that that kind of never ends and people argue it from principal positions but what we find is you know manufacturers and their supporters constantly arguing about you know you can't use qualities you know qualities are this qualities are that um so I don't think that ever ends even if you just take that narrow dimension of health I guess what I would say is that um the real difficulty I think is okay spit it out Carl mainstream economics has got a particular view of how to characterize social welfare based on markets and their surrogates and that's what lies at the heart of cost benefit analysis that is not a social a definition of social welfare that I recognize or I think is widely recognized and these ideas of social welfare are quite rightly profoundly disputed and that kind of leaves you in a bit of a difficult place when you've got multiple effects that need to be brought together certainly what we tried to do for decision making when we have done projects that go way beyond health health is one component of the benefit is to try and set out the analysis saying look if you this is this is this is the implication given these sets of trades between these different attributes and this is the evidence about how people have thought about the trades between these attributes but it's not up to us as analysts to tell you what those trades ought to be or to impose a social welfare function but if you choose alternative a versus b it implies that you believe this attribute is 10 times more valuable than this attribute do you really want to go on record saying that minister? Steve raised a point that was similar to something else going to raise as well and that's what scale do you sort of make these trade-off decisions I mean you could look at your research budget within a department and say well how is the money best spent in terms of the expected benefits or that could be done at national level you could even do it at global level and say well you know a lot of the funding should go into developing countries because you're getting so much greater benefit to the amount spent I guess it comes down to you know who owns the money at the end of the end of the day but I wanted if you got into those kind of issues in terms of the scale at which you make these trade-offs absolutely so I guess the way I would characterize it is unlike some economists mainstream economists we don't believe our job is there to impose a view of social welfare on the rest of the world and imagine that everybody's done what we think ought to be regarded as efficient rather our job is to say look for example let's take the NHS given the amount that you've devoted to the NHS this is what you're getting at the margin it's about £10,000 per qually are you happy with that I hope I'm making sense of it so we've done work that's broken down health expenditure in the UK so we've estimated this for public health expenditure at local authority level that's £3,000 per qually we've done it for social care so it's so we regard our job as setting that out saying look you know this is this is what's happening at the margin are you happy with that do you think this is reasonable that you cut public health expenditure but protected the NHS given that it's only £10,000 per qually for the NHS how does that play out when you do your public expenditure review I hope I've I'm not sure I've really hit you I think I think what I would argue is that our job certain the way I see I see our job is to express the value of the constraints as they currently are communicate that effectively to those in positions to change those constraints by raising more taxes reallocating public expenditure or doing other things to release other constraints but by exposing the real value of those constraints and their relative value at least we make them accountable for their action or inaction I guess that's what I would say that I don't think I don't regard us and my research teams is in the position of telling society how they should reorganize things but exposing this is the way it is right now are you happy with that yeah perhaps a final one comes from Alex Bush I've seen a lot of interest in the health benefits of more healthy environment and fact there was something on the news yesterday about the health of the oceans benefiting human health in an argument that public health investment in our environment would indirectly save money on public health expenses given how much we struggle to characterize environmental health and the points about called to adopted life life years where would a value in a value of information begin to address this I guess in terms of trying to prioritize you know what the gains from research which include public health benefits but I'll let you handle that yeah yeah it's a bit like early stage modeling in a way some stuff that we do which is everything I've said so far is about evaluative research in other words what is what do we believe the value is of acquiring more information about interventions which are kind of already out there the other question is I hope this is addressing your question the other question is what is the value of developing new stuff and the value of developing new stuff requires us to think carefully about what we think that might look like and how uncertain we might be and okay apologies I've drunk too much coffee today I'll try and say it more clearly there's a kind of real options thing going on here that when you're at an early stage of developing an intervention and you don't know whether it's going to work and you're going to have to put money into that r&d program it's wrong to just look at your expectation of the net present value of that effort because actually by putting the effort into developing that you actually resolve some of the uncertainties as well in other words the value of that effort is both the net present value plus the value of the information you get through that effort once you've developed that and then the question becomes okay should we implement it and by implementing it you shut down the possibility of acquiring more information about it then actually it's kind of the opposite then uncertainty makes that purchase less valuable because you're giving up the future evidence so I kind of think that's part of it that if what you're talking about is developing new strategies that you don't know how they're going to perform and that requires some upfront investment like r&d to figure out what this strategy is going to look like and how it's going to perform then actually the value of information makes those efforts more valuable because you're investing a real you're getting real option value here I'm not sure if that works yeah I know I have some other thoughts about that so double benefit before I think we've been wrapped it up here but I mean I'd be really I'd really like to see some of these applications being taken over into environmental prioritization even in prioritizing environmental monitoring but also intervention strategies so I really hope we can see some of this work coming across being applied with that with thank you very much indeed Carl for giving this talk and giving us these insights it's been quite fascinating to learn from the health sector and we'll put up the webinar on the site for people to download and see if they're leisure and thank you once again for taking the time to deliver this thank you yeah thank you Carl that was great thanks guys thanks to Charlotte and NERC for hosting hosting this webinar yeah thanks very much