 Hello all to this much darker evening than we're used to of late. I'm Mandy Charman. I'm a member of the Victorian Regional Australian Evaluation Society Committee. And that's why I'm here today. But I'm also the Manager of the Outcome Practice and Evidence Network at the Centre for Excellence in Child and Family Welfare. So I'm here, luckily enough to host this session with Kevin Robbie. First of all, we will just acknowledge the traditional owners of the land we are each on today. It's been a beautiful weather and there's nothing better than beautiful weather to make you really appreciate where you're located. I'm in the lovely Macedon Ranges. I'd like to acknowledge the traditional owners of the Wurundjeri lands and Elders Past, Present and Future and any Aboriginal people in our group. Welcome. Just first, before we do formal introductions to Kevin, there's a little bit of housekeeping. It's always wonderful for a presenter if people leave their cameras on. So if you feel comfortable doing that, but be aware we are taping because we're hoping to be able to share this recording up to the event for those that couldn't make it. Kevin is really wanting an interactive session. Every speaker loves great questions. So feel free to put questions into the chat as we go and as they occur to you. And I'll collect them so they don't get lost because I think Kevin's got some interesting questions to ask us to respond to in the chat. I will collect them and we'll put them back to Kevin at the end if you have any technology issues, usually getting in and out sort of helps or message me individually and we'll be able to sort something out. If you can make sure that you have your names, your proper name on your Zoom face just so that we know who we're talking to. Sometimes we inherit our children's Zoom accounts and sometimes it can be a bit sad. I think that about covers it and thank you again for making this time available. Now I want to introduce Kevin Robbie. He's presenting on this with this magnificent title, Frank from the Magnificent Seven to the Hateful Eight, Why Social Return on Investment is a Good Evaluation Approach. I'm just so curious about the title, it's so cool. Kevin is the Managing Director at Think Impact, a leading social impact and sustainability consultancy in Australia. Prior to Think Impact, Kevin held a variety of executive leadership roles in Australia and the UK. He started his work on evaluation and social return on investment in the UK in the 90s, was a special advisor to the UK government on the role of social return on investment in impact investing. Now he is an advanced accredited practitioner having worked on 50 social return on investment reports is on the Insurance Panel for Social Value International. I'll hand over to you, Kevin, looking forward to it. Thank you, Andy, and welcome everyone. I can only see about two or three faces. It's always strange doing Zoom sessions, but I'm going to try, as Mandy said, to make this a bit interactive as we go through it. I should also point out the obvious thing here that I have an accent. I'm somewhere on the following spectrum. The very first time I spoke in Australia, somebody came up to me afterwards and said, I didn't understand a word you said, but I could listen to your accent all day. The last time I spoke in Scotland, I got told I was sounding very Australian. This is apparently because I now go up at the end of my sentences and sound enthusiastic, which is culturally inappropriate if you're Scottish. But hopefully you're able to follow what I'm saying. The backup version here is we've got the recording and if it's really, really dreadful, we can put subtitles into that for you. I'd also like to acknowledge that I'm dialing in from Wurundjeri land in the Kulin nation, pay my respects to elders past, present, and emerging, also to acknowledge this land was never ceded. Mandy was kind enough to do an introduction to me, so I'll not dwell a lot on a think-impact of my role. Other than to say, as Mandy pointed out, I've been in the sort of evaluation space and social return on investment in particular for a long while. I did my first SROI in 2002, 2003, and I've worked both sides of the fence in the sense, or maybe three sides of a fence actually, in the sense that I've predominantly spent a lot of my career running not-for-profit for-purpose organizations trying to implement impact measurement approaches, trying to make sense of SROIs that have been done for my organization and what that might mean for us. I've carried out a lot of social return on investment analysis over the past 20 years, or coached or mentored other people in terms of how to do the approach. But I also spent 2007, 2008 as a special advisor for the UK government thinking about this, particularly around impact investing. So I bring some different perspectives to social return on investment and social return on investment as an evaluation approach. I'm going to go through some of the background and principles to social return on investment, but end by talking through sort of three live case studies, and then we'll have a chance for sort of Q&A at the end of the session. And hopefully I could answer the questions that either come into the chat as we're going through the session or that you've got at the end of the session. The other part of this is social return on investment, and we know this in the evaluation space, it's jargon heavy. There's a lot of jargon. Hopefully we're getting out of the people online here who understand some of the jargon, but I don't want to lose you in other elements of the jargon. So if I'm talking about stuff that you're like, I don't really understand that. Again, put the question into the chat and we'll come back to it because there's technology that I'm sometimes so steeped in that I use without thinking about what's the plain English way to actually say that. But it would be good just to start from me to get a sense of where people's level of expertise is in terms of social return on investment. So quickly, just put it into the chat. I know nothing. I know lots. I'm also an assured practitioner, something like that that, you know, just a one line that sort of gives me a sense of where you are as an audience, because then I can sort of dial up or dial down how we actually go about some of the elements of the presentation. So I'll just wait to see stuff coming into the chat around that. As I say, the other bit of this is I am going to try and make it interactive. And so there'll be points where actually I ask you questions and we'll hope that you put an answer into the chat or else I'll sit feeling very lonely at the end of my Zoom screen here. And as Mandy said, if you've got questions about social return on investment as we go through, then I will try and get to them at the end. But social return on investment has been around for about 20 odd years, maybe 25, originated out of the US from an organization called the Royal Gold Center Prize Development Foundation, now known as Rediff. And they were trying to think about the value that they were creating as a funder. It then came into Europe shortly after that. And as I say, I got involved in my first SROIs in Scotland back in 2002, 2003. So through that time, the methodology, the approaches evolved when I was at the UK government, one of the jobs I had to do there was to do an analysis of how many had been carried out in the UK over the sort of eight or nine years, no, five or six years to that point. And we only found about 50 or 60 SROIs that had actually been done. And then look at what the quality of the work was and what needed to be done to improve the approach. And that was bringing in standardization in terms of the methodology, improving the training and actually helping to set up what became social value international. So it was originally the social return on investment network in the UK. And then it became social return on investment network worldwide and has now become social value international. So operates in 30 odd countries. And so as Manny said, I'm a member of that and on the assurance panel as well. Great. Now, I often been able to see the chat. I've seen there's the chat there. Good. All right. So limited, limited, keen to hear. Okay, that's great. Right now, let's get again another interactive piece for you. So this is a top floor apartment in Sydney. It has two bedrooms. It's been very tastefully renovated with an amazing kitchen. This isn't where I live. I live in Melbourne. It's in a built up area with high density population in the neighborhoods, very noisy. Can you quickly either write down or take a mental note of how much you think this apartment would cost if you were buying it? Anyone want it? Okay. We go. Give you a few seconds for that one. Okay. So you've got your written down or major mental note. Okay. Well, somebody put it into the chat. 800K. Okay. One million, right? Okay. So we've got cut the written down. What do you think it's worth? So it's actually in the suburb of Kirribilly. It's a few minutes walk from a train station, takes you straight to the other key stations in the CBD in Sydney in around 10 minutes. So now you have the opportunity to revise how much you think it's worth. And yes, the building was ugly. Okay. All right. So we've had a few jumps here in terms of costs. And then, if you're coming in and then the last one here, this was the view out of the main living room window and the bedroom windows. If you leant forward from the balcony on the living room balcony, you could see the bridge as well. So it's two minutes walk from a ferry stop and in the vicinity there's an open air pool, several parks, good local shops, bars and restaurants. So let's do our last guess at how much we think that might actually be worth. Okay. So with a few bits of information, we seem to have gone from the lower end of the spectrum year was 800,000 to two and a half million in terms of people's guesses. And so it's interesting spectrum of what actually helps you understand things about value. Now, I didn't live in that apartment. I had friends that lived in it and I lived on the ground floor. So I could see like the top of the opera house while I rented there. But it was really interesting because that apartment was up for sale when I moved into the building and in one week it lost $350,000 in terms of its value. So it was up for sale for a certain price and then one week later, it was up for sale for 350,000 less and nothing about the view or the actual, the way the apartment was built or where it was situated has changed. Can anyone have a guess at what might have happened this 2008 to change the value of this apartment by 350,000? And you can come off mute and shout if you want. Maybe some kind of major body corporate issues. All right, major body corporate issues. Yeah, cladding. Yeah. It was September, 2008 if anyone can cast their mind back to what we were all doing in September, 2008. Yes, the global financial crisis. Okay, so one week before the global financial crisis it was on sale for 350,000 more than a week after Lehmans had shut down. And so confidence affected the price or the value of this property. That was it. It was just a change in confidence and 350,000 was wiped off the value. So one of the bits that is interesting about coming into the social return on investment space is people say, oh, it's really hard to value soft outcomes. It's almost impossible to value soft outcomes. What we should be valuing is things that are hard and firm like bricks and mortar. But just through that little vignette there, you can see that actually there's a lot of things that affect the value of things that we think we know how we value. And so I'm not saying that social and investment is perfect, but it's moving in a space of attempting to put a value on software outcomes, but recognizing that even in the hard and fast world that is hard to do at points. Another quick one for you to just chuck words into the chat. So if you were thinking about the principles of good evaluation, what would they be? What would be the principles that would underpin good evaluation use? So we got logic of the evaluation, objective perspectives, standards, love design, all right, stakeholder engagement, independence, good detailed info, relevant assessment tools, okay, good. Understand the client need, what's being valued. Yeah, great, okay. So some good things coming in there in terms of what underpins and people keep on putting them in as I go to my next slide. So one of the bits that I've always liked about social value and social return on investment, I describe it as the sort of magpie of evaluation methodologies because it's collected bits from other evaluation methodologies that's gone along and we'll talk a little bit about that as we go. But now, SROI sort of sits in a suite of ways of thinking about this concept of social value. And Mandy and Kala and I were talking about this just before I came, just before the session started that when I worked for the UK government, one of the challenges for social return on investment is that it's expensive to do it properly, it's costly. And when I was at the UK government, they went to me, what we'd like you to do is think about can you make this cheaper and more accessible? What would it take to do that? And this was back in 2007, 2008. And at that point, there was way other bigger problems with SROI in terms of getting the standards in terms of how it was done, getting people trained to do it to the same standard, et cetera. But I kept that in the back of my mind and over closer by time working for social benefits Australia and now with Think Impact, being part of the movement to actually think about how we do this and how we make this whole area of social valuation more accessible. And by that we mean cheaper in some circumstances. There are now really five different approaches that are sort of immersed in this space. Social value appraisal tends to be done upfront before money goes into something. It's the equivalent of doing a sort of social due diligence on an entity, done a lot by some of the impact investors in the world. And we did pioneering work for community sector bank a few years ago to help them develop a calculator in this space so that as they were doing their business due diligence on something they were going to put impact investing in, they could do a sort of social due diligence on it in terms of what the potential return was. Social rating and weighting is a piece of work that is being driven out of the impact management project from the UK. Again, looking at what's the relative value of certain outcomes, what's the relative value of certain programmes and approaches and can you rate and weight these? Social value modelling is happening a lot, particularly in the corporate sector. And one of the, we were talking earlier about this, one of the best examples is Lend-Lease who have set themselves a public social value target of creating 250 million of social value by the year 2025 from their corporate Philanthropy CSR Foundation activities. And they are now tracking and reporting against that using a social value modelling approach to do it. Then there's social return on investment which is more rigorous and particularly requires reports to be assured to ensure quality around that. And then you get down to cost benefit analysis which has done a lot in and by government in terms of understanding the relative costs and merits of programmes. And so all five of these approaches now exist in the social value space and sort of sit alongside each other. We're going to look at social return on investment because it's the one that we're going to talk about the most today. So there are a set of principles that underpin SRY and this is the sort of explanation of the title of the presentation. Several years ago when I was doing the presentation I used to put this slide up and I'd asked people what it was and it's the magnificent seven film for those of you that recognise it. But one time I had to do an emergency trip back to the UK and one of my colleagues at SVA had to do a presentation for me. And when I came back they were then presenting another event using my slides and they put this slide up and they went, well, the reason we have principles is because it's a bit like the Wild West if you don't. And I was like, no, it's the magnificent seven. Well, I didn't shout that obviously in the audience. But then I went, well, that's actually quite good. It's because it's a bit like the Wild West if you don't have them. I caught them afterwards and I said, why did you not say the magnificent seven? And he went, what's the magnificent seven? And so Gen Y, no acknowledgement of great movies and no understanding that this film would ever be made. So I lobbied Hollywood to get it remade just so that I could retain having the magnificent seven as the set of principles that people would do. And I used to get people to hum the tune to the magnificent seven so that they would remember that there were seven principles that underpinned SRY. I can't do that online obviously. You would just have me humming the tune and that's not really what you want. But then social value international came along and added an eighth principle. And that really stumped me until the hateful eight came out by Quentin Tarantino and I could keep my film theme here. So really it's just to remember that there are eight principles that underpin SRY and underpin the approach. So what we're gonna do now is another interactive bit where you will get another image appearing on the screen and we'll see who can get the fastest typing as to what you think the principle is. Usually people would shout out, well, let's go for fast typing. Silence, no typing, no clues. Transformation, it's in the disguise. It's about theory of change. So you're in the transformation business. When you're wanting to evaluate something it's usually because you think something's changed. And so you've got to be able to understand and articulate what that theory of change is. Second principle usually gets a grown in this one. Involve your stakeholders, okay? And so it's the big principle and some people have it as principle number one in terms of social return on investment. And it's what differentiates it quite dramatically from cost benefit analysis is that it's a stakeholder driven approach. Third principle involving the stakeholders and maybe I don't need to dwell on this a lot but it's critical when it came out because the more stakeholders you involve the different perspective you get on what these outcomes might be. So within a scope to what transition program three different stakeholders have three different perspectives. And I remember working in, I used to work a lot in the employment space, talking to government about what were the benefits of getting someone a job and the people would be reduced taxation and increased taxation and reduced welfare spending. And I've never met anyone who got a job who saw that as the benefit. And so really it's about broadening out that understanding of what benefit is actually happening for people. I'll maybe try and get a buffy picture for the next time. So next principle. Oh, why is it not clicking? Any guesses, range of values. Yep, close. It's value what matters, okay? So again, talking to the stakeholders what's changing for you? What do you value about that change? What's the relative value of that change? Social return on investments got there the notion that you put a ratio on that value. And so it's about placing value on the change that's happening. This one's a bit obscure but any accountants in the room, the principle from accountancy. A golden thread of logic is really good. When I was at SVA, they wrote an article on the golden thread based on this principle out of social return on investment hence the image, but it's actually materiality. So you can't measure absolutely everything. You got to, you make judgments about what is the right thing to measure when you're doing an evaluation in accountancy, it's materiality. So, and then the materiality within social return on investment is driven a lot by what stakeholders see as the outcomes that are material and should be valued. Yes, transparency, someone got it in one. So be transparent. As you're doing a social return on investment you'll have to make a whole set of judgments. So be clear about those judgments. So from the social research background of concepts of reliability and validity that someone coming along, if they were making following the same steps as you and making the same judgments they would come to the same conclusions. So SROI has that transparency principle sitting behind the reports. And it's a crucial one in terms of assurance. Don't over claim. Someone's now looking up SROI principles online. Yes, don't over claim. You know, again, it was this bit around there are things to consider when you're talking about elements of social return on investment and elements of what's happening when you're evaluating and someone's just said attribution. It's a good one to think about. So any physician, no physicians, any mathematicians in the room. Now there's a lot of shaking of heads. That's quite good. So how many of you can remember Pythagoras's theorem? Yeah, there's a few people nodding. This is Robby's theorem of SROI. I'm trying to get it as famous as Pythagoras, but I reckon I have to die before that happens. But it's essentially thinking about how you think about impact. OK, there's there's people that you will meet that will go look at can't be determined without a randomized control trial. OK, and that's one view of of the assessment of impact. But if you're active in the field working in frontline action research, trying to understand what's changed, what's working and you don't have the money for a randomized control trial, then you have to go another way. And that's the background I've come from and a lot of the work that I'm doing. So when we are thinking about impact in an SROI space, you're thinking about what's the change? What's the outcome that's happened for someone? What's the proportion of that outcome that's not sustained? So the drop off. So for example, I can do a session on SROI for you. Some of you might leave this Zoom and forget about it by the time you have dinner. And some of you might, you know, I remember one guy coming up to me four years after a presentation and going, it was something that changed my life in terms of your presentation and it really set me on a different path. So outcomes can be sustained for a duration and they differ for different people. What proportion is shared with others? So attribution. So you might not be doing it all on your own. What would have happened anyway? The dead weight and are there any negative effects displacement? And if you're working in systems, evaluation systems, SROIs and a few of them that have been done, recognising that can't be done by one organisation, so what's your contribution? So, my clicker's not working. So these are five questions I think it's good to ask. They're not the only five, but the five good questions to ask if you're in a VWator. What changes this organisation trying to bring about? How sustainable is the change? Who else is contributing to it? What would have happened if nothing had been done? And what are the negative effects? And when I used to do work at social ventures Australia, and we used to say this to social entrepreneurs that were trying to set things up, it's like you should be asking these questions of your entity. Like what are you trying to do? How sustainable is it? Who else is contributing? And who else is doing what you're doing? What would happen if you did nothing and you couldn't cause any negative effects? And so they're good questions to frame into an evaluation sense, but also into getting people to think about what they're doing. Principle seven, this was stolen from social accounting. So get your results verified. So, you know, this can be taken the results back to the stakeholders to talk to them about, right? We have, these are the outcomes, these are the relative value of the outcomes. This is the ratio that's come out of this and get the stakeholders to verify it. Put your social, so we do this a lot of think impact. We do SRYs and they go for peer review within the team or get the reports assured where they go into the assurance panel and they get peer reviewed by international peers who can look at them. But again, it's making sure that people are adhering to the principles and the process of doing SRY properly. And everyone I've done probably 20, 30 assurances now, everyone says it's a really beneficial process to go through from carrying out evaluation, going into a process where you're reviewed and challenged and it helps to refine and make better the SRY. And then the last principle that was added, be responsive. So when you have, when you've done this, when you've done an SRY, actually do something about it. So it's very much the, right, what are you gonna do now? How, you know, what have you learned? What's gonna happen differently in the organization or in the corporate or in the funder as a result of this? How are you gonna be doing things differently? And so SRY, Social Value International brought this principle in about a year and a half ago, they're developing the standard for it and it'll become part of the process going forward. I think it's real critical one in terms of this concept of evaluating to improve things. But they're the principles that underpin it. What does it look like in practice? And I'll give you three quick examples of SRYs that we've done and the links to them are on here. So what I'll send the slides through, man, they can distribute them. You can click on and see the actual SRYs. So we did this SRY for Vacho a few months ago. This was the culture and kinship approach. It was funding out of the Department of Health. It was into four initiatives, so four of the ACOs that were looking at the role of culture and kinship in terms of improving community health outcomes within an Aboriginal context. One of the bits that was interesting about this SRY report, I think it came second in the Seminole Awards recently and we did it in partnership with Aboriginal evaluators called COA. And they have a process of impact yarning, which is culturally sensitive in terms of stakeholder engagement. So they trained Vacho staff and then went out with Vacho staff and did the value yarns and then the impact yarns to collect the understanding of what changes. And then they trained our team to go out and do the value yarns within the Aboriginal communities to have the conversations about the value of outcomes and the relative value of outcomes. So as I say, it's a really high quality, I think, evaluation. Interestingly, we went for assurance in the international assurance panel and we got it assured by someone from Canada who had an understanding of working with First Nations people. And when it passed, they went not only as this passed but we actually think this should be a case study as to how evaluation is done in Aboriginal communities or First Nations context because they said it was so good and they wanted to use it as a case study for working in Canada. In terms of the SROI, it was $8.29 of value that was created for every dollar invested and Vacho are now using this for discussions with the Department of Health in Victoria around ongoing funding and expanded funding into the programs that they're running through the ACLs. Wise Ways to Walk is a program run by Wise Employment and Wise Employment are big, what used to be disability employment services, what was Job Active, Workforce Australia provider, but they have this innovative program that sits in their organization which is working with people with severe mental illness and trying to help them return to work. It's called Wise Idea now, it was called Wise Ways to Work and we were asked to do the SROI of this just at the run-up to COVID. And so it was interesting evaluation because we were evaluating it but we're evaluating it in real time as they moved from delivering it all online, all face-to-face to all online during the six months of the evaluation. And we were able to look at the difference in value that was created through that. It was also a, they had links in the NDIS wanted to look at it in terms of what was happening for the people in the program. And so again, it was really high quality of a piece of work. The other bit that was sitting in it was we have a model called Enduring Impact which is an employment specific model that underpinning it is this concept that if you're running employment programs there's actually three dimensions you should look at. Intrinsically, what's happening to people being a part of the program, extrinsically what's happening in their networks and what are they getting in terms of foundation skills and knowledge and that if you can run a program that covers all three, it's got a better chance of success. And so one of the bits that came out the wise ways to work on that wise idea approach was that they were working heavily in all three of those areas and the program was delivering a $4.97 return. And so what wise employment have now been able to evolve that program further on the basis of the SROI and are running it in different formats now. And then the last one, which might be one of the interesting ones here was a Hope Street first response evaluation. And this one was interesting because we didn't do this evaluation. I don't know if there's anyone online from Lerata but this was actually an evaluation done by Lerata. But what happened was Hope Street to a homelessness service had a deal with one of the big four companies to do pro bono piece of cost benefit analysis for them. And then that company pulled out and couldn't do the work and we're gonna leave the evaluation high and dry. So the Ian Potter Foundation paid for Think Impact to do a social return on investment analysis as part of this overall approach. But what we did was we used all the data that Lerata had collected through their evaluation. So they had gone out and done the stakeholder engagement, they'd done survey, they'd done all the work on the theory of change and we just came in to actually do the social return on investment modeling. So it was a really interesting piece for us because we'd never just done that. And it was quite interesting to see that bolt on to an evaluation approach by a well-established evaluation company. And actually how we could partner around that. And again, showed where the first response program out in Melton was working really well and had a $3.14 return for that one. And Hope Street were again able to use that as a sort of lobbying advocacy approach. So I think that's it from me in terms of any questions. Right, this is for me to jump in, Kevin. I always have lots of questions. I'm gonna start because there's some really obvious questions that I think probably everyone is really interested in. I've sort of been blown away by your content, the content and the delivery actually as well. Really interesting, wondering if you could just go through what in practicality are the key steps? So we have the principles for, this is the IAS. So we're thinking about key steps, methodological steps. Would you be able to walk us through what might be involved? So essentially depends on who's doing it and how they're doing it, but there can be anything from three to six or seven steps that people would go through, clarifying the scope of the evaluation. So what's to be evaluated? And that can be the years. So for instance, we are starting an evaluation at the moment and the organization has data from halfway through FY19 up to the current. And so we can't do the full FY19, we're actually just doing FY20, 21, 22, and then we'll do through to them, we'll finish it so that they've got the data from FY23. So it's clarifying the time period to be evaluated and getting clear on that. Then you work out with the organization, the theory of change that they have and how that thinking came in to, to help the program get established, what actually has changed over, anything altered from the theory of change over that timeframe. Usually there's the development of an outcomes framework at that point if the organization doesn't have one so that we leave them with something for ongoing measurement, but we would develop an outcomes framework or utilize the existing outcomes framework that's there to clarify what those outcomes were. A lot of that work is done through stakeholder engagement. So you go out, you talk to the stakeholders and partly that is to understand the change that's happening for them, but also to understand the relative value and what changes are more important for stakeholders. And then from that way, we then go and look at identifying financial proxies for those outcomes. And there's a set of established methodologies for developing financial proxies. So there are cost-based methods. So these are actual costs or notional costs. There's willingness to pay. So stuff around reveal preference and stated preference. And these are all established ways of doing cost-benefit analysis. And then there's a couple of other SROI specific approaches. So there is what's called wellbeing evaluation that has come out of the UK. It was developed by a guy called Daniel Fujawara and his team in Symmetrica. He was ex-treasurer from the UK and they've developed this wellbeing evaluation approach. It's used by social value bank in Australia and so they have a bank of proxies. And then there's another approach that's developing called anchoring, which is getting the value around the main proxies. So using the main proxies, the anchor to understand valuation. And so this comes back to when I was talking about transparency, you then you make choices around what proxies to use and you're just transparent about why you use this proxy and this circumstance for valuing this outcome for this stakeholder. So that someone else looking at it would go, yeah, that's a reasonable way to go. And it can be, look, it was easy to access. There was a lot of data around this. And so we just use this, we found this in other SROI reports. And so we want to benchmark. So we've used the valuation methods from here. Once you've done that, then you build a social value model, which is usually a sophisticated Excel spreadsheet. And then you calculate the investment, you put in the, the series of factors that I talked about there. So you look at the outcomes, you look at the outcome incidents. So how, you know, you had a hundred stakeholders and this was the outcome, but did it have, did all 100 of the stakeholders achieve the same outcome? What happened to those that didn't, did they achieve a degree of it? And you have developed subgroups and subcategories of outcomes. And then you look at drop off displacement attribution and dead weight. And again, work with the stakeholders to understand what they're understanding of these factors might be. Going into research to look at, okay, there's research here that tells us that the dead weight is X or Y. And so you apply the thinking to that and then that helps you churn out an SROI calculation, which is a ratio for every dollar invested X dollar of return. Anything over one for one is seen as a positive result. Anything under one for one is seen as not positive, but there are degrees of not positive obviously. If you go down to minus 10, then it's probably more severe than 97. And then you take that back, you verify it with the stakeholders, I get it peer reviewed, you write up the report and then we would say all reports should go for assurance, but not all reports do. I reckon only half of our SROI reports that are produced go through the assurance process because it can be quite rigorous. And people tend, look, if there's a, you know, if I, there's critiques I would have of SROI and I think there are some companies that try and push the ratio up and let's go for a maximum ratio. Thinking back, we tend to be more conservative about what we're doing. And if you go into the assurance process, you pretty much guarantee that someone will knock your ratio down. So if you're doing an SROI to get the biggest score, then you'll probably shy away from assurance. If you're like Vacho, where like we want a robust piece of analysis that we can take to government for a conversation, you'll go through the assurance process because it adds a degree of rigor to it. And what I can do is, I actually should have thought of that. I've got a slide with the process on it. I'll stick that into this pack before I send it over to you. That would be amazing. And I'm immediately thinking of, oh, I'd love to look at one of these sort of reports. And the three that are there are all publicly available. That would be really wonderful. That's why I picked those links. There's lots of them that are not publicly available. The Vacho one is on their site. LaRatta one is on the Hope Street site. And Wise Employment is on the Wise Idea. Wise Ways to Work is on the Wise Employment site. And so the links are there for each of them. Do you find as well with the calculations? So this is a very precise process. But the stakeholder engagement, I'm really interested in the way in which stakeholder engagement is harnessed to this process and whether the artifacts of that contribute to additional value as well. Sometimes I've seen these reports and they come up with a number. That's really cool. That is cool. But it actually has had all this work underneath it. And I'm wondering how that's represented through this process. I think there's differing views within the SROI community about that. I once produced an SROI, not a report, a compilation of a set of reports. I think there was 14 or 15. And I didn't put the ratios in the reports. And this is when I was in Scotland and people were like, this is scandalous. SROI is all about the ratio. Why aren't you giving us the ratio? And I was trying to make the point that the SROI was more than a ratio. And actually what had happened was 15 really good quality evaluations that had gone in and engaged with stakeholders, had identified change, had led to changes in each of these services. And actually that was way more important than the number. The number, I think, for my experience of being on the other side, when I worked in an organization, we had a social return on investment analysis done on us. The number was the headline that got us through the door to have conversations with people. I still remember it. We had £5.87 of return for every pound that was invested in the hotel social enterprise that I ran back in the early 2000s. It got us in the door to talk to people, but we then needed the substance behind that headline to actually have the conversation why are you creating this value? Who's it been created for? Can you tell us more about it? That report was done, interestingly, looking at the value to health service providers, employment service providers, and social inclusion funders. So it actually split the value into different funders for that organization to make the case for a blended funding model. So there's a lot more underneath the ratio. And the other bit I should have said about the process, there is one step I missed out, which is doing a sensitivity analysis. So once you have the ratio, you do a sensitivity analysis on the key aspects of the things that are driving the ratio. And so lots of SROIs are presented as a ratio within a range, but there's now a move just to present some SROIs as a range, because we're not going on, this is precisely $3.20 to the dollar. It could be, this is between $2.70 and $4.50, depending on the judgments and interpretations you make. And so there's the explanation of if more work was done here or if we understood this a bit better, or because it was a small sample size, we made this judgment. But if it was a bigger sample size, we found this, so yeah, there's more SROIs bringing that range aspect into it. Oh yeah, because it's such a lot of effort to go to for a single number. And I know that's not the case. So to get to the number, you have this extensive, like we do with all the valuations. So it's really fascinating. We have some other questions, and I think these are excellent actually. When would you not use an SROI from Verena? Why? Look, I think don't use it if you don't have a lot of money, is an obvious one. SROIs are not cheap. Some of the other social value model, like social value modelling is cheaper and more accessible. So it really depends what you want to do with it. I think SROI is good in advocacy. It gives you a good hook in terms of advocacy. And it really depends, you know, if you think about an audience, and so I used to work for Social Ventures Australia, and SVA back when I was working with them, had a lot of their funding came from high-net-worth individuals. And they were new into the space of funding. And SROI really appealed to them, because it spoke their language. They came from a return on investment background, and SROI was like, right, we understand the number. And so if you're looking for funding in that space, if you're looking for advocacy around that space, it's a very good model to use and a very good approach. With government, it can be a bit hit and miss. There are some government departments that respond well to it, but there are some government departments that don't like it because of the, because it's not cost-benefit analysis, and it's not cost-benefit analysis because it's stakeholder-driven, and it gives a broader specter for value than cost-benefit analysis. And, you know, depending on the decision you want government to make, you might be better going down a CBA route and or there are things in the middle. So we are doing a bit of work at the moment with a large not-for-profit with an economics consultancy looking at how we can shift the dial around making the case for different type of funding that's not pure SROI and it's not pure cost-benefit analysis. It's sort of halfway house in there. So it really depends on audience and purpose. But I think if you're sitting thinking, I've got a small program, I've got 20, 30,000 through an evaluation, then I wouldn't be, I wouldn't be jumping on the SROI bandwagon at this point. I don't think it would benefit you. I think you'd end up, you'd get as much from just theory of change. What changed for people? What can I do differently type of questions without necessity having to put a value on it? Really pragmatic advice, Kevin, which is probably what a lot of people want to hear. We have what resources can be used to identify proxies for SROI? Yeah. Well, this is one of the bits that identifying proxies for SROI used to be the biggest headache in doing a social return and investment analysis 15 years ago. It used to take, used to take lots of time and lots of money. And part of the reason that SROIs have become more accessible and the whole social value in space has become more accessible is there's a lot more proxies that now exist that are established have been used and being used repeatedly. So they have a lot more acceptance of validity around them. So there are proxy data banks that sit in different organizations. So within Think Impact, I think we've got 70 to 80 SROI reports that we can go into if we want to find proxies. We've got access to the social value international database of reports, which I think is another 150 internationally, maybe 200 plus now. We don't tend to use them as much because they're not Australia specific. And then we've also done a bank of cost benefit work that allows us to access proxies. But if we don't, if we don't have that, then we sit and we do work with stakeholders around stated preference, reveal preference, actually analysis of costs from other reports. So I remember a couple of years ago, we were looking at work around homelessness and we just managed to find a whole set of cost benefit work that would be done on homelessness. So then that became the source for the proxies. If you get beyond the report into the data that sits behind the cost benefit analysis, which we did, then it became the source for the proxies. And then, as I said, it's about being clear where the proxies come from. Is it being used for the first time? So you just be transparent about those things to help with the reliability, validity judgments for others to make. So we said, interesting in the principle be responsive. If this is for the commissioning organization, do you need upfront agreement commitment? How does this work in practice? So if I am a commissioner of services and I get an evaluation, or I would do an SROI evaluation, and that SROI tells me that I've been funding early childhood development work and there's more value being created in the work that's happening with improving people's social connection and less value Caribbean happening around improving literacy in terms of people getting ready for school, then the be responsive would be, well, what can we be doing in the mental health social connection space to actually improve that value? And then what's happening in the literacy space that's not working? So the be responsive is, how do we need to redesign or rethink the service provision from a commissioning point of view to actually drive better outcomes now that we know that there's more relative value here than here. So I'm sort of busking my way through that one, but hopefully that makes a bit of sense. As I say, it's the really, what are you going to improve? What are you going to do differently as a result of what you've learned? And so for my commissioning point of view, I think how are you going to fund differently? We know, you know, et cetera. You've given us a huge amount of food for thought, Kevin, in terms of both the mechanisms and I mean, even some of the framing we bring to bear on evaluation actually and the centrality of stakeholder value. And I mean, I found that really powerful that people will reflect on different values for depending on where they're at in the process. I'm really fascinated by the potential for quantifying this because I do think it's really powerful, but take on your comments about the cost. I mean, the assessment of cost versus value for the purpose, which I love the sophistication of that because so often we sort of jump on the bandwagon of this is the way to do it and we can see this all around us really. But what's in our best interests and when and how and for what purpose? I'm fascinated with VATCHO and I'm just going to I'm really looking forward to looking at that report because I think that is hugely innovative and some of the most innovative which is coming from our Aboriginal colleagues at the moment and just it's inspirational a bit. You know, I think we did good modeling and good work with that but Koa did really great work around their impact yarning and bringing that into an SROI context. I do think it's an excellent piece of work to look at. Yeah, I mean just sort of a bit more I'm blowing really given cultural fit and the type of methods. So I'm really interested in exploring further. Sky from Koa has done a lot of thinking about this and we use her as a sort of First Nations advisor on a couple of the other bits of work we do particularly conversations around like actually, you know, because there's bits of work when we're working in Aboriginal communities where it's like when you can't value this, this is priceless. And so yeah, there's bits we've done with the organisation in Catherine where they were likely this is where value has been created but this is so priceless to us we don't want to proxy put on it and again just being steered around the cultural nature of having to think about that from the intent of your white person thought about it. So yeah, and that's what I think hold a lot of disparate bits of work that have been done into the virtual report in terms of the way Koa worked with us on that. Yeah, I mean it just breaks your ears up I think when if you can make it applicable in that context that's pretty cutting edge frankly. I just I can't wait to have a look. I am aware at seven o'clock. I know I've asked more than questions than anyone else and it always makes me concerned that I've dominated the agenda. If there is anyone who's got a burning question right now I'm sure another minute wouldn't hurt because I don't want to feel like I've taken over. Anyone know? I do think you've given us a great taste to Kevin and a real thirst to know a bit more. And I think that's a really successful presentation frankly. So thank you very much for making the time to both present to us Kevin with such care and engagement. And for all our attendees thank you for coming along and hopefully this is where your appetite to find out a little bit more and work out how this methodology could work for you or whether some of the associated frameworks that sit underneath it the thinking behind it I think have very broad applicability as well. I'll just do my blatant advert at the end. If you want to find out more we're running SROI training. There's also a link in the PowerPoint if anyone wants to do a bit more. But thank you Mandy and thank you everyone for sitting and listening to me. It's been a pleasure to do this for you. I want to ask can a lay person or an evaluating person become proficient in doing this by doing your training? No. I think if they do the training and then they do an SROI then they will be well on the way to proficiency. But it depends how proficient like I've done I've been involved in 50 and I'm still learning. So if you think you've got it then someone will come along with something that knocks it out of you. But do the training and do an SROI then you'll be well on the doing an actual report you'll be well on the way to a high degree of proficiency in doing it. I think that's the story of an evaluator's life. We have to be as generalists but have a huge amount of expertise across a whole heap of things. The training is two days. So it's two days and it goes through the methodology. So the principles and the methodology in depth and people it's very practical in terms of working through it. Working through how you would do it. Thank you very much. Thank you everyone who made the time tonight to be here. We all have intense work loads. We know that because COVID has ramped them all up. Lovely seeing you all and hoping you will see you at the next AES seminar as well. Thank you very much.