 So welcome everyone to a free event that's put on by the AES committee, the New South Wales Committee. My name is Laura Baker and I'm a principal at Aesle Island Consulting and also a member of the New South Wales Committee. First, before we get any further into today's session, I'd like to start by acknowledging the Gadigal people of the Eora Nation. That's where I'm based today, but also to acknowledge the traditional lands on which you meet with us today as well. So I wanted to pay my respects to traditional custodians of the lands and to acknowledge their deep connections to this country. As I said earlier, and for those that are still joining us today, please acknowledge where you're coming from when you're joining us today in the chat. So the AES, just a little bit about the society, it aims to strengthen and promote evaluation practice and theory so that the use of evaluation makes a difference more broadly in our society. I'd like to encourage all non-members here today to join the society and always happy to have a discussion about what that looks like at another point in time. At the end of today's session, we'll have a quick feedback survey where you have the opportunity to give us some feedback on how this session went and to suggest some topics for the future as well. And we always like to take input from the community and hear what you really want to know so that we can inform our sessions. Based on that past feedback, we're trying to make these events as interactive as possible, and that's really helpful from what we can tell in promoting shared learning and then also networking, so giving you the chance to meet up with other members of the community. So to support this, we encourage you to turn on your video and engage in the discussion. There'll be an opportunity after the presentation to have both a Q&A session with our presenters, but then also to join breakout rooms and have that discussion in smaller groups as well. Today we have the pleasure of being joined by Cherie Perth and Victoria Baird from Mission Australia. So Cherie is Impact Measurement and Evaluation Manager and Victoria is an Impact Measurement and Evaluation Specialist and they're going to be speaking with us about Mission Australia's multidisciplinary approach to monitoring evaluation and learning. So Cherie presented this talk at a standing room only room at the AES Conference recently in Brisbane and they've kindly joined us today to present this talk again and to share their passion for Mel more widely. So I'll hand over to Cherie and Victoria to walk us through this. Thanks Laura for the introduction. Hi everyone, thank you so much for coming along. We're really I guess pleased to be here with you today to share with you some of the journey that we've been on at Mission Australia to establish an integrated organisational and multidisciplinary approach to monitoring evaluation and learning which I will call Mel living forward for the webinar because it's much easier to say for everyone. Before I get started though if we go to the next slide Vic, I just really like Laura, I want to acknowledge the traditional owners and custodians of the land on which all the lands that we're meeting on today. Vic and I are dialing in from a Gadigal and Gringai land which is in the southeast coast of Australia which is always was and always will be Aboriginal land. I'd like to pay my respects to elders past, present and future and also extend that acknowledgement and respect to all First Nations people who are dialing into the webinar. So in terms of what we're talking about today, as Laura mentioned this is going to be a bit of a free run of the presentation that I did at the AES a few weeks ago. The big difference though is we've got lovely Vic who is joining us in my team and Vic has really been integral in the development of this work and particularly in establishing the homelessness monitoring, evaluation and learning cycle. So really awesome to have Vic here, thanks Vic. She's been on this document for a few months so I've worked doing to come here long and just sharing her insights and her experiences with us as well. So essentially what I'm going to do is I'll start out by setting the scene. So talking to you a little bit about Mission Australia and also the homelessness and housing context in which we're working. I'll then give you an overview of our Centre for Evidence and Insights but the main component of the presentation is really going to be us providing you with some practical examples of how we are working with a range of different people with different experiences across different disciplines to evaluate, sorry to monitor evaluate and learn all about our homelessness services. And at the end as Laura said we want this to be interactive so please if you've got questions on the way pop them in the chat but at the end there'll also be an opportunity for you to ask us anything that you want to know. So I'll get started I guess with I guess setting the scene. So for those of you who may not know Mission Australia is a large national diverse organisation. We have the goal of ending homelessness and ensuring that people and communities in need can thrive. We have over 2,000 employees and offer over 400 different services across the country which includes 78 homelessness services. What we're seeing in our services and we're seeing it every day unfortunately is that there's a growing need and growing demand particularly in the homelessness and housing services or the programs that we're delivering. And there are a few things that are happening in the communities that we're seeing and that we're working with. First there is an absolute stark shortage of social and affordable housing. We've got the skyrocketing cost of living which we're all experiencing but also I guess our most vulnerable people in communities. But we've also got really inadequate levels of income support from government. So all these factors particularly over the last two years have really combined to escalate what was a homelessness and housing situation into a national housing emergency. We can tell through the ABS data that we now have over 100,000 people who are experiencing homelessness on any given night and even more people who are at risk of homelessness if they don't receive support from community services such as ours. We can also see in our own data we recently released a homelessness impact report that there has been over the last three years a 26% increase in demand for our services and a 50% increase of people who are seeking support after they have actually become homelessness, homeless. So really really stark and alarming figures that we are facing in this country. So what do those stats tell us? From our perspective what they tell us is that it's absolutely critical for government and for non-government organisations to use data, to use evidence-based practice and also innovation to find out what's working but also what's not working and what can we be doing to end this emergency that we're facing. But we can't do it alone and that's what this presentation is all about that we're going to talk to you about. We can't do it alone as evaluators. We can't do it alone as non-government organisations. What we really need to be doing is working together across different disciplines, across different organisations but also across different systems and really using the power of mail to drive positive social change. So to address this context if we go on to the next one, in 2021 Vision Australia formed the Centre for Evidence and Insights. The vision of the centre is to inspire curiosity for evidence that leads to learning and action and you can see there's a picture of us on the left-hand side of the slide there. We were having a bit of a giggle vic and I yesterday because we actually were not there when this picture was taken. So what you can see at the bottom right hand corner is we've photoshopped ourselves in so it doesn't look suspicious at all that Vic's got a cat on her shoulder or that it's my team's picture but we are part of the Centre for Evidence and Insights and we are a big multi-disciplinary team. So a critical function of the centre over the last few years has been the establishment and the implementation of an integrated organisational and sustainable mail system and the aim of our system which you can see on the wheel on the right hand side is our four things we're trying to achieve. Firstly we want to be measuring our impact so we want to understand deeply if and how our services are contributing to outcomes for people, for families but also for communities. We want to be driving excellence and we want to be doing that really through our monitoring processes and we want to be finding out what's working really well so we can do more of it but also finding out what may not be working so well so we can address any barriers and any any challenges as quickly as possible. Thirdly we also want to be using mail to build a really strong evidence base so we want to be doing that through conducting rigorous evaluations to ensure that we're not only delivering services but also investing in services that we know will have the greatest impact. And finally most importantly for anyone that knows me we also want to be sparking action so we want to be learning we want to be sharing evidence we want to be sharing data within our own organisation but also more broadly with the sector. So there are kind of big picture aims to do that go to really establish that system what we've done is we've taken a multidisciplinary approach so the centre brings together different people with different experiences across different disciplines really focused on creating that enabling environment for mail activities. We have researchers I'm going to forget people but we have researchers we have evaluators we have data scientists we have business intelligence we also have a statistician now in our team we're all really working together with that common goal of looking at ways that we can gather insights and evidence to end homelessness. So that's I guess the the big picture and the scene setting now what we're going to do is I'll hand over to Vic and we're going to start talking about some of those practical ways that we are bringing together different people in terms of the implementation of our homelessness male cycle. Hand over to you Vic. Thanks Rhi. Yeah so as Rhi said I'm going to talk first about how we set up our monitoring system and how we worked with people across our organisation to do that. Obviously we want to have access to as much data as possible but everyone who actually works in the services is really busy they've got lots of competing priorities and data collection is never a top of their list so we want to make sure that whatever it is we're asking them to collect is actually useful and that we're focusing on the right things. So at Mission Australia we have this concept called flagship service models and they are developed by a team called Service Design and Innovation and so they have really formed the foundation that we build upon with our male framework so with our homelessness and stable housing support flagship it has a theory of change it articulates the core activities critical success factors and expected outcomes that we want to monitor across all of our homelessness services and that was developed by the service designers in collaboration with practitioners, managers, people with lived experience and representatives from the impact measurement and evaluation team. When it came to developing the monitoring framework we wanted to make sure that the outcomes that we were measuring really were the things that were important for the people accessing the services so even though we'd already had some client involvement in the development of the service model we also undertook some additional consultations with clients from different states representing different demographic groups and we actually paid them for their time to explore you know what are the things that are important for them in terms of the outcomes that they want to achieve when they get in touch with our homelessness services and it wasn't just a tokenistic we actually did really take on board their feedback and it led to some adjustments to our monitoring framework so one of the things that really came through in the interviews was the importance of both physical and mental health so that was something that we'd kind of touched on a little bit in the framework but it came up so so frequently and so strongly in the interviews that we knew that it was something that we needed to focus on a little bit more so we did incorporate that into the monitoring framework. Another important part of having the clients involved in the process is it really helps us to communicate with our practitioners because if the practitioners on the ground aren't collecting the data we have nothing to monitor we've got nothing to analyze and evaluate and generally speaking people who work in you know our sector they care about people and so if we can sort of show them how this monitoring framework actually links back to the people that they work with and the things that are important to them it really helps to get that sort of buy-in from the practitioners and helps to get them on board for how important it is and so building that culture with our practitioners has been a really long-term process at Mission Australia. We started I would say about seven years ago as we started to roll out our impact measurement program which is what we call our national approach to measuring client outcomes through surveys. We center around subjective well-being but then we add in other additional service specific measures into the surveys and through that impact measurement rollout we recruited around 200 impact measurement champions across the country so as Shuri mentioned Mission Australia has over 400 services across the country and although our impact measurement and evaluation team might seem quite big compared to some other organizations when you think about the fact that we have 400 services to support it's not enough for us to actually provide that one-on-one support at a service level so that's where our champions can come in they can be those representatives on the ground that's the first point of call for any practitioners that might have questions about their impact measurement or males and they really help to embed that measurement culture in practice and help practitioners to see how the data can actually be used in their day-to-day interactions with the people that they support. Another really key thing for us at Mission Australia is the way that we work with our IT teams and the business intelligence team so we have a data lake yesterday I was learning about data lake houses that's a whole new concept to me but we used to have a data warehouse now we have a data lake maybe we're going to have a data lake house but all of our survey data that feeds into the data lake all of the data from our client record management systems also goes into the lake and so because we have all in that centralized location we can then combine it together and report it in different ways so we work with the business intelligence team and they can create all these amazing dashboards that can feed into external fund reportals so we can sort of report directly to DSS data exchange we can report to AIHW the primary mental health care mental data set all these funders that we have so it really reduces the burden on program managers because they don't have to sort of manage all of that reporting themselves but they have validation reports where they can check what's going to be sent fix up any mistakes before it goes and then we also have internal reports that everybody can have access to and another great thing that we have been able to achieve with the help of our business intelligence team is creating our mega mail data sets so for example with our homelessness and stable housing support mail we have a data set that gathers together all of the information from our homelessness services over the last three years and it's just growing every day it gets updated in near real time so it currently has over 25,000 reports and around 500 different variables that we can explore so obviously that's a really strong foundation that gives us lots of different options for how we want to explore the data lots of different techniques we might want to apply and different approaches we could take so an example of that in practice is as Cherie mentioned our recent homelessness impact report and our wonderful statistician Sayali got to play with the data and what we could see when we just looked at the monitoring data at a high level was that our services were really effective at supporting people at risk of homelessness to maintain their housing so that was a really great win a great thing that we could celebrate but what we could also see was that if people came into case management when they were already homeless it was very difficult for us to transition them into long-term sustainable housing for all of the factors that Cherie kind of describes earlier it's just really hard to move them from homelessness to housing but what we wanted to understand was are there any factors that we can control that are more likely to lead to somebody actually exiting homelessness so with the mega data set Sayali was able to do a predictive regression tree analysis and was able to identify two strong predictors of a successful outcome for people experiencing homelessness and those were the length of time that people were engaged and the intensity of the support that they received so as you can see on the slide in general less than a third of the people who are homeless who access case management exited homelessness so it's 31 percent of people if we could keep them engaged for 4.2 months at least then the likelihood that they would exit homelessness increased to 49 so nearly half of them actually exited homelessness if they stayed engaged for that amount of time and if during that time they received at least 27 instances of support their likelihood of exiting homelessness increased again to 55 percent so that gave us some really practical useful insights that we could share with our services that they could then apply in practice you know how do we keep people engaged for longer how do we make sure that they're getting that intensive support because that's going to be more likely to lead to that positive outcome and we were able to do that with this routinely collected data without sort of undertaking a big evaluation or asking our staff to collect any additional information that they weren't already collecting it was all based on information that they just collect in their day-to-day work and we could share that not just with our own services but more broadly across the sector so Cherie is now going to talk a bit about some of the evaluation work that we're doing yeah so the next component of the cycle is all about evaluation so we have just I guess to manage expectations we have spent the last few years really setting up that strong and sustainable monitoring system what that's enabled us to do now is to really identify high priority and strategic evaluation projects that we want to be undertaking really building on that existing data set that we already have we have an annual evaluation work plan which we update in collaboration with community services every year we've got around I did count it last night I think we've got seven homelessness evaluations which are actually happening within this financial year and into next financial year but I just thought I would identify for you three of the projects that we've got going on and these are collaborations with external research centres and also the government in terms of us evaluating some of our services so the first evaluation that we are pursuing and we've been working on it for about 18 months now but it's finally got through ethics is a partnership with the life course centre up in Queensland and they're essentially going to be looking at our mega homelessness data set that Vic spoke about and they're going to be looking at which homelessness service models work for whom and in what context so the first phase which I hopefully will kick off shortly is really just going to be descriptive analysis but the long-term vision is that we're going to be able to do to do some more sophisticated analytics on it so for example doing some matching or looking at if we can isolate the impact of different service models for different people so we're really excited about that project to get it started the second collaboration is actually with the Australian Institute of Health and Welfare and we've got a few projects on the go with them at the moment and those projects are all about looking at what happens to people who receive a service from Mission Australia once they leave our service so the interactions with the border homelessness service system and for us that is really exciting because it's the the black box we have no idea or no visibility around what actually happens to people once they exit our services so being able to look across the service system will give us a really good picture of I guess that kind of longer-term impact and lastly which a lot of people are very interested in at their ES conference to have a chat to me afterwards is a collaboration that we're starting with the New South Wales Better Outcomes Lab this is actually going to be a hybrid evaluation which is looking at exploring our youth homelessness services so really having an early intervention lens it's going to have two components one will be an internal evaluation where we're looking at kind of the processes and implementation of our services as well as short-term outcomes but the partnership piece which is really really really exciting is all about doing some data linking so accessing data from the New South Wales human services data set and that's going to enable us to get an understanding of the longer-term impact of early intervention programs as well as hopefully Fingers Cross to create a comparison group so we can actually attempt to isolate the outcomes and longer-term impact of our homelessness services so that's kind of three examples of like the high priority evaluations that we're conducting really interesting these are actually the three I've highlighted are collaborations so they're not evaluation projects that we're actually commissioning we've been reaching out to a range of different partners with similar goals as us and really just having those conversations around how can we utilize the data that we have particularly around reaching our strategic goal of any homelessness. Big back to you for the last part. The most important part of all. So yeah I'm going to talk about learning the last part of our male cycle so as much as we might enjoy monitoring and evaluation and diving into that it's really kind of pointless unless we actually put things into practice so that's why we believe that learning is actually the most important part of all of this. I spoke a bit earlier about those BI dashboards that our team create they're not just used to create things like our mega data set or to report to external funders they also enable people across our organization to have real-time access to their data so we have service-level dashboards where managers and practitioners can have a look at their own data but we also have aggregated dashboards that show national and state-level data that are embedded on SharePoint and everybody across our whole organization has access to that so it means that like our policy and advocacy team they can look at that and use the data to inform their positions or business development as well and the program managers can also sort of compare their individual service-level data to maybe trends that they're seeing at a state or national level as well so that visibility is huge for us that's revealed to the game changer but then we also have a big emphasis on our annual evidence to action process that services are expected to complete so the staff are provided with a reflection tool that's been developed by that service design and the innovation team who we work with so closely and then they have access to the data and they can be supported to explore what it means so we run workshops where we support people to review their data that might be done at a service level or we could potentially get together a number of services maybe within the same geographic area or running some civil service services and explore what some of those insights are so we get subject matter experts involved we have some homelessness specialists or say people in our family preservation teams as well who are specialists so we get that sort of practice knowledge in people from practice quality as well and then we really just unpack the data and unpack those insights we celebrate the successes and think about how we could you know replicate that in other places and also get the contextual information from the practitioners and the managers because obviously they know their clients they know their communities so much better than we do and they can really provide that additional information that tells the story behind the data and really helps us to understand it and then most importantly of all once we've kind of gone through that process they come up with this evidence to action plan so they identify some priority areas that they want to focus on over the next 12 months it doesn't need to be anything super complicated it could just be sort of two or three things that they want to change or try to do differently and they articulate what they expect that to look like in the data we then run the data again in sort of 12 months time and go through the process with them to see if the changes they made have actually resulted in the outcomes that they intended so an example that we have of that is one of our youth homelessness services over in WA they could see through their impact measurement data that when young people were leaving the service they were quite unhappy with their future security and they were actually feeling quite anxious so we explored that with the team and they identified that they could be more consistent in the way they celebrated achievements with the young people throughout their engagement in the service really helped the young people reflect on their own growing capacity and their own ability to be independent so that by the time they were actually leaving the service they felt more equipped for the future to face the future without the support of the service and they also decided to put some more effort into improving their exit planning process again so that young people would feel more supported through that exit process and feel kind of ready to leave by the time it was time to exit so they made those changes we rerun the data a year later and we could actually see a measurable improvement in the scores that the young people were providing so they were saying that they felt happier about the future and they were less anxious so it was exactly the kind of change that we would want to see based on the data that we had originally presented so that's an example of our learning in action so now it's just our top five tips in summary we've talked a lot about all the different people that we engage through our mail process it really is a team effort we don't do this alone and it's really great to see that input from different people and the expertise that they bring another really key factor for us is the leadership buy-in that we have as Shuri mentioned our mail's linked back to our strategy we have really great support from our senior leaders there's KPIs for our strategy that are attached to the work that we do so it really helps us to get that that buy-in that then feeds through to staff on the ground technology is also super important and having that ecosystem and automation is essential like we said Mission Australia is a huge organization it has such a variety of services if we had to do this manually it would be impossible so IT are our best friends we talk with them all the time and also our BI team that help kind of bridge that gap between the IT technical people and you know people like me that don't know what data lake is so we really work collaboratively together to make the systems work for us and to make them work for the staff on the ground because if the systems don't work for the staff on the ground we're not going to have any data to analyze another really important thing to keep in mind is that this has been a phased long-term project from Mission Australia it's something that we've been working on for like I said probably seven or even eight years in terms of laying the foundation for impact measurement improving our systems like we've done a lot of work around our client record management system to improve the quality of the data that we get from that and we did take a phased approach to impact measurement and now a phased approach to metals as well we test things we pilot it we see what works we see what doesn't work and then we make improvements so we can't you know you can't just jump to the end this has been a long-term project and we're really just starting to see the fruit of those you know eight years of work now so that's really exciting and like I said as Cherie said learning is the key thing there's no point in doing this if we're not actually going to learn and change anything we we want to in homelessness we want to support as many people in the best way that we can and we really want to apply the insights and that's why we do what we do and not only does it result in those better outcomes for people it also kind of helps reinforce the cycle because the more staff can actually see the data can understand the insights see the impact that it has for the people that they work with we get more buy-in in terms of like the quality of the data collection the consistency of actually collecting feedback from from clients because they see that it's not just data that's going into a black box and nothing ever happens to it they can actually see the value for the people they work with which is usually their their main motivation so learning is kind of a key for everything I would say so those are all of our texts I think it's maybe over to Laura and any questions fantastic thank you Victoria thank you Cherie please post your questions in the chat we're happy to do it either either by chat or just raise your hand and we can work around the room and see who has questions for Victoria and Cherie to start us off I had a quick question around the data set that you're building and then other data sets that are available so you mentioned the Australian Institute of Health and Welfare and then also New South Wales customer service and health data as well and I'm curious to see how easy it's been to link in with those data sets gain access to them and what kind of value you're getting from that from that combination of data sets yeah good question Laura it's been a bit of a process I think what we have done not strategically probably opportunistically is pilot projects now that don't actually require data linkage does that make sense so the project with AIHW they actually already have our data because we report into them monthly all of our specialist homelessness services so for them it was just a conversation around you can see the services and data we would get great benefit from being able to understand I guess that trajectory from our services perspective so we were able to build that partnership and with the New South Wales Better Outcomes Lab to get access to the New South Wales human services data set that obviously needs to be focused on young people vulnerable young people and families so we were originally pursuing two projects or two service types one was our youth AOD residential service and the second one was our youth homelessness services we ended up going with youth homelessness services because again a DCJ which is a government department already has all of our data because they're specialist homelessness services so we don't necessarily need to go through that data linkage process but we are going through that process now where we're doing for another project where we're doing a privacy impact assessment so that's obviously taking a much longer time and we're going to be reviewing all of our like consent procedures and stuff as well so the long-term vision is to do the data linkage but at the moment we've chosen those projects where our funders or partners already have the data available. Fantastic thank you. Ila you've got your hand up. Yeah look thank you for a fantastic presentation it's really interesting exciting to see what's what's going on and I just wondered in the early days how did you go about actually getting Bayon from the organisation for the whole male strategy and work that you wanted to do. Vic do you want to start with your pet measurement and then I can jump in with you. I think the real key was getting our execs on board with it so we I guess not our current CEO but our previous CEO was really so passionate about it I mean our current CEO as well I think she's that but yeah the I think getting the execs on board was really important and then just testing it so there was you know a pilot in 2015-16 really listening to the staff and then the champions was really important so as we developed the kind of impact measurement process we really took them along on the journey with us and helped them to feel like they had some ownership of it so that when we sort of produced these surveys it wasn't you know random questions that they didn't think had any value it was questions that they'd actually had some sort of input into the development and so we also just had a big comms push so like as we did the pilot they you know recorded interviews with practitioners and managers in services that had been using impact measurement to then talk about you know how they had actually used it in practice because you know as much as we can tell them it's great I think they sort of trust their colleagues in service delivery a bit more than us and then also within our own impact measurement and evaluation team we have staff who come from a service delivery background as well so I myself used to work in service delivery in the Kimberley and WA so when people sort of talk about the challenges of working in sort of like remote contexts or with Aboriginal and Torres Strait Islander communities I can speak to that from my own experience and similarly other people in the team that come from that service delivery background can kind of help to bridge that gap as well so I think kind of again it's that like multi-pronged approach that helps and I guess I love from my perspective so I started the organization or in this role about two and a half years ago and I think I was pretty fortunate that that kind of foundational work had been done but those original conversations that I was having with leadership was around we've got all this data we actually need to now like look at what does this actually mean and what does it mean for our services so I would say I spent a good six to nine months just talking about what is Mel we ran demystifying sessions because people were like I don't know what you're talking about and I just really tried to strip it back to this is just about us making using and utilizing all of the existing data that we have in the organization so strategically we haven't introduced any new data sources because the organization the phase or the journey it's been on we've just had system after system after system the next few years is just around we've got all this data how can we actually use it and feeding it back into services and I think over the last 12 months the light bulb moment moment for people has been actually seeing the data so putting together data packs Vick and I worked on I think 68 different data packs for our homelessness services even though there were dashboards as well going out talking about the data having that become real for people to be like oh now we understand why we enter data that way and having those conversations around what it means has been really important thank you question coming in from the chat here from Julie Elliott just around the type of valuation approach you're using so she's mentioned that the evaluation component seems to lean to a realist approach if this is intentional can you explain how this works within an within a transdisciplinary team so for example it could be the evaluation questions you pose yeah great question we don't lean towards any I guess evaluation specific evaluation approach it would depend on the context of the projects that we're working on generally speaking though we lean more towards a utilisation focused evaluation approach where we are wanting no matter what through all our projects to feed data back into services as quickly as possible that's our key the life course centre one does take a realist approach in terms of looking at what works for whom in what context but that's a specific project that they will be running does that make sense so I would just the approaches that we take really depends on the context and how we're working and the questions you want to answer fantastic there's another one here from Kieran Galvin so given the externalities facing the homelessness services sector for example the cost of living housing availability and inadequate government payments how do you account for these factors and the barriers they may present to success in homelessness services yeah great question and we just acknowledge it so even with that predictive analysis that Vic was talking about we were really really mindful that there's a wide variety of different external factors that are contributing to whether our programs can get people housing or not so we specifically for that kind of predictive modelling looked at what we believed were modifiable service elements so elements that we could have some control over seen we really have very limited control over what's happening in the external sector so we looked at things like length of service delivery the types of support received the intensity of support things that we believed we could have some control over and ran that analysis and as you can see even with the predictive analysis we didn't get to 100% of people got a home I think we got up to 51% of people and that's really factoring in all of those external factors that are totally outside our control but what we have been doing though is using that predictive analysis for advocacy purposes yeah so going out talking to people about our homelessness services the homelessness impact report I think we reached around 3.2 million I would say people audiences so just using our data to start that advocacy piece around the fact that we need an increased investment in social affordable housing and we need to be addressing those external factors as well so using the data from a policy and advocacy lens. Fantastic and I got a couple of questions here I think just for a bit more of a practical take on how you socialise Mel across Mission Australia how you made the program team an ally in using this data and seeing impacts from it and then how did you get the senior leader leadership team on board with Mel. Yeah good question great questions I think I've been able to we've been able to piggyback of the great success of the impact measurement program the fact that the impact measurement program which is all around us collecting reporting on outcomes has been well established so we have I think it's 93% of our services collect outcomes data so that groundwork was done originally around you know we need this data for basically all of our funding contracts now but not only that we also are really committed to measuring wellbeing across the organization so we want everybody who comes into our organization to feel better about themselves and that really connects to people on the ground because they want that as well so what we're doing is essentially providing them with a way that they can be measuring the impact and having those conversations with people and with communities so as Vic mentioned earlier there's been years of work to get that comms piece out and to get people on board so when we started I guess talking about an integrated approach to impact measurement which also is now Mel impact measurement is one of our monitoring data pieces that we use it was really just continuing that conversation around we have so much data in the organization and we need to be looking holistically at our services so how they're being delivered but also the outcomes that they are contributing towards and then demonstrating so showing them the data telling that story we didn't show it today but we in our monitoring and reporting framework we have signs of success which is closely linked to our theories of change so every time we're presenting it we're talking about sorry signs of progress I should say we changed it to get buy-in and engagement signs of progress we're presenting in that way what we're producing looks the same it looks nice interactive just different strategies that we're getting to get people interested in the data Vic, do you have anything with that one? No I think just even like the data visibility is a big thing in terms of getting the program teams on board we got some dashboards built into our client record management systems as well so I think that really helped the practitioners to get on board because they could actually then you know see in real time the data for for their own specific client so just yeah I think that's another like when people can see it it makes them a lot more interested I think. Thank you while we're on data I'll just skip down to Mary's question are your service providers submitting data via a portal or a CRM also did they order Mission Australia on their behalf report data via a data exchange for the Department of Social Services funding purposes or for all of the funding received? Yeah great question so we have our own CRM that I think 80 percent of our services are using it's called MA Connect in that fancy diagram that Vic showed before so our services report data into MA Connect and then through our data lake we then transfer data into our funders so for example DSS or DCJ or you know a range of different funders so we don't encourage and we're getting better as we get new contracts coming on board we wouldn't directly enter into a government system we would be always looking at ways that we can push the data from our data lake so we have visibility over it. In terms of so this next question relates to the cultural perspective that's put on mail and whether there was any purposeful decisions that were taken around applying a cultural perspective whether in the evaluation questions the approach or how it's implemented. Yeah so this has been a challenge for us I would say that probably one of our biggest challenges is around particularly how do we measure outcomes for our First Nations communities it's something that we've been talking about for a long time. We do have one evaluation project at the moment which is our integrated accommodation and support program up in the Northern Territory otherwise known as Batten Road and part of that evaluation that we're undertaking is actually looking at how we can develop a culturally appropriate culturally safe way to collecting outcomes data so we're engaging a First Nations consultancy up there to do that work for us and to start us really thinking about a better way for us to collect outcomes data so we know the approach that we're taking at the moment doesn't work so we're really looking at how we can authentically work alongside our Aboriginal and Torres Strait Islander staff come up with some solutions or some ways around how we can be really measuring well-being so it's going to be now I've done that kind of foundational piece a really strong focus over the next six months is around how we can be doing that much much better but it has been one of our biggest challenges. And just to follow up from me on that one is that around what you're measuring and are the the impacts or the outcomes of value to those communities are they different or is it about the practice of actually collecting that data both so it's essentially at the moment we use the personal well-being index so we get really strong feedback that it's not culturally safe or culturally appropriate and that's around the constructs of well-being don't necessarily relate to our First Nations communities but it's also actually just the process of sitting down with a survey and collecting data in that way as well so I think we're pretty excited about being creative about it we've said to the consultant up there we just like we want to try something different so you know it's not going to be a standardized tool we get that we're okay with that but it's just really important for us that we are providing a culturally safe and appropriate way to have conversations about well-being with all of our clients. Fantastic thank you and then from Ben so in terms of the L in Mel are you suggesting that the key step that the key is building it in as a step the evidence to action process so I'm not sure if I've read that correctly Ben did you want to provide a little bit more there is that well yeah I mean I think many of us perhaps don't pay enough attention to the L in the Evaluation Society so I'm just wondering whether you know whether there's something more you want to say or whether you think the key is actually just to build it in so it is a step so you have to do it and so it happens. Yeah look Ben it's been one of the kind of key comms pieces that we've used so even that slogan it's all about the L in the Mel that's actually come from one of our executives he actually said that after a meeting that we had talking about why we're doing this and why it's important I think we spent quite a lot of time focused on compliance so this had to happen particularly in terms of impact measurement but because we're now shifting it a shift in the conversation it's now about the so what piece and the so what piece is all about us creating space and creating time for people to come together and look at the data so a lot of our evidence to action process is based on monitoring data doesn't necessarily have to be an evaluation it's the fact that we're sitting with them looking at impact measurement looking at the data from the client management record record management system but then we have embedded it into the evaluation process so we have an evaluation management procedure which is in place in the organization and that provides step by step I guess instructions or guidance around how we project manage evaluations and it's in them so there's a whole part around evidence to action so the expectation set before a program even starts an evaluation that we don't do this for the shiny evaluation report although that's really really helpful to have um we're actually the purpose of this is for us to learn as an organization so it's really continuously having those conversations and building that learning culture at different levels great thank you thanks um and then another here from Nigel McPaul what if your practitioner is now most useful in the internal dashboards you've developed Nigel did you want to add anything to that now I'm going to throw a shower of ice while you're asking questions great that's no that's always nice thank you um I think like what we found initially was people were really good at integrating impact measurement into the onboarding process and sort of having it as part of that initial assessment but then it would really kind of drop off when it came time to do you know follow-up assessments because we really wanted to them to do an impact measurement survey when they're doing case reviews um and it also wasn't as well integrated into the kind of exit process so with the dashboards the staff can see when was the you know most recent survey done what did it show it reminds them to kind of do it again and then also really kind of visually shows them what has changed since the last time they did it so rather than have to look at what was the survey that you know they did three months ago and what's the survey now and sort of manually compare it it's actually sort of in their face so I think it really helps them to reflect on what's changing with the client um and also just to keep it front of mind in terms of part of their practice it also makes it easier for the managers to kind of track um which clients have got surveys which clients don't you know when they need to do the follow-up so yeah I think that's been the game changer. Nigel I will say though what we probably didn't anticipate so much or maybe we did and we and we're now really focusing on it is around data literacy so particularly you know people on the ground we think oh we've done this beautiful interactive really fancy dashboard um people are going to use it and they do they definitely use the impact measurement one um but we do really need a piece now around data literacy just around using dashboards but also interpreting data and what does it mean what does it mean to practice so um we're now planning to do some quarterly like capacity building workshops and one of the topics will be around uh dashboards so we're going to have a focus on that over the next 12 months um because that is the key uh portal I guess we want people to be going to to access their data so we do need to build capacity at all different levels of the organization to feel comfortable with them using those Power BI reports. We face the same challenge thanks very much. Thank you and for understanding from from internet um so evaluations help us to identify gaps in our business processes do you have an example of how Mel helped identify gaps in your business? Yeah so a good example is the one that Vic spoke about before which I spoke about as well as around that predictive data analysis so that was really um based on our monitoring data set um it clearly showed us that we needed to be engaging people for longer so really simple stuff we need to be keeping them on engaged we need to be keeping them on our service um when we went out and kind of socialized that and spoke to services around what does that actually mean and one of the key things that we realized was we need to get better as a large organization around connecting out our the people who access our services to other Mission Australia services so it's particularly in rural and regional areas where we have other services like Community Mental Health or AOD looking at ways that we might not be able to get a person a house straight away but we can also work with that person to address some of those underlying risk factors for homelessness so that we might have known that on the ground but having that kind of data monitoring data front and center really allowed us to be having those conversations on the ground um and hopefully we started having those conversations around six months ago and we did the homelessness impact report hopefully what we're going to see is some changes over time around how we engage people um on the ground but that was a big I guess thing that we didn't know the other thing we didn't know was how important brokerage was so we didn't talk about that in the analysis but brokerage also came through as a really really important factor um for people who are experiencing homelessness so again something we can modify within our services but we could see that a lot of our homelessness services were not we're not using brokerage so that's I guess was a big gap in our business processes so working with people on the ground to really work out how they can embed brokerage within their support planning processes as well was a another good example in terms of those planning processes so somebody's got a question in the chat um Ali around mel and when you plan that is does that come in at the program design stage or during the implementation of the program yeah great question so um what we're piggybacked mel on in our organization is as what Vic mentioned earlier was our flagship service models um so a lot of the flagship service models um that's all about grouping similar services together which have common goals under one theory of change so for example we have very different homelessness service models we've got 78 of them but they're grouped under our homelessness and housing support flagship service model so in that context a lot of those services were actually already existing but it was about bringing them together identifying the theory of change and then introducing the concept of mel um and that we're using mel to find out what's working really well to find out what may not be working um and then new services that we get contracts for are on boarded onto that mel process so we've got a bit of a mixture happening because we've kind of come in with mel when we already have 400 services in the organization um so we're working backwards with them but when we're getting new contracts on board that's an onboarding process and um we start the conversation at the beginning then