 So I'd like to welcome you all. We're about to have an interactive session with Sophia Erickson from the Department of Foreign Affairs and Trade and Dave Green from Clear Horizons, facilitated by myself Kim Vessely with Zoom support from Robert Grimshaw, as co-combiners of the Australian Evaluation Society's Queensland Regional Committee. I'd like to begin by acknowledging the traditional owners of the lands on which we meet today, the Yagura people and the Turbol people as the traditional custodians of Mi'anjin, the lands on which myself and Robert work, but also to the non-Indigenous representing the land on which Dave and Sophia work. And I pay my respects to elders past and present. I extend that respect to Aboriginal and Torres Strait Islander peoples here today. This session will explore the role of DFAT standards that they have played in efforts to improve the quality and use of monitoring, evaluation and learning in the Australian Development Program over the last decade. The session will explain how the standards came about, how they are used on a day-to-day basis and their strengths and limitations as a tool for evaluation capacity building. The Australian Government's new international development policy, performance and delivery framework, commits to continued improvements in monitoring, evaluation and learning practice, underpinned by the Department of Foreign Affairs and Trades, design and monitoring, evaluation and learning MEL standards. The standards set details and expectations for key products in the programme cycle, such as theories of change, MEL plans and systems and evaluation terms of reference, plans and reports. First, we welcome Sophia Ericsson, who is the Mel advisor in DFAT. Sophia has worked in international development for 30 years and for the last decade has focused on monitoring, evaluation and learning. She has worked for both Australia and Sweden's bilateral development agencies at headquarters and on postings including China and Papua New Guinea. Sophia was an integral part of the evaluation capacity building programme delivered to staff in Indonesia. This programme developed and piloted the design and MEL standards that are still used today. Most recently she was involved in the refresh of the standards and is now building up internal DFAT capacity to use them. Sophia is passionate about getting staff involved in the use of MEL. Next, we welcome Dave Green, who is a lead principal consultant at Clear Horizons. Dave has worked on Australia's aid programme in the Pacific and South-East Asia for over 15 years, across the public, not-for-profit and private sectors. And he has a deep understanding of DFAT approaches to aid, design and delivery. His experience spans investment design, monitoring and evaluation experience across diverse sectors including education, wash, health, governance, community-driven development, civil society strengthening, impact investing, gender-based violence and at the country portfolio level. He has developed strategies, MEL systems and performance reports as well as driven systematic efforts to institutionalise portfolio-wide MEL capacity. Dave is passionate about improving international development practice by providing credible and useful insights to policy makers and practitioners. Before we get started, just a note on housekeeping. You can post questions to the chat box. We will respond to intermittently at the end of each presentation. All other questions will be answered at the conclusion of the presentation during Q&A. Please stay muted as necessary unless you're asking a question. Please keep cameras off if not asking a question. And lastly, please advise us who you would like to respond to your question. This session will be recorded and uploaded to the AES YouTube channel in the coming weeks. Today's session will start with Sophia before moving to Dave. Over to you, Sophia. Thank you, Kim. Thank you, everyone, for participating in this session. Both Dave and I are really excited to talk about this subject as you might have gathered from the introduction. And first, I also would like to acknowledge the Ngunnawal people, Lusland, Dave and I are meeting you from and pay my respect to the elders past and present, as well as acknowledging any First Nations peoples in this presentation. So, Robert, if you could have the next slide, please. So I just wanted to set the scene a little bit because if you are involved in the development sphere, you will be aware that we now have a new international development policy launched a couple of weeks ago. And that policy makes a number of commitments around increasing monitoring and evaluation and learning across the development program. And it's very exciting because, of course, we have been doing this for a long time, but this policy is, I think, increasing the ambition on MEL. And I have a favorite quote already, effective monitoring evaluation and learning are critical to achieving results and ensuring our programs continue to innovate, improve and reflect best practice. So for us, it's very exciting to see words like that in the policy. Next slide, please. And these commitments, I think, are there because not just because we need more MEL in international development, but it is also in the context of the government wanting to increase evaluation skills and use across the Australian public service. So in this group, you will be aware of the 10 million over four years commitment to set up an Australian Center for Evaluation in Treasury. And this is also reflected in DEFED, of course. So I think we are very fortunate in the international development sphere because we have worked on this and focused, even though we can always get better. But there's also more interest across other areas of DEFED, like foreign policy, trade, and passports. Passports have been doing monitoring and evaluation for a long time. You might have seen during COVID when they were a bit hammered and in Senate estimates, they would have talked about how they monitored and dealt with those issues. We also have a lot of internal, there was an ODE Office of Development Effectiveness Evaluation published in 2018 that looked at contractors monitoring and evaluation systems and the findings there were that we can become a lot better. Next slide, please. So also just to set the scene a little bit is that when we look at performance reporting, it's different levels from the international development sphere. So the policy has a performance assessment framework. It's available online on DEFED's website, which has determined, I think it's 32 indicators that investments are reporting towards. We're also developing, it's starting now to develop, let me get the name right, Development Partnership Plans, which is sort of like a country strategy, what we want to achieve in each country and also regional programs. These will also have their specific performance frameworks. We do investment progress reporting, which is annual self-assessments, and all of these reportings rely on good mail systems at the investment level. I would say 90% of the information to feed into these overarching systems would be coming from individual mail systems. And I should also say that in DEFED, we as a DEFED investment manager, the role of us in terms of mail is to buy quality assure and use mail. It's contractors that deliver both mail systems and reporting. And also when we do independent evaluations, it's done by contracted in evaluation specialists. So I'm going to hand over to one of them, which is Dave, to talk a little bit about what happens at the investment level. Over to you, Dave. Thanks, Sophia. And hi, everyone. Yeah, it's just a quick note, I guess, for those who haven't worked in international development. And I'm still really getting my head around the difference between how this works domestically versus internationally for Australia and for Australian evaluators. But the bottom line is that most of the, as Sophia said, most of the mail expertise and the mail resources sit at the investment level. So at that country program level, it's really mostly just monitoring a set of performance indicators. It doesn't usually go much further beyond that. But at the investment level, as shown in the standards, there's a fairly expectation of a fairly comprehensive approach in cooperating, monitoring, evaluation and learning. And the resources and the expertise to do those things usually sit within a DFAT implementing partner. So that could be a contractor or it could be a multilateral partner or an NGO. And that team of mail people may sometimes be commissioning evaluation work, or they might be doing that themselves. But they'll also have a pretty heavy load of monitoring and learning and reporting work to do. And then, as Sophia said, there are independent evaluations that are commissioned by DFAT directly. But I think compared to domestically, I would say that those evaluations are generally shorter and have smaller budgets than you would see in some sectors domestically. So hopefully, I just give you a bit of a picture for those who are new to international development of how the space is kind of resourced. Back to you, Sophia. Thank you. And I also just wanted to point out that, as you see the last bullet point, because one of the things also in the new development policy is an increased focus on accountability and transparency. And of course, that also is good news for us that are interested in mail because the information stems from all our mail systems. So it's all exciting news. Next slide, please. So what we wanted to do also is to give you a little bit of history, because when we talk about building capabilities inside organizations, so this work, I think, started 15 years plus ago. And it was sort of within an Indonesia program. So it wasn't sort of at the corporate level of DFAT. And we wanted to see how we could build up the capability of the staff within that area. Because at that time, this was, I think, the largest program that Australia funded overseas. And what was needed to do this was definitely to have mandate from senior management that they were really interested in performance and performance data and gave space for people to build their capability. And then what is also needed is the right expertise. Because I mentioned earlier that all the mail expertise really sits outside of DFAT. So our role within DFAT is to identify by quality, sure, and use it. And in our case, we were really lucky because we had Dr. Sue Dawson, who had already worked with Osaid and DFAT and worked on evaluations and supporting mail systems and really investigated what was not working well and how that could be improved. I was extremely lucky because I came back from a posting and slotted into this severe, because it is really important to have someone inside the organization that can network and see what's going on. Use the political, economic sort of who is your with you and who needs to be nudged to work with you. And also to checking in on the mandate, so it's still there. And then the final thing that we definitely discovered is this systematic approach that needs to be there because a lot of organizations that I've been involved in, it's you do a lot of training on M&E, but you don't really see any changes in terms of capability. So in this case, when we talked about a systematic approach, it was in the early days, we didn't obviously have the standard. So it was working through templates and supporting people having help desks. It was coaching and training on the job. So more looking at what people actually needed to be trained in relationship to the job rather than two years before they were going to do an evaluation, for example. And in this work, it's over time, Sue Dawson looked at internationally, looked at standards. The Canadians were ahead on this and came up and wrote the standards that we have built on since then. And it was really appreciated by staff and also even though we didn't focus on the implementing partners, male people that worked within those organizations also really liked having the standards because it sort of will talk more about it later, but it sort of made it easier to have a framework that people could agree around. I think that is what needs to be said about history, but I could ask Dave because he was actually part of this history before he ventured out of DFAT. Is it anything you want to add here, Dave, that I've forgotten? I used to work for AusAIDS at the time and I was on the Timor-Leste program and so Timor along with some other countries, I think Vanuatu and maybe one or two others replicated or adapted this idea of an evaluation capacity building program that Indonesia was implementing to other country programs. So it was a real sort of diffusion of this idea across various at the program level within AusAIDS and it was multi-faceted. So in addition to what Sophia was saying, there was also usually an effort to build in-house capability to deliver some of the training, for example, that SUD Awesome was delivering at the time. So it was a pretty comprehensive approach. It was picked up to some degree by different country programs, but then organizationally AusAIDS adopted the standards, but not the ECB program. Obviously the standards were a critical component. Thanks Dave. That is a very good point. So next slide please. So the design and mail standards that we are now using, they were refreshed. So not new standards and then published in December 2022, which was quite timely because it was 10 years after they were first published. And I had actually been away for about seven years overseas. So coming back, so I left when AusAIDS existed and came back to DFAT. So I was extremely pleased to see that the standards were still there even though ECB was not implemented as it had been. With these standards that we have now, we have added some standards around concepts, which is a tool that DFAT is using to make a case for some programs that we want to implement in countries. And then that concept is made into a design. And we also have a standard on program logic. And in DFAT, we sort of talk about program logic, slash theory of change. We do not differentiate between these terms like others might. So the good thing for us to have a standard is because of, as all of you know, the terminology in mail can become quite difficult to, especially when you work overseas also. So at least we are trying to have a terminology that is consistent within DFAT and with our implementing partners that are contractors. But it's still a work in progress. We really emphasize that this is not a checklist. And I will come back to that a bit later in this presentation. And we also try to focus on that mail systems are user focused. Everything we talk about today, Dave and I are working progress because it requires a lot of behavior change on all parts. But it's no point creating mail systems that are too complex and complicated and then not very user friendly. So people don't use the information for decision making. When we refresh the standards, we had what we call a brains trust. So we invited some of our mail experts that work regularly with DFAT. I was lucky because I had just returned. So I could sort of provide, which is quite unusual, but I could pry the corporate history here. And we also had some of our local staff that had been involved in the previous years to participate. And that's then what became these refreshed standards. Next slide, please. So this is what it looks like. You can find these standards on the DFAT Internet. So the highlighted in yellow are the new standards. And then from five to 10 are refreshed from the previous versions. And the good thing for us now is that the standards actually cover the whole program cycle. And it has been also, it's a bit clearer in terms of mail, where what the expectations are at different stages. So that there are certain expectations for mail at the sign stage. And then at six months, we have, we expect a plan, which is basically a paper product. And then at 12 months, that plan has been operationalized, which we then call that it's become a system. And as you see also standard eight to 10 is focused on evaluations. So number eight, which is about terms of reference. That's very much an internal that DFAT needs to be in charge of writing the terms of reference. We know what questions we want to ask. And then nine and 10 is to be able to assure quality and discuss with our evaluators what the expectations are. Next slide, please. I just wanted to go through a little bit on. So as Dave said, we do not have right now, it might come on board again now when the new policy has a bigger focus again on mail. But we do not have a formal evaluation capacity building program. But what I'm trying to do in my humble little way is to try to replicate a little bit of what we did over 15 years ago. So what I emphasize to staff within DFAT is that the standards have been in place for a long time. So it's not something new. I know that our contractors have used them and mail experts have used them over all these years. Really emphasizing that it's not a checklist because this is a tool that we really want to encourage dialogue and discussions around the standards. It's also not something someone else does for you because what is could very easily and has happened is that a staff member gets a mail plan. They send it off to another expert to check the quality and then they get back the quality assurance. Then they send it off to the contractor and the contractor responds and then the mail plan doesn't really become more than a paper product. So it's really important. One of my sort of indicators is when people engage with me to talk about the plans rather than having the absolute perfect plan. I'm happiest when I get people calling me up. And the other thing that I do talk to people about is that to get good mail systems actually working and producing credible information that we make decisions on requires behavior change. It requires behavior change within DFAT with all our investment managers to really engage with the mail system and use the information and discuss it with partners. But it also will require change on our implementing partner side because if we haven't demanded in a consistent way what we want then it's very difficult for contractors to respond and not to be mean but sometimes they also are in old habits and respond in a way that they've always done which has not been the best mail systems in that we would have like to see. It is included in all contracts. So these are implementing partners that are contractors. They have in the contracts that they have to apply the standards and what is going to be a little bit new for people is that with the new policy in the performance framework we have something that we call tier three indicators which are indicators that assess ourselves and how we work. And we are going to have one of those which is that the mail plan is assessed against the standards and I'll go through a little bit what that might entail from my end. And we have said that we want to see mail plans at around six months and this is also to encourage that this doesn't slip away because in my all the years that I've worked on with in both Osaid and DFAT it's very easy that the mail plan does not get done and then it starts slipping and then you're halfway through the program before it's an agreed system that is actually in place. So this is also something that is in that behavior change box that we want to see this move in a more reasonable way than it has been up until now. And finally what I also really emphasize is that this having the standards and it's not a checklist it's really important to have a dialogue and create a partnership because without the implementers we have no information whatsoever since it's all done by our partners. So it's really really important to have a very good dialogue and create that trust with each other also because if you have that trust you can tell people when it's not going so well and actually learn from that. So I'll go through a little bit next slide and apologies because this is very small text but it's just to give you a little bit of an idea what an assessment of a mail plan that I do can look like. So here you sort of have the upper part of the standard standard five of the plan next slide please and then I've created this little tool for the plan so basically just added a box underneath each element which includes if a standard has been met partially met mostly met not met or not clear. And this might sound like a lot but what I discovered when I started pilot the standard is that to be able to create a dialogue you actually have to differentiate a little bit on what you are assessing because if you go hardcore and say that nothing is met which could happen then of course it will be very difficult to start a relationship and partnership so from my horizon it's better to have possibilities to differentiate a little bit broader on when things are mostly met or partially met for example. They're not clear is also very good for staff that are starting out on this because these standards should be possible to use by non-technical people and of course some of it is more technical than other parts of it but if you use not clear then you can have a dialogue and your implementing part can then explain to you what is happening and make it clearer and I also have many instances when I have to put not clear so it's a good thing to have there. Next slide please. So again apologies it's just to give you a little bit of a sense so this is sort of what it can look like when the assessment is done so hopefully you can see a little bit that you have partially met and met and then also the important thing here is that you have to articulate how it can be strengthened or improved because this is something in my experience in DFAT is that it's very easy to say I'm not happy but then not that easy to say how are the partner going to do better and so that we we are satisfied both of us so that's a little bit what it can look like and then next slide please and this is an ideal situation which as an advisor you rarely get to because you send off your advice and often don't hear back again but I use this this particular investment as a pilot so I talk to all the the partners that I was wanting to use it as a one of the first pilots and in this case so this is ideally what I would like to see for all future assessments is that the partner came back and said how they responded to the assessment and the reason why this is good is that there might be things here that cannot be responded to at the time the plan is submitted for good reasons and then you can agree between us and the implementing partner when that is going to be done and maybe in some cases hopefully unusually but maybe in some cases it can never be done but then that is agreed and documented this one of the problems we have when we come up to the end of the program and look at what's happened it's not been documented when things have been agreed to not meet standards for example so an ideal world you get a response and you can document and agree that it will be updated in the next annual plan because mail plans should be updated annually context changes program logics need to be adapted so this is a live document that investment managers should engage with so now I'm going to hand over to Dave to talk about strengths and opportunities in terms of using the plans yeah so if we just go to the next slide we've just put together some ideas here some sort of reflections on you know what what appears to be working well with the standards and areas where you know there are limitations or or risks this is really just a discussion starter that you know there's quite a few people in the call I think who have very practical experience with how these standards are being used so really interested to hear you know whether this resonates or what other views people have the obvious you know main strength is that they provide you know a level of shared understanding shared terminology which is a useful I think in international development where you can get a fair level of diversity of practice and you know as we all know evaluation is a very broad discipline too so there's a combination of those two sources of diversity but you know on a practical level when I or other clear horizon staff are working as the part of a mel team within and within an investment within an implementing partner you know often a new program are we starting bringing together a team that hasn't worked together before or coming from from different backgrounds and it can be useful to have the standards as a bit of a starting point to to build a bit of that shared understanding and consistency I think also at the sort of policy level you know like there's an interesting discussion going on now around impact evaluation and the role of RCTs in the mix of evaluation methods in the Australian government I think you know within DFAT it's really helpful for that dialogue to be situated within a the sort of the foundation really that these standards can provide because the standards are relatively agnostic around methods and just emphasize fitness for purpose so it's they provide a bit of a solid grounding I think that you can that can support those debates yeah they obviously have a key role to play in helping DFAT staff to get across their role and perform their role but that really you know that kind of strength is much more likely to manifest if these other sources of support to DFAT staff I think are in place that we talked about earlier like training or coaching or you know support from someone like Sophia internally to help them work through the process and you know just from the perspective of a an evaluator or a male practitioner it's the the standards empower you really to to build buy-in within say an implementing team or if you're part of an evaluation team to to get everyone on the same page about you know the importance of methodology in the evaluation process so it's sort of like a way to empower male advisors or evaluators as part of broader teams do you want to add anything on that slide Sophia or you're muted sorry a classic problem just to add that when I work with staff and contractors everyone really likes to have that framework that it's sort of it's not the opinions of people and especially because there is quite a staff churn on the DFAT side that you have some a framework that people can discuss from which seem to be very much liked by both sides over okay let's go to the next slide so this is yeah the flip side limitations and risks and again you know just questions here to get a conversation happening it's obviously quite a dense document so it can be overwhelming which can lead people unlike us who engage with that detail to disengage or for male to be seen as too heavy particularly if they are treated as a checklist and you know there's an interesting point here to be made about how accessible the standards are for staff for whom English is if English isn't your second language then this can be quite a dead dense and difficult document to get across it's actually interesting that this like say over the last five or ten years even there's been a lot of streamlining of of aid management processes within DFAT and yet these standards have survived that and in some ways have become more detailed which I think probably points to you know their value particularly to to implementing partners but do you have a perspective on that Sophia so you mean because of capability within yeah it's just interesting that a lot of other guidelines and processes have been simplified and streamlined over the last few years but the standards have have withstood that it's just an interesting reflection I guess well I think when we refresh the standards and then because of COVID it took a bit longer before they were actually published between having had that Brainstrust and people within DFAT including some of my bosses said oh there's so many standards so many numbers but if you make it to if you streamline it and and remove it I think sometimes it becomes more difficult actually because then you have to do a lot of elements within each yeah center so yeah sometimes a longer is in a weird way easier maybe it's maybe that's that you've been able to argue for that specificity yeah um yeah so the second question is just whether they are to or that they can encourage I guess compliance oriented approach and this is all comes down to how they're used but as I think Sophia flagged earlier there is a risk that the focus becomes too much on the document and not enough on how the document is used there's also a point here you know it's Sophia's mentioned a few times that it they shouldn't be seen as a checklist but I think that's still a work in progress and so you know you do come across situations where the application of the standards is a bit context blind in that there's an expectation that they should be treated like a checklist and I think that you know the taking a more tailored approach requires a certain level of confidence or experience with using the standards so that's you know it's probably not surprising yeah what about performance culture most of the standards are focused on the technical quality of mel processes there are some important exceptions but you know sometimes they get lost in the the focus on on the more technical aspects of mel and yeah it's probably also fair to say that they are heavily focused on the responsibilities of implementing partners but also working level DFAT staff and there's not much in there about the role of DFAT senior managers and yet we know that that's obviously a critical underpinning to whether mel actually you know has much impact these sort of themes were picked up quite well in the ODE evaluation of investment level M&E which Sophia mentioned earlier and that's available online and is worth a read the other critique that you hear sometimes is that they the standards kind of take the focus away from the main game which is strengthening mel systems within partner countries counterpart mel systems and the standards are actually there are a couple of standards in there that mention or that require the use of counterpart mel systems or alignment with counterpart mel systems but but yeah it's probably I think this this critique is still still fairly reasonable anything you want to add Sophia yeah I think the it is interesting with the strengthening counterpart mel systems because that is if you go back and look at the the standard for design you have to really look at or partner systems and see how they can be used and I think there are many sort of problems one of them is that because we are using Australian taxpayers money so we are accountable to Australians but we are working in a sovereign country which then there might be an example was when I was working on a program where we it was a multilateral organization that was responsible to help the government to build their systems but they were not very good at it and meanwhile we had to sit and sort of wait then and then you're accountable to your government and your people so it can become quite tense in terms of getting that data the other issue is that one of the things that comes up a lot is that oh we're going to have a mel system and then yeah let's just build a bit of capacity for the government partners as an afterthought so one of the things that I'm trying to emphasize is that if that is going to be part of our design it has to be resourced and also monitored so that it happens because it sort of comes like we're going to build capacity as we run along here kind of language and and that's also a risk I think yeah over okay do you want to take us through the next slide Sophia on how to get the most out of the standards thank you so I think one of the important things is that the standards and I am completely biased but the standards are good and it's a really useful tool but it will also require continuous training and coaching networking and sort of managing knowledge so that people investment managers know about them and and can use them as Dave said without treating it as a checklist and that that's obviously building that capability is a long-term endeavor and it is what I find is that in everything the more you know the more flexibility you dare to have so that's looking at the context and what's actually needed it sort of comes back to when you see you know investments with a massive amount of indicators because maybe something will stick rather than banking on three that are actually the three right ones so it's having that confidence and really the emphasizing again the dialogue and the partnership because four d-fat investment managers do not have to be M&E experts so it's a bit onerous on the implementing partners to make sure that the d-fat person actually can understand the mail system and and what's going on so that's important and then the the culture the performance culture is very important and when I do training now within d-fat we always have a discussion about the performance culture within d-fat and it varies a lot between different areas and it's no surprise to anyone that it comes down a lot to leadership and what they are how they allow performance culture and especially to bring up things that are not working well so we can really learn from that and the first time Penny Wong came to visit d-fat the thing obviously that stuck with me was that she said I'm very happy to learn from mistakes I don't know if if that's true but that's what she said so that was that's that's a good good scene for that the the other thing just to throw in there Sophia on on tailoring to context is the you know the need for culturally responsive or culturally competent approaches to mail and that's uh I guess that's that's a net let's I don't think the standards are incompatible with that but but they don't you know they probably could do more to to promote that and you know maybe that's something to think about going forward yeah and and I think that that comes back to even the debate that we're having right now with methods for impact evaluations because it's also how comfortable people are with methods they are not familiar with or they because from d-fat horizon if it's rigorous methods that gets credible information it's all good but it's sort of having the confidence and an understanding even by mail experts if if they can promote that and implement it so that is I agree that is definitely something that we need to think more about and I think also the other thing is you might have noticed if if you paid attention that we are now talking about mail in d-fat rather than M&E and if you're an M&E person like myself the L the learning is always front and center but I think words matter so it's really good that we are adding the L now so that it is really a lot more focus on on learning and sharing the learning from evaluations across d-fat and beyond so that I think is also really important I think next slide so where to from here so we we are very aware because you could say that Dave and I are working on on two sides here with the same tools and it's not a quick fix as I said I've been sort of involved with this for over 15 years and the really good and encouraging thing is that I can see improvement it has become a lot better than 15 years ago but of course it's still a long way to go it's sort of like two steps forward one back but that means that we are moving forward and the other thing is for me within d-fat as an organization is to really get staff to engage with mail and not see it as something someone else is doing on the side and sometimes when you talk to mail experts they also have that challenge within the contractor because in the standard now we are emphasizing that mail is the responsibility of senior management so to engage people to actually have good user-friendly mail systems is a key and then it is a challenge with capability which I'm defining defining as having the knowledge to do things and then capacity having the time to actually do it or the resources to do it and both those are I think within d-fat depending on the areas they are both weak but I think also that sometimes we think in d-fat that there is an abundance of capability and capacity outside of d-fat but that's not actually true we need more mail people in this world so I'm very happy to see all of you here on the screen and spread the word