 This is one of these internal evaluation issues, and a preoccupation of many people talking about evaluation. And I have a somewhat counterintuitive view on this. I don't think independence has near as much an impact on quality as most of the people in the evaluation argue. I think much of it was made of this. And I think a perfect example of the sort of confusion, or what I call the confusion, is represented in the recent piece Richard Moore, friend and colleague of mine, had in his piece on DFAT, where he updated, made some comments on where DFAT was five years after the change. In that note, I'll quote his recommendations. The head of ODE should be an external candidate on a non-renewable contract to maximize independence. You know, if you ask me what the head of OED should be, it should focus much more on quality of the person. You should get the best person available. And you should focus not so worried about whether you're going to have a tenured person or a non-tenured person. If the person's good, they should be kept. If the person's not so good, you should probably move on and find another person. So Richard and I reflected a very important theme of evaluation, which I think is simply wrong. And indeed, I think the message for DFAT, any evaluation group for an evaluation, should be to find the best person available. And more important, I think, a message of not taking people from the organization is a terrible message about the quality of people you want to do evaluation. I mean, you want to grow people. And the ODE has been pretty effective at bringing new people in and growing people on evaluation. And you want to assure the staff that if someone really does rise to the level, that they'll have access to the top position. So I simply want to say that remains my view. And I think it's important that it's reflected very much in my experience in the bank. The best people I saw, and as I said, I did deal with them over a period where people that from within the bank who rose to the top position. So independence and evaluation is a topic people like to talk about, but I don't think it's a very sensible topic. I want to tell a couple of things about what I've seen happen during my time in ODE, because I think it's important. The first thing I want to talk about is really what I think has become a best practice, which was a little bit, it's a bit of a hidden gem, because it wasn't, we weren't aware of it quite frankly when we began. I'll say DFAT have always had a process of doing operational evaluations within the working staff of DFAT, and so they're not the big strategic evaluations that ODE tends to concentrate on. And about four years ago, in the discussions when this was discovered, a decision was made to actually do an evaluation of that work. And I have to say, it was some nervousness on the part of the committee about doing this, because no one knew exactly what to expect. There had not been any comprehensive review. And indeed, when the review was done, it was quite interesting, because what one found was actually an important resource of good evaluation. Certainly, I would argue that consistency is not the same with strategic evaluations, and not the same oversight, but there was a lot of good evaluation work being done by operational staff. And I think this is a message that really is important to give, that DFAT both has a capacity to do ODE evaluation, but also allows and supports staff in doing operation evaluations at the working level. So to me, it's a strong message about the commitment of the resources and time of the management of DFAT, and I think it should be complimented. There's been a recent review done again, more recently. Those reviews have resulted in suggestions to consolidate the work. And I think that's important, because I think that will lead to some improvements, because very often ODE is asked to help support that work, and it also has limited resources. The other thing that happened was in the first report, the discovery was that not many of these reports were available to the public. And we pushed transparency very hard in the 90C, and as a result of that, you're now seeing a record of actually getting these reports, not just completed, but they're also all being made available to the public. And I think that's an important additional asset for the discussion of evaluation in the region. My fourth point is a point that I think, I didn't realize as much as we began with, but actually ODE has played a key role in working with particularly the management division, the aid management division, on focusing on and ensuring about the quality of oversight of DFAT operations. So this wasn't in the original terms of reference, but it's interesting, we've had a very good, we've had a very good relationship with the management division, and when issues have arisen about quality, there's been good discussion between IEC and that group. So it's been an open and constructive relationship, and I'll just give you two specific examples, because I think they reflect the sort of work that is being done to reinforce and assure quality. The first one is, you know, AQC quality. These are the reviews of individual investments over $3 million. And so there's a regular process of ODE reviewing the quality of those reports and ensuring that they meet appropriate standards. When there are weaknesses seen, there's a discussion with the working level staff, and that work, I think, has provided important signals to management about the quality of staff, about the quality of the actual projects. Two years ago, two or three years ago, when some very modest changes were made in the instructions on AQCs, all of a sudden there was a dramatic improvement in project quality by at least the ratings. And so this was something both the APE Management Division and ODE worked on. And within a year, there was an ability to go back, review what the instructions were, and get, we felt, and the APE Management Division felt more accurate and more consistent represent performance. And so you had the one example of where I think the work actually made sure that the focus on quality remained high. Second example is a very new one. There's a new acronym, FAQCs. These are the final AQCs. And historically, the instruction in originally, and I'll say it in a de-fab, was the staff had a requirement once during the life of a project to do an independent review. And this was one instance where I'm afraid my biases from the World Bank came to the fore. And fortunately, Wendy, who was also involved in this, but who's also worked in the World Bank, had the same bias. That this didn't make a lot of sense. That what would be much more useful for management would be a review at the end of, an independent review at the end of the project cycle. So that at the end of the project cycle, management would have more consistent messages, both about where investments were successful, where they were not, and where there were problems either at the country level or the sector level. And I'm pleased to say that's now been accepted. And so I think these are now a part of the quality review function. And I'm hopeful that they'll contribute again to ensuring that there's accurate views of where portfolios stand, where there are problems and where there are successes as well. My next point is just to summarize a little bit. My view that a lot of excellent work has been done. And I'm talking now not about IEC work about the work that ODE has done. Both meeting the standards of delivery that were expected, ensuring that with strategic reviews that they're all made available, as well as other important pieces of publication that they do. And so I want to emphasize that, the role that IEC in this is actually fairly modest because all the real work is done by ODE and it's been a pleasure to be associated with that work. Just a couple of things which I remain very proud of. A lot of work has been done over the last couple of years on disability. And this has been a particularly hard area because there's not a lot of work in the outside world. And I think Australia has emerged as a leader in this area. And I think the evaluation both has underlined that point. But there's also began a struggle of trying to figure out how disability work can be better reflected in the eight projects that are done. So this is a work in progress, but I think it's been an important activity. As some people know, I'm an Africanist. I spent most of my back career working on Africa. And I think ODE should be very proud of the work that was done on the Horn of African, humanity crisis in 2011. And it's important not just because it was a good piece of evaluation, but I think it's sent an important message about the capacity within DFAT in terms of its global role. DFAT's not a big actor in the African community. It wasn't even a big actor in terms of amounts in dealing with that crisis and that food crisis. But some junior DFAT staff that were on site made the decision to get involved in the crisis in an active way and to work on a coordination. And in fact, from the analysis, and I think it's accurate, played a much more effective role in improving coordination in many of the longer time big actors in the food business in East Africa. And so I think it was a clear discovery of a capacity that Australia does have to both work effectively with other donors, but also to play above its weight in terms of its role on an important humanitarian issue. So I think it was a clear message there. Very proud of the first work that went completely through the process on Australian volunteers. We did have a discussion. It was the first document that was discussed. There was some criticisms and some sensitivities, but in the end, everybody felt better because it was given an award as an excellent piece of evaluation in the annual review. But I mean, I think there are two messages. He still doesn't agree. I think there are two messages. One, this issue of actually having these meetings with your comments and inputs I think is very important. Evaluation doesn't get this much attention more generally and actually I'll come back to that. The second point I was gonna make on documentation is that he doesn't only do strategic evaluations. They have a full range of other work that they do, both to monitor the work that's going on. They do regularly updates on evaluation policy to make sure people know what's going on. Policy notes come out and this review of past recommendations, I think, as I mentioned before, is an important aspect of the follow up on the learning. And finally, the spot check work which is done on AQC has been, I think, an important part of the overall quality function. I'll comment a little bit on the AAMU issue because I think that is another best practice. I mean, I'm a little bit surprised at your crowd today. There aren't many places that evaluation gets this level of attention. So I'm certainly pleasing to me to see that. Generally, attention falls very much on the sideline. And I think the AAMU defects interaction to put together these on a regular basis has been an important way of both sharing lessons from experience but also encouraging people to take a little bit more about evaluation. And as I said, it's been an opportunity to critique and to make suggestions about how reports can be improved. So I think all of those are important aspects and I wanna both express my appreciation for the AAMU that was done on that front but also really express my hope that this can continue to be built on and sustained because I think it's a somewhat unique. I haven't been able to find any other examples of where this sort of support has been given to evaluation in other areas. I would be dishonest if I didn't suggest some of the issues that I think still remain. These are issues that we spend time on so it's not gonna surprise anyone on the DFAT side. And they're really free. One is if you look at most of the evaluations somewhere it says they're qualitative evaluations. And the explanations are many but I worked at the World Bank. I was supposed to be an economist in the World Bank and I just think you gotta do more work on the quantitative side to produce credible evaluations. There are some evaluations that are good job on that. Some work was done on roads in Indonesia which I think was excellent work on evaluation putting in place rates of return. But this is an issue that we've raised often in the IEC meetings. In yesterday's meeting we had a discussion of one case where we think it's coming to fruition in terms of the evaluation of teacher training where it is gonna be an attempt to produce some real data so that judgments can be made about the impact of teacher training on the performance of students. And I just make the general observation that I think is an issue that has to be continuously pressed. Related to that is the issue of data. And I have to say here I contrast all the time in discussing data, macro data and sectoral data. The colleagues across the street from where I worked in Washington, the IMF have worked for now for over 75 years on improving the quality of macro data. They do it in every country. They acknowledge their countries with stronger weaker data but they retain that responsibility and so you have systematic and consistent macro data. It wouldn't weaknesses but it's there. When you go over to the non-financial, non-macroside, the data situation globally is a mess and it's a mess particularly in low income countries. So when you look at data sets and as I said I worked in Africa, you look in Africa, you look in the Pacific, you know what you see are blind spaces and what you see are spaces with footnotes which says the latest data 2013. And I think data is really important for policy work to make judgments about the impacts of policy, how things have changed. And I think the data issues and getting more data on the table is important. There's not only an evaluation issue, I think it's a broader issue in respect to analysis and how analysis is done and I'll come back to that. My third issue is if there's an area I'm a bit disappointed we haven't done as many country studies as I think should be done by the ODE. Country studies really represent that you give signals about a different unit of account in a project study and it's an important unit of account because I think once you draw lessons more broadly from project experience it can well help understanding where problems and issues are and it can provide an environment for discussions with not just the counterparts at the sector level but also the important counterparts at ministries of finance in prime minister's office to try to bring about broader reform in countries. So these are three areas we've talked about a lot. I think I can point to examples of where there's been progress but I think these are still a work in progress which certainly I expect IEC to continue to focus on in the future. And I'm sure with Wendy there that's going to happen. A couple of issues, I said I'd talk a couple of issues about broader concerns about, not concerns but issues that I see about DFAT, about the aid program. And it's a spiritual modest list because I don't consider myself an expert on DFAT or Australian aid but I've left a couple things I wanted to comment on. The first one is the level of aid and that's been an interesting issue during my tenure where I think I have sympathy with some of the criticisms about the declines but I have to make one positive observation before I come back to some of the concerns that have emerged as a result. The positive observation is just looking at it as a budget issue and how the budget issue was handled. I'm quite impressed with the way the cuts were handled. If you look at the first round of cuts those were largely handled by a sort of consolidation to focus on regional issues, focus on Asia and Pacific as well. And I thought that was a sensible way as a former budget person in the bank. I thought that was a sensible way to deal with that. The second cut was handled in a way that I was actually impressed with because as it was being discussed the one question I had is, does Indonesia get hit? Because Indonesia's had a long and important relationship with Australia. It's an important economy in the region but also Indonesia has gotten more prosperous and probably is less dependent on aid. So do you go down the road of simply endorsing past history or do you reflect the recent developments and actually force a cut in Indonesia and defect did force a cut on Indonesia? So I saw two cutbacks which given the decisions the actual budget allocations I was impressed with. At the same time I think there's been a cost to this and it's not the cost that people so often focus on in my view. The cost people focus on the macro numbers. We're giving some percentage of aid and it's reduced to another percentage. That is a cost certainly I don't disagree with that but out of the school that says development comes not just through money but also through policy change and so money alone doesn't impress me that much. But at the same time I think there have been real costs and I just want to underline from my experience a couple of them. You know the removal I feel personally the removal from Africa was a cost but I have a more subtle point I think more important to that. In my experience the government didn't always like it but I can point a hell of a lot of experiences where African challenges and Pacific challenges look alike and I think cutting off that experience has been done at some cost. For two reasons, one because you've got more countries and they're bigger in Africa and so you have a broader range of experiences which come to the table but also because I've just seen things that have happened in Africa that haven't happened yet in the Pacific and vice versa. But one of the things that I've seen in Africa everybody in the Pacific is talking about the private sector, the bigger role of the private sector. But one of these have happened in Africa that hasn't happened in the Pacific is there's a lot of private investment going into self-standing solar programs in rural areas which are totally self-standing, self-financing. And I've done some work in the last six months in the Pacific and nobody's doing that and there's still major challenges of delivering electricity in rural areas in the Pacific. And if the choice is between doing that through governments which already have financial constraints or being able to bring in a private sector to do it I think it's something that is actually important advice for the region. So that's one thing that I feel. The other is that you saw a significant reduction in Australian participation in global programs and global interventions. The one I know the most about is Australia expecting to become a major donor in the global education front and not essentially got precluded by the cuts. A final comment is a more general one that for a while Australia was getting a lot of attention and interest from the outside. And there were a lot of visits and maybe sometimes for the wrong reason maybe people were coming to see if they could tap into some of the additional funds. But the point is I saw much more activity of people coming in from the outside including said with a little embarrassment from the World Bank. But the point is being in place where there's noise and there's interest and there's enthusiasm and there's all the resources is not a bad place to be for working on development. So I think in all three of these areas I think there have been costs. And again it's not so much about the aggregate cuts it's just about the messages and what had to be dealt with as a consequence. My final comment, my final broad comment is about economics. You know I come from an institution that's well known to be dominated by economics but I have to say I am still consistently taken back by the limited economic skills in the Australian aid system. And it's important for the programs you're working on. I mean growth in the Pacific is the issue. If you can get growth I'm not one that's gonna argue every country's gonna be sustainable. But I think the right challenge is our government's working to do what they can do to improve their environment, their policies and their positions. And you know that challenge is a real one. And I think the need economists to work both on the growth issue but more generally on policy issues economists have a useful role to play. I'm saying this in spite of the fact that I think economists in my home country are there embarrassing themselves constantly with poor analysis and poor policy recommendations. But I hope we can be higher standards than that in the development field. And I do feel, I mean I've met with every chief economist since I started working on the Pacific in 2008. And everybody agrees we need more economists but it hasn't happened. I also think economists are a vital part of getting some of this work done on financial issues and on quantitative work to support the work on projects. And economists play an important role in that. So it's a message that I think everybody gets tired of me talking about. I always say where are the economists? I think there has been an effort certainly under the work program of Modi to ensure that when there's a case for economists that they are an important part of the team. But I just think it's more general that this is a shortage in the system. I have about six comments on the Pacific challenge to conclude as I said this is an area that I've been spending some time. When I went to work in East Asia I didn't think the bank was taking the Pacific very seriously. And so in that job I think I can say with a straight face that I certainly encourage more work there and more attention there. And I'll come to that because a lot more has happened since I was there than when I was there. So I want to be honest about that. But as I've emphasized I think the growth issue is key. Getting growth up in the Pacific is key. The slow growths of the past are not going to provide for the types and the levels of quality of life that people want to achieve. As related to this I think the private sector challenge is key. And here again I feel a very strong similarity between what I see in the Pacific and what I saw in Africa. Every government now is sensible enough to say that donors bias the private sector is important. Yes we're focusing on the private sector. And so what I do these days is actually I don't always push banker IFC work but I always go look at the doing business numbers. And you know a place like when they began in 2007, 2008, a little over 10 years ago Fiji was actually a pretty good performer. It was in the top 30. Today it's around 100. When you look at all the other countries in the region just like the countries in Africa they all fall in 100 plus or around 100. And what that shows I think are two things. One is the absolute performance isn't very good. But I think more important it's a message about relative performance and what's been done because you know the doing business stuff is not magic and it has to be always handled with some judgment and perspective. But what it does show is some countries move up and other countries move down. So you see relative performance where I think it's a fairly reliable guy. And you know a country in Africa who wanted it was you know in the high commitment hundreds you know is now in the top 30 because they've really worked hard on improving policies for the private sector and it's showing up on the roadside. So again, that linkage. My third point is to come back to the use of the the MDBs in particular the Asian Bank and the World Bank. And I have to compliment the government of Australia here. I think they've done a really good job of waking up the MDBs with respect to the Pacific and getting involved. Over the last five years, ADB support and ADB had a more traditional relationship. But ADB has doubled its support. The World Bank has tripled its support from a lower base than ADB. And so for the first time you really have credible engagement in the region by the two major MDBs. And so I'm hopeful that they can certainly play a role. They now have big offices in Sydney, big regional office in Suva, increasingly putting staff in individual countries to work with the countries under reforms. And I think that's an asset. And I give the Australian government enormous credit for this because certainly during my time I saw the emphasis that's obviously been sustained. If anything become more effective subsequent to my time. So I think it's an important aspect. One of the things that I see, it's hard to see from the region but I see as a comparative basis is when I was country director in 10ths of the year, Uganda, I had 30 donors out there. All of them had money. And you work in the Pacific and it's four or five donors. And of course you face sometimes the challenge, sometimes the asset of always being the biggest person in the room. So I think by engaging the MDBs it helps because they bring experience from outside but also because they're bringing funds. One of the things the World Bank has just done is made Fiji eligible for IDA for the first time. And I think that's gonna help increase the program that the bank can do in Fiji. Fourth point, capacity building. Certainly the center's doing a lot of this with respect to some of the neighboring countries. I'm a big fan of that. But I think more generally and I feel the same way about Africa, an enormous push was made in independence in these countries. And that generation is starting to leave. And these are some of the senior good people in these countries. And for a variety of reasons, I think there are challenges for the next generation that haven't been very effectively addressed by the international community. So I'm a big believer in trying to think through this issue. There's always frustration when people say, ah, we've done all of these scholarships. The two observations I always make is one, that you've gotta be more selective and identifying people. And two, I think the effort is too broad. I think we've gotta make a decision to focus on some key skills and make sure those skills are properly supported. And this gets people upset when I start discussing which areas, because I always begin with economists of course. But the point is that I think as I look at donor programs generally, they've treated low income countries in a way too often that you were trying to replicate capacity that existed in countries of much higher levels of income. And that dispersion has resulted in not so consistent capacity areas where you really do need capacity. And high level capacity deal effectively in today's world. So, as I said, I think capacity is still an issue. I have to highlight the work being done here on labor mobility. I think it's really important in the context of the global climate change and some of the other issues. The bank had done some early work on this largely with New Zealand and I think that it's been picked up particularly here is very important. Obviously climate change is a critical issue in the region. I was hopeful that new instruments were going to bring a lot of new resources. I was never one to believe these numbers of 100 billion a year. Those were completely fabricated numbers, basically counting anything anybody did on anything as support for climate change. I've been a little disappointed that the institutions haven't picked up as quickly as one would have hoped, but there clearly is additional resources. And what I find interesting is, you know, on the ADV and World Bank side, their view, which I think is the right view, is that if the institutional problems continue, they're prepared to scale up and use more of their resources on this because it's such a critical issue. Just a complete issue for the region. And finally, as I said, you know, I think this African story and the interactions and strengthening those debate, you know, some important contributions on both sides. You know, one of the strange things I found, you know, coming from Africa, you think of Africa as aided dependent. And when I joined the East Asia region and the bank, they produced all these briefing books, none of which I look at. But very shortly after I joined Bob Zellick, the president of the bank, was coming to Australia. He was coming to an APEC meeting. He had been involved in the formation of APEC. So I had this nostalgia. So I accompanied him and then, you're on the plane, it's a long ride. Once you read the briefing book, and I remember the exact number, which I thought had to be the state, that the average per capita assistance to the Pacific was $315. You know, I was a hero in East Africa if I raised $50 per capita, I mean, I was a hero. And so that was what, you know, made this impression on me very clearly. You know, it is an A-dependent region and there are some linkages that can be exploited. I'll conclude by making a couple of points. First of all, I thank Wendy and Steven who are my counterparts. Steven for the best four-year, Wendy from beginning. We've had a good relationship, everyone's still out. We are you, but we are for it. And we've always managed to give consistent messages. And I think their part of that has been key. And finally, I think OED, there are a lot of OED. I did it. This is an inside joke. For my first four years in IEC, I kept talking about OED instead of OED. OED? OED? OED? The Operations Evaluations Department in the World Bank. So I couldn't deal with it. But I could much better at that. Now I've screwed up, but I apologize. That's a public statement. But I wanna thank the work that's been going on. We've really had probably two or three generations now stay up in OED. And I think the quality has been sustained and built, I think, to work. And they, as I said, they do all the work with respect to the evaluations. We certainly are in a privileged position of being able to comment with Missouri. But I think, and I'm sure you can be thankful, that it has a very good group of people working on evaluation within the development side. And that group is given appropriate support within DFAT. So I've enjoyed my time. And I guess I'm available for questions or comments. It's very much for the gym, I'm sure the economists in the room are super happy. I'm very proud of the International Development Economics offerings at the Croppett School, if anyone wants to make a shift. So we do have a little amount of time for questions. So please do use the microphone though, because we are recording this session of DFAT Internet. So, any starter? Thanks for the talk. I was wondering how the World Bank deals with evaluations where you have a simple religion in this, but you have 15 programs, and all the programs are insurrated, and how do you evaluate this impact of the single program? And that's obviously a very difficult part of the question of wondering how the World Bank dealt with those issues. No, let's take a couple. Okay, we'll take a couple. Thank you. My question is about your seven-year summary of OBE work. What success in improving the quality of the M&E frameworks in individual programs, things like baselines, realism, attributions? Thank you, Jim. We miss you very much. I just wanted to ask you about a couple of things. First of all, do you have a comment about country ownership? Country ownership of the monitoring evaluation systems that belong to individual activities, but also country ownership when it comes to evaluation and what that role should be, what OBE's role should be relative to country ownership? And the other thing I'd like to hear your comments on is that Australia, if the local government is elected, is talking about an evaluated general office, that would exist as an independent office in the company and system, I wonder what your thoughts were about and evaluate the general? First to start with a question, particularly in the World Bank, I mean, the evaluation board is much bigger. So you're producing scores of reports, both at the strategic level, but also enormous amounts of the transaction level. The bank does, to make it related to the FAQ policy change that I talked about, the bank has every test manager complete in evaluation at the end of a project and then has a formal review by the independent evaluation group. So there's much more going on in the bank. It has to be both do things at the macro level and do more targeted evaluations in a way that I think is harder with the operational size of the fact. It doesn't have to make too many hard choices in that respect. The big choices, I mean, I think the one issue that we have a new director general and she's struggling with what the priority should be. I think the big question in the bank is evaluation move from a big focus on investments and quality of the program to a lot more focus on less focused discussions on bank procedures. And I think it's gonna be interesting to see how she reacts, but my suspicion is she's gonna go back to be a bit more traditional because you had IG commentating on sort of everything the bank management did. And the bank management, in my view, wasn't particularly impressed with that. So it has the capacity and it certainly does do both broader studies but also is able to do them and has a lot more flexibility than the effect to do that. M&E, well I'm gonna join M&E and the discussion on country ownership and because I have a somewhat eclectic view on this, it's not the view of most agencies including the World Bank. An enormous amount is spent on M&E in individual projects by all donors. And yet, the performance is not still including in Australia, including the World Bank. And I think I've discovered the reason for that and worked hard in the bank to convince people that I should be listened to or wasn't very successful. The point I made was in the end, the ownership is the key thing. And if you look at a minister or PS, why would they be interested in M&E for an individual bank project or an individual Austrian project? The issue is to get a broader management system and a broader data system actually works. And so the data thing that I talked about is part of it and it's getting more attention in the bank now because the bank sees these weaknesses and I think the whole issue of big data has put pressure on shouldn't development be thinking more about this than it has historically. So I've been a big believer in if you're interested in the ownership question, you should be developing government systems to monitor what's going on generally. And as part of that, yes, have some ability to do some granular work. But that's not where the money goes right now. The money goes, everybody tries to build projects just in the monitoring evaluation systems. I mean, you don't want to say the defense staff must spend hours and hours harassing their contractors. We need better data on the project. But I think it's a get the overall system going better. It's a weakness. So I have a view on that. It's not an accepted view. We certainly spend time and I can see talking about grooming data. It's an issue with management and it's certainly a concern of the management that it gets. But on that I'm afraid we'll see that the fat face is exactly the same problems that all the other donors do. And it's a very mixed record as it was. And as I said, I have a logical explanation for it but the traction of that explanation has not been very effective. Okay, evaluator general, it's interesting. During my seven years, certainly more than five times we've been called by other parts of the government to say, what are you doing? Why is it useful? We go to those meetings. We have interesting discussions. The conclusion I reach from that is that in order to evaluate general whatever you want to call this probably makes sense. Because right now, if you ask me honestly, the aid program is under a much more serious microscope than the rest of the government program that has emerged in these discussions I've had. Now, in hardly it's a function of aid generally has emphasized the role of evaluation more than most government programs and has strength and it has weaknesses. But I think it would probably be constructive that some other groups be subjected to this sort of discipline and other expenditure items that you have in the aid program. I also have to say that in DFAT experience you've got enormous support from the leadership of DFAT. And the concern that emerges in that is a concern to make sure that all the evaluation doesn't simply focus on aid. That there are other parts of DFAT that could also use some attention and focus in terms of evaluation. So I'm impressed at least within the ministry, at least with the secretary, there's a clear sense that she's giving this priority who wants to see better work in this area. Now I'm sure knowing diplomats as I do, you'll have long explanations of why you can't do evaluation, so we'll see. But my instincts are clever people could probably produce some interesting observations about what works and what doesn't. We do have time for another set of questions. Hi, thank you. And earlier on you mentioned the operational staff doing evaluations in DFAT. And I think you referred to it as a hidden gem in the department. What do you think are the relative strengths of operational areas doing, monitoring and evaluation work and specific evaluation areas doing that work? And if you were to build the ideal system, how would those two things be next? Jim, I just wanted to thank you on behalf of the AU for your engagement over the years and your contribution. I can still remember when you were announced, which I think was on the budget line. And of course we hope you'll still continue to visit and contribute to your time. But I just wanted to ask, you're probably the only person associated with the A program who's doing the same job now. Who were you in 2012? I just wanted to give you a good reflect on differences. You know, the means of bishops, but sort of government and of course from Aussage to DFAT. So any questions on that was good. You're engaged and I'm sure we'll interest us. Thank you, Kim, we're a new kind of Indigenous affairs. We've been very lucky to be able to follow in the work of your experience because you are a leader in the approach and the ODE takes I think in the Canberra Public Service. So it's been really inspiring to see how you've done what you've done with the community. I wanted to just pick up on the issue of independence that you mentioned and you focused I think on the independence of the head of ODE. And I wondered if there were other aspects of independence versus fully that you were going to elaborate on reflection. Because I think there's a lot more to it in terms of the independence, which is used as a flag I guess for something that's high quality and expertise in an independent external evaluator. But I don't think the world is quite that simple and I think everyone is subject to various forces which make us do whatever we do. So I think there are lots of risks on the side. So Mike, do you have any reflections on that? I'll talk about the positive side of independence and get away with some of these narrow ideas because I agree with that. Let me start our self-evaluation. You know, I'm a big believer in self-evaluation because there's one way to ensure more direct learning that people actually do the work. So in that sense, I think the operational work is very good. Within DFAT, it's often an interaction with ODE staff. And so you do have an interaction. They reach out to ODE for help. And so I think it's quite a constructive, quite a healthy piece of the evaluation puzzle, which as I said, was not really elaborating on or explained when we started doing this. You always have to make choices and you still have to deliver your broader strategic program. And so there are constraints. My instinct, but if I was asked where I would put more money now is I think DFAT has to be better at ensuring that the lessons that are learned are more broadly disseminated across staff. And it's a tough issue because it means taking staff and I'll actually come back to this on Steve's question because I do have some views on what's happened as a result of the change. So I just think you gotta encourage self-evaluation. What you gotta recognize, it can't be all self-evaluation because you do want someone to be over people's shoulders. You need a mix. And I think DFAT in the presence of this situation has a very nice mix. Now, Steve's asked a hard question, which is what I would have expected. I'll say one thing where I agree with Richard Moore's analysis. The government has a focus on reducing the number of staff in the bureaucracy. And this is pretty common now in these countries these days. And I think there's an issue there with respect to the aid side because in the end you had an organization focusing exclusively on aid and the delivery of aid. It's been absorbed in another organization and you both reduce size and you put the staff in a situation where if you're a reasonably ambitious young person, do you wanna do the aid route which is now a smaller piece of the puzzle or do you wanna do a more traditional development route? So I worry what's happening. Richard talked about this, is that there's a movement of resources in an overall more constrained situation which means less resources are going to the aid program. So I worry about that because first of all, it's a different business. And interestingly, it involves a lot more resources. I mean, when the reorganization first happened, I figured I better take a look at what this DFAT is about. And DFAT's main resource issues are about maintaining the quality of embassies and the security of staff. That's a very different business from trying to do development to cross Pacific and particularly of trying to do it in the Pacific. So I think that's challenge. And I think that's the big one. And I hear the rumors of what different people were saying and doing about aid. So I'll leave that to you in terms of those discussions. But I hope there's an ability to do serious analysis of that issue so that there's a confidence that the staff will be included in place to properly oversee the program. I've had to learn a lot because it's a very different system than the World Bank. And the World Bank, we make the governments do the projects. Now, the consequence of that is that I think you're better on the policy side, a little bit better on the Basque building, but everything takes a lot longer. You know, it's sort of a quiet secret that ADB and the Bank take longer to do things than most bilaterals. Because bilaterals traditionally use the Americans and the Australians have a model and heavily focuses on bringing contractors to it. So I can do a long speech on the relative strengths of weaknesses. I won't. But in that piece that they're doing, you've got to have the capacity to oversee contractors. And I will say that Australia nicely complements technical assistance to that work so it doesn't take no more working with the bar and strengthening the borrower. But again, that takes time and resources. And if those resources aren't there, the capacity thing is probably the first thing that will slip. The next thing will slip as the analytic work that's done. And in the end, you'll still be doing projects, but they won't be as effective. And you won't be dealing with what I think are the key issues, the underlying policy issues. So you can leave all of those to the World Bank and the ADB, but I don't think that's a good strategy. I think you have some advantages in working in the region that are important and should be exploited. Now, I was tough on independence because I was caricaturing what so many people talk about. But on a small line, you want independent people. You don't want people to come in the room and feeling they have an obligation either to the bosses or to someone else. And it's a tough issue. I can honestly say we've never had anything edited. We've never been told we have to change anything in the analysis we've done. It's been done by the ADB, but they are members of DFAT. I recognize that at times it's going to make them think about some of the judgment and issues. But I think it's of course pretty high on the independence front. I've seen no problem of management interference or management questioning of what was done. And I think most likely that the work that emerged on equality issues emerged sort of quietly and effectively. We were not kept out of the problem. We raised an issue, people invited us in and said, we're worried about this issue as well. Do you have any suggestions? So I feel the independence is pretty robust factor in DFAT. And I always say, you get so independent. I mean, this is the story of what the UK did. Actually, just as I was coming to head IDC, the UK essentially threw out the IDC model and went and set up a very small group of people that were part-time, they were going to oversee consultants doing work. And so I was asked in the early discussions with, when it was still on stage, should we be thinking about that? And my reaction was very strong. I said, look, this is going to be disaster. I said, what's going to happen is consultants are going to think it's their job to criticize DFAT, which they're good consultants in the UK. So they'll do that. DFAT will then have no choice, but not be constructive in response, but fight like they'll approve those observations wrong. And that's exactly what happened. Now, interestingly, the woman who's now head of the IDG went over to that group and actually really changed the incentives and got away from the model of using outside consultants but had a model which much more liked what DFAT is using. And so she was able to turn the program around, but it goes to the point that you could push independence to a point that you're dysfunctional in terms of actually working within the system. And as I said, I have no fear of that particular problem, given my experience, though, that's something. Wonderful. I think we will have to go up to a close then. I just want to say a few quick thank yous. I want to thank half the leads of the Office of Development and Effectiveness for partnering with us again on this event. It's a great opportunity for us to bring you and have you take on Australian aid. I would also just like that we had a look up this week by Rob Christie from ODE on performance culture on Australian aid. So if you haven't read that, we'd like a deep wrap to take. I'm sure we can wrap it today. Do check out the blog at deadpolicy.org. It will definitely be helpful. I'm going to see more events together with ODE on so past pesky tech up here it is. But most of all, I would like to thank Jim, who, as Steven noted, has come to the centre many, many times over the years. I've been at the centre six years, so you're probably coming a little bit more than me, but you're definitely one of the visitors that we really look forward to welcoming on, you know, once or twice a year and seeing you and hearing your perspective. So please do join me in thanking him and wishing him the best on his next visit. Thank you.