 on the call. No surprise it was of interest to folks, so I'm really pleased to be able to share our insights and our experiences and to give them you as well about your own insights and experiences as well. Before I get going too far, I also would like to acknowledge the traditional lines of the lands on which women today, on joining from Wurundjeri country, the land of the Kulin nation, and pay my respects to this past, present and emerging. Also like to acknowledge in Aboriginal and Torres Strait Islander people that are joining us in today's webinar, indeed, and with the series that the areas run as well. We thought this is kind of a little bit in two parts today of what we talked about. I'll do the front part and Nick will do the second part and then as I said, I'd like to hear your questions and your comments and your experiences as well. We thought we'd start with a bit of a review of why collaborating in the first place. What are we actually talking about? What are some of the reasons that might drive us to collaborate and the reasons why others might try to collaborate as well? A little bit about what do we mean by collaboration? There are all of these words and Stuart mentioned some of them, all of these words that we use to describe collaboration, like it's different forms and functions and the different answers. So a bit of a review of what do we mean by collaborating, what is it? Move on to, well, why are we going to invest in learning and evaluation for collaborations? What are some of the reasons that we think that it's important to invest in ineffective learning and evaluation for collaborations? And what are some of the benefits and the results that can come from that? A little bit about, well, what can we go about evaluating when it comes to different collaborative initiatives? What are some of the different questions that we might ask with collaborations? What are some of the different methods and tools and approaches that are available to evaluators and those that are working in different collaborative structures to better understand the work that we're doing? And then a little bit about some of our sort of suggestions and recommendations for what those that are working collaborations can do to better undertake their evaluative work, but perhaps also what we as evaluators can do to find different ways into collaborations to help bring them to life a little bit through the work that we might do. So a little bit of a menu of what you can expect to hear from us in today's seminar, as I said, really, really pleased to get in for discussion and give you some of your own thoughts and experiences also. So why collaborate in the first place? I think Austin and Sidney described it well when they talk about the complexity of the problems and the challenges that are facing us in societies that are all living in, really transcending the responsibilities, capacities, capabilities of any single active organization or need sector of society that you own. It's really through the working together, the pooling of resources, the pooling of risks and talents and working across different sectors and organizations. It's through that work that we believe that will change when it comes to the complex problems that we're all facing. When we think about complex problems, I'm sure all of you will have your own challenges that you're working in through your own evaluative work. But we think about things like climate change and poverty in communities and chronic disease prevention. It's really thought in complex problems that don't sit neatly within one sector, within the remit of unorganization. So it's really through the pooling of resources that real progress against those challenges can be made. The core of all collaborative initiatives is really this idea of collaborative advantage, which I'm sure some of you will be familiar with in schooling. And it's really that notion that more can be achieved by working together than it can by working alone. So that it's through the combining of our skills and talents and resources that we can do things better, more efficiently, more effectively, than perhaps what we might be able to do as a meant to be working by itself. And that collaborative advantage really recognizes we think that the benefits that both individual collaborators need to gain from the work that they might do within a collaboration, as well as that collective good that comes from working for it. So you can see up there some of the other things that we might expect from a well-run collaborative effort. You can see that we might expect things to be more effective. We might be able to better invest our resources in order to be more effective in achieving the outcomes that we're sitting to generate. Through working in good collaborations, we might be able to use our resources more efficiently. Some collaborations are about reach. We might be able to connect with more people in more places through the skills and connections of different collaborators. And finally, collaborations enable us to deal with risks in different ways as well. They allow us to distribute risks amongst different organizations and amongst different entities. It might also allow us to take on bigger and perhaps more ambitious risks of what we might do by ourselves. So just some of the reasons that might drive why we and others might find ourselves either in collaborations or seeking to get involved in collaborations. And that's obviously not a resource to us. So if there's some of the reasons or drivers or motivation why we and others find ourselves in collaborations, what do we mean by this word of collaboration and how might it apply in the work that they're doing? I'm sure all of you will be familiar with all of these words and confused to describe collaborations. They're like networks or alliances, communities, coalitions, consortiums. So this goes on and on and on. We, like other folks, would often use the word collaboration. I suppose it's more of an umbrella term that captures some of these other terms that then fit within them. And they're all different in different ways. And I won't go into the details of that, how a network might be different from the publisher or the lies, but they're different terms of the range of different parameters. They can differ in terms of their formality. For example, some partnerships can be really formal types of structures, but there might be partnership agreements and contracts that exist between different organizations or between different entities. And compare that to a network, for example, where it might be much less, there isn't necessarily that formal agreement or contractual relationship that exists, that exists among members. So they can different terms of their formality. They'll differ in terms of their purpose, the reason why they exist and what they're seeking to change in the world. They'll differ in terms of their size, some can be really big, some can be really small. The time horizons against some partnerships can have a very sort of enduring long-term vision of how long they're going to exist, whereas other things like task forces, for example, might be formed in a more immediate form that they're not intended to exist for a very long period of time. They're much more short of formality. They might differ in terms of the nature of the relationships or the things that are exchanged amongst and between different partners, their knowledge, their money, their time, their equipment, their evidence or information. And they can differ in terms of the intensity of the relationships that they contain. And again, I'm sure all of you will be thinking about the types of partnerships that perhaps you're in and the types of partnerships that you've been in as well, and some of them are super intense, where you're meeting with partners on a really regular basis. You're almost working hand in glove with them, or other types of relationships that you might have with other folks through these different collaborative structures are much less frequent, much less intense and require much less help. I think one of the important things that we remind ourselves often and as well is that more isn't necessarily better. So there's this sort of overriding tendency to think within, within the literature, within partnerships and in collaborations that we need to have more partners or stronger connections or deeper connections or frequent connections, when in actual fact what we know is that sometimes lucid connections are much better. So it's really the purpose that will drive the form and function. And so one isn't better than another, they're just different. And that obviously has implications for thinking about what we might do as evaluators and how we might seek to learn about these types of things when we're thinking about what they are and how they're structured. I wanted to give you just a couple of descriptions of how we might go about understanding or describing collaborations. And there's lots of different models, lots of different ways of understanding and talking about them. One of them is through sort of a life cycle then. Sometimes we think of partnerships, bang, here they are, they exist, all of a sudden we're in them and they're fully formed, they're fully structured and now they're doing something. We're in actual fact, we know that it takes time and energy to build and broker and develop these types of collaborative structures. And so thinking about them through a life cycle might be valuable, particularly in the context of evaluation. This particular model comes from the partnering initiative from a couple of years ago and some of you I'm sure will be familiar with it. But beginning up there in that right hand quadrant is really the beginnings of what a partnership might look like. This is that scoping and building stage where we are finding our feet, working at who we might partner with, what we might do together, the kinds of projects or initiatives we might work on together, who those different entities might be. So that's sort of early stage and sort of working out what the foundations of this partnership might be. As we move around that, so we then get into the managing and maintaining phase of partnering. This is really the doing of the work of partnership and Stuart mentioned. This is going quite broadly, lots of attention in terms of the folks that are working in partnership. So we're finding the right structures about how we might work in partnership. We might be delivering research projects, not be implementing the services, we might be going about doing the work of what a partnership is doing, what the election is doing. So that that doing phase. The next phase on reviewing and advising, interestingly is one of those phases often just gets skipped over. So we spend so much time in working out the building, the broken, we're doing a partnership. We often just jump over this. But we think, and I'm sure many folks on this call will think that this is actually a critical phase for effective partnering. So this is where we do get to pause and reflect on the work of what a partnership is doing, that we delivered what we wanted to deliver on the right parts of the table. We're doing what we said we were going to do. What are we learning about the quality and strength of our relationships and what are those relationships and enabling us to do that perhaps we couldn't do by itself. So that reviewing and revising phase has been a critical one. And that fourth phase of our sustaining outcomes. Some partnerships can think of themselves as the existence of their success. So they want to endure forever. Whereas other partnerships might get to the end of the work that they've done and say, okay, we've done what we wanted to do and now we can dissolve this partnership. And perhaps it might be reborn into something else or perhaps not. But that third that was at that fourth and final phase is really that finishing that finishing point of have we done what we said we were going to do, could we do more of a put or do less of a put or do it in the future. So that's one way for understanding partnerships, which again might have implications for those that are in the valuation or doing the valuation partnerships. The second way we can think about partnerships or a second way we can think about partnerships and understanding them is different types of partnerships. And again, lots of different typologies. And this is one that you might have seen that might be used and it essentially buckets partnerships into three categories. The first one being about those that are leveraging exchanging partnerships. So this might be where we're seeing partners coming together in order to exchange a set of resources that might be knowledge, might be information, might be evidence. Importantly for achieving their own strategic plans. These are often transactional, short-term and nature types of relationships. You can think of something like a networking event or meeting or society that brings together people in research, for example, that might share their insights around a particular topic. You get a bit of information and you go away and do what you're going to do. It's one type of partnership. Combining and integrating, this is where we're seeing two or more organizations or entities that are bringing a complementary set of resources in order to do something that they can't necessarily do by themselves. See where it's around cogeneration and mutual accountability, creating new things in these types of entities. You might, for example, see something like a funding agency and a research organization and maybe a service delivery organization coming together in order to test and scale an early childhood program, for example. Combining a complementary set of resources in order to do something with accountability. The final type in this particular model are those that are called transformative partnerships. You can see by that work there they're a bit more ambitious than perhaps what some of these other partnerships might be. Again, we're seeing organizations with complementary sets of resources coming together, pooling those resources in order to do something that they can't do by themselves. Importantly, with these types of partnerships, what we sometimes find is that when they first come together, they don't necessarily know what it is that they might do. There needs to be that time and space for new ideas to form innovations to take seed and to take root into emerging the work that they might do together. Three different types of partnerships that obviously have implications for how we might learn about them and what they do. Importantly, again, one isn't better than another. They're just different, but as implications for how we might learn about them. That might all sound very theoretical. I wanted to give you just a couple of examples of partnerships that we've worked with, perhaps some of you may have heard about, but that we've worked with over the years that fit into some of these different categories. The first one is the Global Plastic Action Partnership, or G-pad. G-pad is, as you can tell, a global partnership that is focused on plastic pollution reduction, removing plastic from our oceans and borders. It operates at a global level. It operates at a national level, at a regional level, and all the way down to local levels as well. It's bringing together organizations from government sectors, from non-government organizations, from private sectors, from civil society. It's got a huge number of partners and members that have now part of this partnership, more than 600 now around the world. It focuses on a whole range of different things. You can see some of the things that it's trying to do on the life and the society. It's focusing on how can we go about transporting the behaviour of consumers as it relates to plastic purchasing and repurposing the cycling and the use of things like campaigns and behaviour change initiatives. It's seeking to inform policy of national governance, where it has a set of relationships around plastic reduction policies. It's seeking to boost innovation and supporting those that are involved in innovation and entrepreneurs within countries, helping them identify innovations at scale and spread them in different ways. Measuring plastic pollution is notoriously difficult and people use all sorts of different measures and metrics and tools in different places around the world. So one of the things that this partnership is doing is trying to find ways that we can better measure through a set of common metrics plastic pollution reduction. It's seeking to help people access finance for the work that they're doing and importantly in the world of plastic pollution, particularly waste cooking, we know that there are variable effects felt by people of different genders and so it's making sure that that lens of gender is included in the work of all the other partnerships and of what partners are doing in local countries. So an example perhaps of a more transformative partnership which is seeking change at multiple levels and multiple places and through a range of different means and mechanisms. Another example perhaps on the more combining integrating or leveraging exchanging type of partnership is the Knowledge Development Exchange Hub in child and youth mental health. This is a hub which is based in Canada and is a pan-Canadian hub and it's seeking to bring together researchers, policymakers and practitioners that all have shared interests around child and youth mental health promotion which is an emerging field within Canada and elsewhere as well. So this hub has really been formed, the hub's a good name for it, it's really been formed as a shared state by which all of those different folks from those different research, policy and practice orientations can come together and to share their insights around what works as it relates to promoting positive mental well-being of children and youth. So this hub has a particular role around supporting people to generate knowledge so it's providing resources to enable people to do that. It's providing a home to exchange that knowledge between researchers but also from researchers to practitioners and policymakers to ensure that there's a mechanism and a pathway for connecting knowledge with action. It's seeking to build strong connections and relationships again amongst those people that are part of the research community but also amongst those that are outside of that community and it's increasing access to a range of different resources as well be they financial resources, helping people access resources they need to the work that they do but also accessing different resources they might need for designing for example the studies they're working on. So an example of a different type of partnership again that would have implications for the types of evaluations we might do with and for them. So that's the kind of why we might collaborate, what collaboration is and some examples of the different ways that we can think about them. What do we know about how to collaborate effectively? I know I spent too long on this because I want to hand over to you in a moment but the literature for those of you that are steeped in it will recognise that it'd be kind of a bounce with these critical success factors, what we need for effective partnerships and platforms and publications and alliances. Ten of them you can see up there on the screen. These aren't necessarily exhausted but these are critical and what we've seen repeatedly over and over again in the literature. So a shared vision and vision being a foundational element of what we need for strong and effective collaboration. Are we going in the same direction? Do we have sufficient alignment between our own goals and the goals we might achieve together across the state? So a shared vision and vision critically. Distributed leadership so recognising that authority and skills and expertise might line in different places within a collaborative structure so making sure there is space and room for that leadership to take place. Accountability and commitment that there are clear and transparent structures that we're all aware of what each partner is doing and hold themselves in each other's accountable commitments. Financial strength and making sure that there is enough financial energy and momentum behind the work of this partnership that it can actually do from what it says it's going to deliver. Trust and goodwill is something which isn't necessarily there right at the beginnings of many collaborative adventures but something which is built and earned over time spent working with people. Clear governance structures that are in place again that are transparent to folks that we can see how decisions are made within this particular collaborative structure. Inclusivity so making sure that those voices and perspectives that need to be part of what that are affected by the decisions by the partnership or collaborations. Doing that those voices and perspectives are foundational included within the work of the collaboration. Reciprocity and mutual benefits so that the energy that's been put into the work of any partnership recognising that there's a huge cost for many folks that they're getting something out of it. Often we like to gloss over the idea of our own self-interest within partnerships and yet it's critical that partners are able to gain things from working in partnership in ways that they value and not to say that that has to be the same or equal for all partners but it has to be of meaning and value to those people. And finally over to the one that I haven't mentioned the continuous learning and evaluation and it's going to talk about this in just a moment but something which we see as being critical to high performing and high functioning partnerships is an investment in learning and evaluation making that front and centre in terms of the work that they do so that they're able to assess where they've come from and assess where they're going to describe the difference that they make in the work that they do. So a little bit about partnerships I hope is useful for sort of context setting in background before we now start talking about for the so what of all that for for the value I should expect. I'm going to stop sharing my screen for just a moment and it's going to share his and then we can come back to some questions at the end. Thanks, Cam. Hi everyone. Nice to see so many people here and interested in something that we're really passionate about which is this learning and evaluation in platforms and partnerships. I think Cam gave a really good overview there. It doesn't stop the amount of time you do this in the amount of work you do and you still kind of find out new things and different perspectives so I appreciate going back to those theories sometimes it's useful. As Cam said I'm going to talk a little bit about this why invest in learning and evaluation. I suppose if we don't have to convince all of you why to invest in learning and evaluation because that's what we are here in Evaluation Society. But I think there's some really key principles and ideas that are relevant to platforms and partnerships. So I think Stuart said this actually right up the front. There's a lot of effort that goes into the processes of partnering. How to build them. What do you do? Who do you get to be part of it? How often do you meet? Who's in charge of decision making? And there's lots of ideas around how do we make them stronger and better and work for more people. But what we've noticed certainly and I imagine lots of you have as well is that there's comparatively kind of less energy that's put into learning about them. So about what the partnership achieves. What they achieve for its members. What they achieve for the world. What they achieve for collaboration. And there's yeah there's just a lot less energy that goes into it and of course we're really interested in this learning space. So this is not an exhaustive list but some ideas around what could be some of those benefits of learning and evaluation within collaborative initiatives or entities under whatever name they fall under as a list of alliances and coalitions before. So I suppose and this is probably a pretty obvious one but the first is that it might allow you to kind of chart progress achievements and the things that are happening or the benefits that are accruing whatever they may be for the partnership or for the world. And as with all kinds of evaluation and all kinds of learning instruments I suppose they create products to talk about what you did and that could help you support efforts to come secure further funding. So you can use it to advocate on behalf of the work that you've done and the work that you might need more support with. And we find it opens up new conversations. So this idea around wanting to attract new partners or wanting to have different people within the mix. So ideas around this is this is what's happening here. So we try this thing out and it's and it's going well. We tried this thing out and we're going to stop it now. And we do see this as a device for bringing on new team members or existing team members keeping up to speed. Not everyone's interested in learning and evaluation but most people are interested in what they're working on. I suppose then that kind of links to that increasing attention from prospective partners and we do lots of work with supporting partnerships to identify who they want to work with and how they're going to get them and how they're going to attract them. And so some of these learning materials can actually help you increase that attention and get the right partners in the room. We would hope I suppose as evaluators as people who work in learning and evaluation that all of this stuff improves decision making, that people actually use it for decision making, but they allow partners to kind of reflect on what's happening, to learn from what's happening and to change course as need be. And finally I suppose this is maybe an interest of ours is that we don't know enough about how partnerships and collaborations work. We don't know enough about what are the things that they are, that collaborative advantages is contributing to the world. So in a kind of I suppose in a useful way all of the efforts in learning and evaluation contribute to a greater knowledge about how collaboration works. Can I mention this up front, the kinds of things that we can evaluate and this is in the context obviously of collaborative initiatives, there's lots of things we can evaluate in the world. We've broken it up into a structure as the way I shouldn't say structure because it says structure on the slide but we've broken it up into kind of a typology of things that you could choose to evaluate within a partnership. There's lots of other things you could as well and there's lots of other ways to think about this but for the purposes of today we'll talk about structure of process and outcomes. And I'm going to talk about these in the context of what you could evaluate within those three buckets and then the ways in which we've seen different evaluators or partnerships start to imagine how they could evaluate them. So it won't be exhaustive but some ideas and things that we've seen along the way. And so we thought because we're talking with all of you you'll all be pretty used to developing kind of example evaluation questions or research questions that go into these interviews so we thought to structure it around some questions today. When we're in kind of the structure space within a collaborative initiative we're thinking about to what extent have you got the right voices, the necessary voices, the correct voices present within there so I hope you've got the right people there. How are they connected and how are you enabling those connections between them? So Cam was talking before about stronger deeper connections that might not be what you want but we're thinking about the nature and quality of those relationships. And then and then how they access and required resources so have they got the right structures and governance processes? Have they got the right backing behind them? Have they got the right platforms to meet on? Are they collecting right information? So some ideas around approaches that we've seen work really well. There's that kind of upfront stuff around stakeholder analysis systems mapping the kind of landscaping work that is really critical to establishing a collaborative initiative. And particularly in the context of a collaboration which you've got lots of actors in you know you want to figure out well who are these people? How are they all connected? Have we got the right ones? Are there more to add in? So that kind of that that beginning state but I would say that we also return to stakeholder analysis and systems mapping with partnerships at different stages of development because of course taking stock of who you've got and who you might have next is really important as well. There's that network analysis type work which is ongoing I think but also about something that exists you know there's this partnership and how the network has formed and what it's generating and how people are interacting and of course resource mapping too to understand what kind of things you might need to allocate to to better improve this this entity. So I'm going to spend a little bit of time on this process slide because this is where lots and lots of work goes into evaluating partnerships and if you've been involved in partnership evaluations in the past you might have seen some examples of process type evaluations that really kind of yeah link to understanding the processes and how people collaborate basically. So Cam mentioned some of those critical success factors before and often these are the things that we're thinking about or starting points anyway for thinking about process so a good collaboration will need to have a or many shared goals that guide the work of the collaboration and so it's important really to understand the process by which collaborators or actors within the collaboration arrive at that shared goal and have they arrived at it now are they continuing to go back to it and that leads to that second question there around how how they hold each other to account so the processes and the mechanisms by which they do that and also how effective that mutual accountability is we we see often with lots of partnerships that we talk about accountability but perhaps we don't hold each other to account very well wouldn't it be great if we used all of this learning and evaluation for decision making or for improving decision making processes I'm sure you can all relate to the times we don't see that happening but it can it evaluating how decisions are made within a collaboration whether they're inclusive or transparent representative drawing from different voices and values within the group is is really important and tells you a lot about the strength and I suppose the nature of the collaboration. We work with kind of Cambridge and G-PAP before and that's the one that's on my mind we work with these kind of entities who are working on lofty goals really how do we remove all of plastic in the ocean how do we stop it happening anymore and then there are changing circumstances that will that will occur over time so how does this collaborative initiative respond to those ex-brainiest environmental things so you know as someone actually come up with a policy to stop making plastic go into the ocean or have have different actors committed to different things so is the collaboration prepared to respond to those things doesn't do it well is it still placed well to kind of respond to its changing circumstances and that final question there about the overall health of the collaboration I mean there's there's lots of literature on this and if you're into collaboration and partnerships it's kind of fun to delve into I don't know if it is fun if it's just nerdy but it's the health of a collaboration is kind of a great thing you know you can see lots of these things exist because they have to because we're told that collaborating is the right way to do things but a really healthy collaboration has an amazing opportunity to contribute to contribute real good to the world I suppose so the health of the collaboration is important and that includes the ways in which people are interacting with one another the ways they feel about one another the ways they feel about the work that they're doing their commitment to etc um so there's different ways to do this I'm not gonna I'm going to huge amount of detail here but but they do that there's you might have seen things like partnership health assessments or collaborative tools for assessing your health I'm saying health twice but they all say health mostly and they're really good they're really useful tools for understanding things about structure and process of your collaborative initiative there's lots out there and they and they are really useful it says there on that third bullet point there that they're around those critical success factors you know do you have shared goals are you sharing resources authority and accountability and how are you engaging with the systems that you want to operate within or shift or change how's communication flowing information sharing are you adaptive are you responsive all of those kinds of things and they they're really interesting because they are the backbone of kind of collaborative work in terms of methodologically or tools and approaches they do rely heavily on survey or the gathering of kind of other feelings of feelings and expressions and experiences of of the actors within the collaboration and we have found them to be really valuable tools for for tracking as I said those critical success factors however and there's this and here's my linking in my linking point to my next slide few of these tools actually document the outcomes of the collaborative initiative so I'm going to try and draw this together with them with the examples came gave before but we were thinking about this in preparation for the for for this session with you today and there's I suppose we we've split it into two groups there's this kind of these individual outcomes and their shared outcomes so the individual outcome questions might be about how how has being part of this collaboration influenced me as I'm part of it or how has it influenced when I've done people within it and what benefits have those individual partners or collaborative collaborators gained from working together and in what ways has that contributed to the to the system of interest and that last question is really around that how does it collaborate to the systems that people are operating within and the changes in the world that this collaboration might be seeking to to do something about and so we were just thinking about some of the some of those particular outcomes that you might begin to assess I suppose or evaluate for members so there's individual level outcomes so it might they might relate to the to mission so have you has it increased your ability to directly or indirectly achieve your organizational mission so to give you an example within the GPAP example that can gave before actors come from the private public and civil society sectors and and often participation within a platform like that is is through an organizational lens or perspective so you know you might have a plastic manufacturer or or a government agency that's that's tasked with removing plastic and so you know if you if one of the outcomes could be that the members themselves are able to to change the organization's missions that they work in and and the organization as in the collaboration that's a really that's a really great thing to have and a great thing to kind of watch how it's happening and I suppose in an organizational level are they able then to leverage resources within the within the working environments that they operate in outside of the partnership and are they together able to leverage resources together so kind of both of those things at once and and this is the the last bullet point there is the thing that we see most about people's participation in collaborative entities you know they there's these always intangible things that that you know you want to get from being part of it but you're not really sure if you're going to get and if it's successful you get it because it fits what you want but anyway the things are those social political capital people join private public platforms because they want access to government or they want access to a company and vice versa that that's what they want and they both know that they're there for that that this opportunity for networking connecting to one another in deeper ways and in and I'm kind of shared issues together and that's a really important joining thing as you know if if you're like me you hate networking these are great ways to network around something that you're actually interested in um they obviously the increased legitimacy thing I think is really interesting because we see it in lots of these kind of complex problems but people join a platform where there's a whole lot of people interested I'm going to go back to the plastic one and then they're able to say actually there's always other people that are interested in plastic as well so there's some there's some good this is the legitimate thing to be interested in obviously reputational benefits market advantage influence and positioning the influence positioning one is a double-edged sword of course because sometimes people are authentically committed to being part of platforms and sometimes they're there for the influence and positioning power that it may give them within their organizational entities um I think these are interesting too we kind of we came up with these as um as a I suppose a typology what are some of those shared outcomes that we see coming from these partnerships um that this is not exhaustive but these are some of the things that we see so that they try to influence policy you know they try to advocate and lobby and softly influence and talk about and shift policy um the collaborative entity I mean when I say they and they have benefits for practice so the the knowledge development and exchange hub that Cam was mentioning for child youth mental health you know one of its benefits is that it's a place that shares practice around what works what isn't working how could it work better in that space and so procedures guidelines routines things that things that could actually improve the practice within that space um hopefully and I think this is this is true we see we see different kind of resources um uh committed to um uh to working on these grand endeavors together and I mean like really tangible things like we see funders at the table and committing um you know large amounts of money to try and solve these complex problems and we also see people who wouldn't usually be there um at the table committing their time and energy and those resources are really important let me talk about relationships before but it can't be a bad thing to have better better collaboration and better cross working between disparate groups and we say that as a benefit um as an outcome for the world probably the I mean this is something that we're really passionate about but this dismantling of power this this idea that you can use um a collaborative entity to bring in unheard voices um that that aren't usually there and give them decision making uh power and authority and responsibility and um you know being in the room where it happened that was that a quote that's a that's the terrible sorry Cam went saw Hamilton last week and so I put it in my head but it you know get people in the room where it happens and I suppose um also it's it is about shifting attitudes values and beliefs you know being part of something and hearing from others that have different view few points to you um arriving at a kind of shared goal outcome ambition is a really important thing uh to get us all kind of thinking perhaps not on the same page but challenging one another and um and challenging our own achieve values and beliefs okay does the whistle stop to a between well of all of all accommodation of work that we've done for years but I thought I'd drill it down into some suggestions and recommendations that we have found useful when approaching learning and evaluation in the collaborative space um here's one we prepared earlier and and it is something we prepared earlier and something we've put a lot of time into it's a distillation of of our work I suppose around what we've what we've started calling these five ingredients for learning and evaluation uh within collaboration so Cam mentioned that before the first thing we find is critical is to set a clear and shared vision that's the shared vision for the platform and partnership and it's a shared vision for uh the learning and evaluation that wants that that needs to happen uh within that space so that might be in the form of about of an evaluation plan or any other um terminology to it but it is it has to include both a shared and clear vision for the platform and a shared and clear vision for collecting information and for evaluating progress so then then there is something really tangible about a data collection plan and when you've got a whole lot of people coming from lots of different places preparation and planning for this we see is really critical you miss lots of intangible benefits when you don't collect information from those conversations that are happening in hallways or whatever although we're all in zoom still so maybe one soon in blackout rooms um developers for sharing reporting communicating what's happening so how are we going to do it where are we going to get it from who's who's it going to all of those kinds of supposed communications knowledge sharing those those important aspects need to be decided up front there's something that we find really uh um what's the word like a really great way in around a green indicators so um often collaborations aren't very good at talking about what it is that they do and what they achieve as collaborations so indicators that are about collaboration uh rather than about the big lofty goals are really important to arrive at in addition to those big lofty goals so that you can track the health and instruction process of your collaboration and finally and I don't really know if we've ever nailed this completely but there needs to either exist a culture of inquiry or there needs to be a group of people that are interested in building a culture inquiry because learning is about wanting to find something out to be inquisitive to um to learn about what we don't know yet and that's a really important backbone of these um these evaluation approaches I suppose for collaborations so you could you can summarize it or I can summarize it for you but um that I suppose four things we'd hope you would take away from today so there's there's something for us that learning evaluation is critical to these things so it's one thing just to say yep go ahead and and create partnerships it's the only way to do work and we must all work together um but I think what we would say is that you if you're going to do that also learn about why it's working for you why is it important find out find out the things you didn't know before and share them with others because collaborative initiatives take energy to take work um and they and there's a lot of people in them too so you've got to kind of commit to a structure process outcomes all relevant as I hopefully have outlined to um and and having a focus on all three is really important from the outset not just outcomes particularly in a collaborative um entity like this or like these that we're talking about um don't start with tools and approaches uh that would be if I could if I could say that to any to anyone who works in these these spaces that don't start with the way you're going to do it start with the other questions you know arrive at the tools and approaches and methods last um and finally I said this before this culture of inquiry try and instill it you know what are we going to learn from how how can we change our factors how can we actually make um make change or impact on some of these huge thorny complex issues I think that's it if we said half an hour that was that was not half an hour that was that was longer but um thank you all for your patience I'm going to stop talking and have questions thanks Nick and yeah thanks Cam that's just um there's so much there that's so fascinating and um yeah I'm not surprised you talked longer as well because you as you said it was was a whistle stop tour there's so much there to cover and you've also gone beyond I think um evaluation too so um some questions coming up through here in the chat now um I mean uh Christopher would you like to talk to your question uh about negative consequences which is yeah quite interesting I'm happy to ask the question as well hello can you hear me yes we can yeah thank you so much Nick Cameron that was fantastic um I I'm really interested you know as you've been on this journey with a few different organizations have you seen some common themes or lessons learned about when is it perhaps not ideal to collaborate um some of the lessons learned are the negative consequences of collaboration really interested in your thoughts or should we always just collaborate I've got one to mine cam from today's conversation and I'm sure you'll have others that the main one that comes to mind for me Christopher is about collaboration being done to people so an example is a funder insist that you must collaborate in order to get the money but you're actually all working on the same thing where you were going to work on separate things but now you have to work together competing for the same fund and sometimes collaboration has negative consequences when it brings people together that should be genuinely not sharing resources or having to share resources um and a force into a space where they where they must do that in order to to appease someone else so yeah I see some resource sharing when it's not done from for the right reasons is a huge negative consequence and and and I suppose just to add to that and everybody on this call will have found themselves in working in a collaboration which wasn't very good it didn't feel very good to be part of it um there are multiple consequences come from that that come from being in difficult relationships aren't they you know like it it erodes trust um it removes our ability to to find a common pathway forward it removes our ability to actually be willing uh to to share our resources in ways that allow us to do something together effectively so collaborations when done badly end up in um a lack of a lack of an ability to partner effectively and then we don't get to realise the benefits of doing that so as as as good as they can be as as bad as they can be as well I think the other thing just to add to that as well is that sometimes there's this um we we had sort of a very lockstep perception about what we will get from collaboration you know we've got a set of indicators or a set of outcomes or a set of things you want to see change in the world one of the things that I think we find most importantly um about these types of entities is that things come from them that are a bit unexpected and so being open to um those unanticipated effects that can come from collaboration with a good bit like that is really important to hold in our hands as we're approaching the work that they do oh that's excellent thank you very much okay lots more questions and a lot of really affirming comments coming through the chat so um probably we'll focus more on some of the questions coming through um Angela you have a question here about um indicators um and data did you want to talk to that just earlier you mentioned um that mainly the data for evaluation will come from surveys or talking to people that are part of these collaborations um so I guess in our way the nature of the data is perception based so it's based on perceptions that people in these collaborations have about the processes but also the outcomes and I guess then obviously people have different levels different perceptions and and they will um you know consider say an outcome different or um yeah we'll perceive it in a different way that others will so I guess then that has the question of how can you arrive at an objective measure for outcomes I guess in my work that's a question I always sort of have to tackle with good question Angela um I remember the bit that I was talking about I was talking about in the context survey in the context of collaborative health assessments which are are tools that have been developed and are largely survey tools so just to clarify that's what it says that those and those things exist and they've and they've been kind of tested I suppose and and and made into something that people can use so and they're really good for understanding as you say the perceptions of people within the collaboration how those processes and structures are working for them um when we get to outcomes we use all different kinds of sources um and data sources I suppose uh depends depends what it is that you're looking for as an outcome but um but certainly don't just rely on survey I mean um yeah I mean I don't know really how to answer your question without a particular um focus on a on a collaboration or or a particular set of outcomes but depending on the outcomes that people are seeking we use a range of different methods and we would see a range of different methods used by the collaborations to understand what they're doing as well okay another it's a really interesting I guess meta question here from Rachel um yeah did you want to talk to this Rachel I think you probably got better justice than that me reading it out uh sure happy to um I'm I'm interested in evaluation of collaborations that are actually an intervention in and of themselves I just wonder if you might be able to speak to the subject of what types of approaches you might use if they are indeed different to the ones that you've covered um for tackling evaluation of collaborations that are an intervention could you um sorry just maybe it's my naivety um Rachel but I'm okay could you describe what a what a collaboration as an intervention might be um there are um there are interventions um there are collaborations that are that are created in order to solve problems so um a problem exists and the solution might be to actually get people to work together better and share resources um that that would be just a high level description of the kinds of things I'm referring to well my colleague uh Dr Willis uh has been writing about this I'm going to throw that to you camera because of this meta question is that we've been trying to play with recently I think that there's it's a really good question and it's kind of viewing in some ways the viewing the the collaboration as as the end goal in itself um and so has this collaboration actually been able to um uh bring folks together that it intended to do has it been able to forge strong and meaningful relationships between these entities that are part of the collaboration what is it that these that these folks are now exchanging through their relationships that perhaps they wouldn't have exchanged before has it enabled them to build trust and new relationships with those that they were seeking for those that the collaboration is intending to seek so in many ways if those I think it's if those are the intentions of what the intervention was seeking to do then I think it's it's sort of it's on us as evaluators and learners to do explore has it actually achieved those things I think the other thing to add to that and so then some of them the methods you you can you can imagine the kinds of approaches you'd use to to capture some of that information sometimes though it's it's through those relationships then perhaps other things are expected to come out of it maybe for example it's through those new connections that have been formed that um two people that maybe hadn't met each other before might go off and do a project together perhaps they wouldn't have been able to do that otherwise and then might form might be new spin-off initiative that comes as a result of their relationship that they've formed to a particular collaboration so sometimes it's important to understand the nature of the relationship if indeed that's been the thing that the collaboration has been trying to do absolutely understand that and sometimes those relationships have either explicit or implicit intention to do something off the back of their formation and so it would be important to try to capture those as well. Helen you had your hand up earlier do you have a question? I have um and then I started coughing and I had to take my hand um my question relates to uh it's a kind of um it sounds a bit slightly frivolous but when I was thinking about evaluating outcomes and I was thinking about collaborations that I've worked in when the collaboration was terrific and the actual achievement of a tangible outcome at the end was zero um everyone very happy with the collaboration but making very little difference to the issue at hand and I guess um my question is and would be for my future the projects that I'm involved with at the moment is linking those two so I think there is an assumption that the collaboration in and of itself is a good thing um and should ensure in theory produce some good outcomes but there is not necessarily in my experience a link between the collaboration working quite well and the outcome which is unmoved might be just too big too wicked too um you know too many variables I'm not sure about yeah as you said I mean that that is something that obviously that's hopefully what we hopefully started to get across today as well as like we see a lot of effort being put into the the health of the partnership and the way that people are working together and and as I said before there's lots of really good tools for doing that as well um I I see what you mean about I've got some examples of partnerships that are really healthy that don't do anything in the world as well but um but I think that where I would where I would um kind of connect the two of outcomes and the health of the partnership is um something that we've seen is that they are they are the foundations or critical success factors of achieving outcomes so I'm trying to be clear about it if you've got all of those critical success factors in place if people are in a healthy partnership then they are more likely to achieve those outcomes and I I do think there is there is merit in and in looking at the health of the partnership or the ways in which people are working rather than just focusing on those outcomes because at the same time as you know you have partnerships that are really healthy that achieve no outcomes you have partnerships that are really unhealthy that are achieving the wrong outcomes or unintended outcomes so um yeah I would say that like having maybe sort of a separate system but a way of appreciating the foundational element of those um those partnership qualities I guess um yeah I think that's out of it negative no no no it wasn't particularly negative in the sense that by the time we had all finished looking at the issue we realized we had not understand the enormity of the issue and and therefore we had actually we had actually got more knowledge than we originally understood um and so our I think that whole thing about curiosity and learning is fantastic I mean we we were all incredibly daunted by the end of it because we kind of went shit we didn't understand this at all and now I think we're going to recalibrate this and we have to bring to be other things on this and it's not just us so it was it was it was a good process um and I think yeah just we would take back to the drawing board so it was actually quite positive but um I don't I think unless someone had pulled that together for the group they would have just continued to have to feel good about the collaboration achieved by much and there's something as well just listening to you talk there can be this mismatch out there between some of our expectations of what we want a collaboration to deliver or what we're told that it will deliver and then what it actually does um and and so you know I think we see that all the all the time um and what we do what we have a tendency to do is actually to miss the value that a collaboration has actually generated so when you say it was a really good collaboration I hear one of the things I hear in that it was it was really nice to be part of the collaboration so it was it was friendly it was welcoming it's respectful there was good relationships and good exchanges between people that that's a nice thing but yet what it actually results in as you just said then um you know we got new knowledge we got new insights we got new um information about the problem that we were dealing with that doesn't sound like it was necessarily the intention at the start of what the collaboration was thinking to do and yet it's come from that and sometimes our really rigid evaluations structures and approaches don't necessarily allow us to capture that difference that gets made so holding space that we can capture that um is critically important yeah that's fascinating Cameron um it reminds me I couldn't help thinking that that quote that evaluation grew up in the projects and um so many of the approaches that we use it's just not all that well um suited to um what you're talking about and it's probably rude this is that's a whole other discussion um Jane you've got your hand up there um and you've been active in the chat too so yes far away hi yes um my question ties in with a question that Claire Hanley also asked which relates to complex programs where you've got multiple components so um in Claire's case it was a program 40 sub projects and nine funded agency agencies in my context it's um a single program with 20 funded agencies are all doing different things uh with some sort of nebulous expectation that collaboration partnerships are going to be a part of their funded activities but um and this is a program that has happened and so our task is evaluate has given that that at the inception or the guidelines for this program with such a sort of a nebulous expectation that somehow partnerships are a part of what gets done but how do you you know have you had experiences um with that and I think Claire's was more the form of you know the start of it I've started that process whereas I'm more at the other end going okay you know how do we sort of evaluate something where there was never any sort of like with respects to partnerships there was never any rule sort of like we want you to do a part we want you to form partnerships uh to help you deliver your funded activity so that was never explicitly stated yeah I think there's lots of examples of what you're describing um one of the things that's coming to my mind is sometimes we talk about like you know a multi site multi agency multi project um thing and then and they call it collaboration because it's you know under one funding or under seven funders or whatever it is and I think there's a difference between um between collaboration as a grouping so um you know there's lots of lots of things together that fall under one um a new theme versus um are these parties expected to actually intersect and work together so we're just because they're working on the same thing uh you know and they're and they're funded by the same agency I would say isn't always the best sign of them being a collaboration I'm thinking and I've definitely seen some of these where they where it's that I was like and can you tell us how our collaborative funding worked which is also a different question isn't it's how did how did how did the funding um collaborate how did the funders collaborate but I suppose in answer to your question when you see those things um and and whatever structure they come in I think when applying some of the things we've talked about today and again not exhaustive it's about during those boundaries of what what are people how are people collaborating what are they collaborating about um why are they collaborating are they even collaborating in the first place or was it just the word being used um and I think yeah I think those are probably good starting places because and I'm just the projects that are coming to mind for me that we've worked on uh the ones that also want to know about the 40 sub projects and um and all of the nine and what each nine agencies have put into um to put into this and what they got out of it so there's a kind of as Stuart was saying before there's kind of like a project-based evaluation within a collaboration evaluation that's um uh that can really confuse matters as well so yeah I think making sure that you focus on the collaboration as well if there was a collaboration battle there's there's also just in your question Jane it just maybe think about this um this offhand way in which many of us some of us um have seen collaborations and partnerships referred to you know like it and and in what it does is it kind of um it devalues the amount of time and effort and energy um of what it takes to go into uh building and working in its successful collaborative structures be they partnership through a lot of the word is but the the it's almost a throwaway line that of course people will fall partnerships and there'll be collaborations and etc etc and yet the amount of effort it takes to do that well uh is is really significant so there's just something for me about and maybe it's something we can all do at our practice as well is that if there is an intention to to build and broker partnerships or partnerships or something that we want some initiatives to to engage that we actually elevate that and make that visible and recognise um in an authentic way what it takes to to to work in collaboration and to make sure that we're capturing investments that people are making in that collaborative work because it can be significant yeah I must I must say that's actually what you've just talked about there's been a reflection of must uh like a reflection that I've had as I've um gone over the data and and the um that absolutely uh developing partnership creation or development you know is is is a project in its own right let alone when you conflate it with a broader programs um goals and somehow partnership creation and development sort of subsumed under other goals and just there's a social that this just happens and it doesn't just happen and much effort and time and resourcing needs to be put in by the organizations and a lot of these organizations are very small organizations yeah and and Jen you're I'm reading I'm reading your question now alongside your comments there and I'm reminded of and I think I said it a little way before but funders often um you know set up like a collaborative scheme or a collaborative funding scheme or they they aim to fund collaboration and as you say people as a resource um scarce aren't they so there's there's an opportunisticism uh to these people combining efforts and putting in a bid and then they've got a program that's suddenly about collaboration and then at the end of it they say how'd the collaboration go it's oh no I was just trying to get the money um and I think we we see we see that a lot I imagine lots of us do as evaluators um I think the owner then is as you said Jane back on the back on the funder did they create space and opportunity for people to collaborate within their funding scheme and if not why not um that could be part of form part of the evaluation I think at least part of the feedback to them absolutely thank you if anyone else like to um talk to some of the questions that have gone into the chat I I am interested in Melanie's question about um any recommended insights or approaches for evaluating cross-cultural collaborations I'm just trying to good Melanie could you would you mind saying I'm just trying to find it in the chat yeah there's a lot of questions there hi guys thanks for the talk yeah um so I work with um uh into in supporting a lot of um collaborative I guess um partnerships and committees and working groups between um traditional owner groups and um their partners so that might be government agencies um or other um other organizations that they'll work with um so a lot of the time there's you know big cultural differences in in what a successful um partnership will look like and and I think whilst there is um there's always respect between um people involved in the collaboration their ideas of what a successful collaboration is might be different and or or have a you know different cultural lens and I'm just wondering if you've worked with that or how you approach it to recognize those cultural differences yeah I mean I don't think there's one answer to this of course no it's a really complex um complex issue and also probably it will never be found out because it's about different people from different cultures in different spaces um but I was reflecting on your question and reading it I think one of the things that we've experienced um has been a necessity within cross-cultural collaborations to really define um the roles of actors within the collaboration and and the example I give is where often when I see from the outside cross-cultural collaborations or partnerships what I'm seeing is one party consulting another or or or drawing the resources of another and your example of traditional owners is um is is one where I see that happening you know there's a there's a as an agency or there's a group of agencies the money government whatever and they want to collaborate with traditional owners but what they really want to do is is is draw information from them consult with them bring information out of them and I think um there's you know there's a history to that of course and not necessarily an okay one but um but and but there's still a place for consultations where I was going with that but if it really is a collaboration between um cultures then the framework by which people are going to collaborate needs to be um designed together it needs to be something that that is that is appropriate and suits one another so I'd love to say we've been really successful in this I don't think we have um we've certainly tried but I think there's a there's a long way to go to defining the roles and goals that people come to collaborations with and how they differ from the way we used to do cross-cultural collaboration um we mentioned the notion of shifting power within these dynamics before and I think that's probably the best starting point um yeah I don't know hopefully that's some way of an answer do you need to add camp oh I think just to it's a really good question Melanie I think there's um there's something about from the lens of of an available of evaluation which is important and obviously to this group uh that's on call today evaluation is critical so you mentioned different expectations about what it might mean to work within this collaboration and that'll have effects on the kinds of things that we can choose to evaluate and then methods and tools and approaches by which evaluates might go about doing that but then as you say as well it actually is fundamental for the the formation of the collaboration in itself um what are our shared different expectations about what this collaboration is going to do and as you mentioned are these are these built on foundations of um collaboration are they built on foundations of partnership they're built on consultation in different ways and so being clear I think an understanding those different expectations and different starting points of whoever might be collaborating both in the context of forming the collaboration itself as well as for its evaluations critical and I just sort of Kate what you put up in the the chat there as well I found that there would be a really useful resources resource as well yeah I would probably recommend it okay there's a few other comments and questions there just about the competitive aspect to um collaborations um there is and one other about um conflict resolution as well so again dealing with some of those unintended consequences um yeah I I want to just open it really I think is there anything in particular there that strikes um unique and cam that's coming out of these discussions these questions now um that's particularly kind of interesting or surprising for you I think there's something there's something in there around I'm just looking at um Angela your question about the ability of partnerships to resolve conflict um and to to to deal effectively with tension and next laughing because you know that I'm conflict diverse uh and so this is something that that I um that I struggle I struggle with um one of the things that we know uh is is really unhealthy for partnerships um uh is smoothing uh so it's the the papering over the cracks and not talking about the things that we're disagreeing on and not talking about how we are different and not talking about the the elephant in the room whatever and we've seen lots of partnerships and lots of collaborations that fall into that trap where uh the things that that there's there's clearly a tension there's clearly something that's not been spoken about but maybe out of politeness it never gets resolved and and that is is such an insidious and healthy thing for any partnership to endure and and often maybe not always will result in the partnership actually failing to generate the results that that will seem to do so the the ability of um of a partnership or a collaboration to actually hold states for conflict and tension to be surfaced in a healthy way for it to be spoken about um for it to be okay for disagreed different perspectives to exist that's a critical uh success factor for for a partnership so I think it's a great question or a great comment that you raise about how can we understand uh and evidence the ability uh of partnerships to be able to do that I think it's a really good point yeah I want to build in this um and Angela both your questions about doing competitive behaviors I was thinking I mean I was laughing it came to conflict diversity but I think one of the one of the things that is a great benefit to being an evaluator or an external evaluator within a collaborative entity is the opportunity and ability to um to hold a different space and so I can think of many times where I have been um uh the target of conflict perhaps so where it's and it's got nothing to do with me it's about the partners having a conflict and they directed at the external person but what's really useful about that if you're okay with it is that the um is that the partners then get to agree air those grievances and then you as an evaluator can kind of get to the nub of it or try to sometimes it works um you know is there shared space here are you actually focused on the same question would you would you be interested in finding out the answer to this um I mean it's a bit of a bleeding between like uh improving the partnership and um and and doing the evaluation but I think sometimes those taking a learning and evaluation lens to those kind of conflict discussions can be really good because they're opportunities for people to come together it's a very funny examples I don't think any of them are allowed to be shared but um but I mean conflict can sometimes be good in these spaces as well there's a bit of storming and norming when you get a group of people together who are there for different reasons and different things the other I'm just looking at some of the other comments do it in the the comment with the chat function and there's there's a couple of comments there about learning uh and knowledge being um an outcome that may be getting the ultimate outcome for some collaborative initiatives and I think that that's um in in many of the collaborations that we find ourselves working with that is actually a key goal uh either the generation of knowledge is sharing the knowledge but the uptake of that knowledge is people using that knowledge and changing the way that they're thinking and increasing their knowledge about a particular field or concept or topic so I think that that's um that's a critical outcome a critical benefit that we see from many of these collaborations particularly for the ones that are bringing you together entities or organizations that come from very different worlds and have got very different knowledge bases or ways of knowing the world so being able to share that in a way that enables others to understand I don't think is to be skipped over or diminished is being a key contribution of many of these partnerships we work with yeah I was just reflecting on some of these questions I know that we're closing out but there's something about if you can as an evaluator help a collaboration whether it be the secretariat functions that people kind of bringing everyone together all the collaborative members themselves if you can get them to focus on um of what they're doing as as one starting point of one way in and getting to appreciate that unique value that's that is coming out of collaborating in cancer this before but get lots of talk about members asking so what hang on have we started achieving these lofty outcomes are we gonna are we making progress towards these big things and that's really important but if you can get them also to appreciate like I had a really great conversation coming together with someone my mind has been changed because of because of the thing that you said I talked about the knowledge then we're generating knowledge together that didn't exist before we're intersecting and colliding and those things are really important they get they're left on set a lot of the time and if you can spend energy on capturing them on sharing them between people I think that's a really important thing really important thing in collaboration