 not only about evaluation, so this is about the intersection between design and evaluation. And so what do I mean by design? I guess one thing to note is that we're not talking about designing and evaluation. I know that's obvious, but I haven't had my view, we've got a little bit confused. So we're not talking about designing evaluation, we're actually talking about designing a program or a service or even an organization. So it's how evaluators can help actually design the substantial thing. And so there's a really a big sort of increase in the amount of interest in design or social innovation in Australia and in the world and there's been more and more of it going on. So it's a really exciting time for us as evaluators to understand how we interface with this other discipline and how they come together. And what we've seen, what Matt and I have seen are three main ways that evaluators are working with designers. And one of them is simply that there's some fantastic tools in design and co-design and helping people to understand how to really work out what the what the user needs are and design things that delight the user. And a lot of our M&E systems are measurement and evaluation systems or M&E. They don't always delight users so we can learn use these fantastic tools to help build better evaluation systems. So that's one use. The second use though is that we get involved in helping designers, evaluators get involved in helping designers build things by bringing all our tools and our skills of evaluation earlier on up front into the design process. Some people call it developmental evaluation but it can just be seen as evaluators working alongside designers. And the third way we're seeing it is some evaluators are becoming designers, they're actually abandoning their profession and becoming designers. So that's happening too, maybe not as much, but they're the three ways. So some people are learning how to do design skills or doing both. And some people on this call I know will already be in that camp who already play both roles. But they're the three roles that we've been seeing and that's what this special interest group is all about. It's about the intersection of design and evaluation. So that's just a little bit about what we mean. Matt, would you like to tell everybody a little bit about the special interest group? Yeah, absolutely. And like all good evaluators, I have some slides. So the one thing I did just want to point people to is the learning spring hashtag there in the top left corner, so DE 2020. So if you're the sort of person that likes to live tweet commentary and various things by all means get involved. What I just want to quickly comment on before we dive into the provocations was a little bit more about the special interest group itself and some of the things that we kind of do day to day. So one of those things, welcome. There you go. There's the welcome that boy. So one of the things that we do is we try to bring lots of different groups together. So, you know, not just evaluators but also designers. So some of the people speaking today don't come from the sort of evaluation heavy background. So we like to bring people together to explore this intersection, which is I think potentially a bit of a unique aspect to a special interest group compared to maybe some others. We also hold events like this. So previously we've also been involved in the 2018 AES conference. We had a bit of a voice there. The other thing that we like to do is we like to maintain a space to share thoughts, musings, links to resources and things like that, which I'll speak to in a second. And the other thing that Jess and I particularly conscious of is making sure that we respond to the interests of those who attend. So wanting to be really responsive to your interests as members or attendees of these sorts of events as well. So while we talk about membership, membership is kind of a loose thing because, you know, you don't have to be a member to attend something like this. But we do maintain a LinkedIn group. So you can see the name of it there, AES Design and Evaluation SIG. And we also have an AES email for the SIG as well. So if you've got queries about anything, you can send us an email there. But in particular, what we're trying to do through this sprint this week is trying to sort of identify where there are those resources or tools or things that you might know of that maybe others don't. And so wanting to try to share those a bit more widely. So if you're already a member of the SIG, then you can post them in there straight away. If you're not, you can request to join and we'll add you in and you can post things. Or if you don't use LinkedIn, or you just want to let us know that something exists, you can just send it to us by email and we can share it with others as well. So we're trying to be really collaborative with this kind of effort. The other thing that I just want to point out is that at the end of the sprint, we're going to practice what we preach. And so we want to reflect on this experience and improve. So on Friday, we'll send around a survey to everyone that registered to basically understand what worked or didn't. And what do you want to see more of in 2021 as we move into a new year and new opportunities as well. So that's all I wanted to just comment on. Jess, I'll just hand back to you for the introductions. Thanks. And I just wanted to say, I'm sorry if I didn't say already that we are recording this session. So if you don't want your face on it, then turn your camera off. But currently, we won't be seeing everybody, but that's something to note. And we will make it available after this. So something to note. Okay. I think that's all the all the stuff done, Matt. We might go straight into the provocations. So there are three provocations. We've got three speakers, and they're each going to speak for about 10 minutes. So I'll just introduce the panel to you. And then without a further ado, we'll go straight into it. And I've got a little bit of a cheeky question for each panelist as well, just so we can get to know them a little bit. So first of all, I'd like to introduce Ingrid Burkett. So Ingrid is a fantastic social innovator that I've known for a while, very creative. She's the first time I ever saw anybody develop a theory of change with foam cubes. Do you remember that Ingrid? Yes. So three dimensional theory of change was my strongest memory of working with you, Ingrid. And Ingrid is a professor and co-director of the UNIS Center at Griffith University. She's a social designer, designing process, products and knowledge that deepen social impact and facilitate social innovation. She's also a bit of an artist. But my question for you, Ingrid, is when you were a kid, what did you tell people you wanted to be when you grow up? So thank you, Jess. It's lovely to be here. And I'm joining you from the land of Yggenbeur and Yagura up in Logan in Queensland. And to answer your question, I always wanted to be a private investigator or a detective. And my family is convinced that I actually achieved my goal because as a social designer, I get to look into lives and try and make connections and investigate things that maybe wouldn't be obvious to everyone immediately. So I think I've made it. Well done, Ingrid. And it's a bit like being an evaluator as well, isn't it? Being a private investigator. Yes, absolutely. Now I did have some slides, but it doesn't seem to want to let me share those slides. So do you want me to just go into my provocation or are you going to question me, Jess? Well, I'm just going to introduce Shani and Chris first. And then you've got time to work out how to share your slides. And maybe we'll make sure, Michelle, that we can share slides in a few minutes. Second panelist is the amazing Chris Vanstone, who's the Chief Innovation Officer at the Australian Centre for Social Innovation. And I've had the pleasure of working with Chris a few times. We've worked together as designer and evaluator a number of times. What can I tell you about Chris? Well, he designed biscuits at one point. That's not in your bio, but this is trying to find a fun fact about Chris. He designed biscuits at one time, but also was the founding member of both the UK Design Council's red unit and the participle whose experiments using design to tackle social policy problems are chronicled in a book called Radical Help by Hilary Cotter, amazing book if you want to look at that. Would you tell us a bit about you, Chris, but importantly, same question, when you were a kid, what did you used to tell people you wanted to be when you grew up? Well, apparently, I used to want to be a human doctor in the morning and an animal doctor in the afternoon, both careers to which I'd be brilliantly ill-suited. Which you think you would have done better at animals or humans? I think there are probably a few reliable claims from the animals. But thank you, Chris. I'm sorry I didn't make it to be what you set out to be, but I think you made a fantastic contribution. Okay, and last but certainly not least, Shani Rajendra, senior consultant from ClearEyes and Consulting, so works alongside myself. And Shani, what can I tell you about Shani? Well, Shani is born in Sri Lanka and raised cross-culturally and brings a keen awareness of power and cultural relativity to her design and evaluation. Been working both with design and evaluation in steps in both worlds. And also, on the side, you're the strategy leader at Moral Fairground as well. Is that right, Shani? Yeah. So, same question, you know it's coming. What did you used to tell people you wanted to be when you grew up? Thanks for having me. I'm sitting on the line of the Roanju people, the co-ordination, so I'd just like to take a moment to pay my respect to their elders past, present and emerging. What do I want to be when I was growing up? So, for the life of me, I couldn't remember this one, so I took it to my mother, the go-to source for all things childhood. According to my mom, I used to want to be a teacher. How interested in people's learning journeys and helping educate or draw up people's awareness to what I felt was important, as I'm guessing a six-year-old. So, there you go. Well, there you go. Well, you certainly been, I think you are a bit of a natural teacher, so you may be doing it anyway, what do you reckon? Yeah. Maybe. Okay. Well, thanks so much. That was a brief introduction to our speakers, and now we're going to give them a chance to talk for 10 minutes. And we've encouraged them to be provocative, so we'll see what happens. One thing to note is I really welcome you all to leave, put little chats in their chat function, comments, reactions as they talk, and that'll be a great way to participate as we go. But we're now going to try, Ingrid, to share the screen. Let's see if we can do it. And if not, just talk. And the floor is yours. You have full permission, Ingrid. There's no permissions taken away, so you should be able to do it. Yeah, it tells me I have to close down Zoom before I can do it. So, I might just talk. I have sent my slides to Jess and Matt, but they're quite big, so I'm not sure whether you'll get them. But I can just talk, and I can send people slides afterwards. I guess my provocation, and it's mostly a provocation to myself, as someone who's, I guess, increasingly spanning the design world and the evaluation world in the space of systems change. And so my provocation is to be more humble. And it's a work in progress, and there's two sub-questions. One is, how powerful are we really in changing systems? These are questions to myself and all my fellow designers. And then the second one is, is complexity actually more important than systems thinking? And I'm saying this to myself as someone who, before this job was actually employed as a director of systems innovation. So I'm questioning myself, my own design work in the space of systems. And I guess I've got to that position because I've increasingly seen that systems thinking is something that's into design in a really big way. And as I see it, so much of the systems thinking is plagued by what I think is actually too much certainty. So we've got things like map the system as if there was a singular system, or we could map it. Find the levers that we can pull in order to create that magical change, move the needle, shift the dial, all of these activities that are really plagued by, I think, far too much certainty. So as I've been traveling this world of social innovation for the last, I'm not going to actually admit how long, because that would give away how old I am. Most of the work that I've been involved in is in this complex systems realm, where actually learning through experiments is the only way forward. And it's critical. And outcomes, as far as this work goes, are really only evident in retrospect. So evaluation is actually retrospectively looking. But in the case of really complex systems, the retrospect needs to be actually quite a long way in the distance. And for a long time, I didn't quite understand the magnitude of that. And it was only when I really started digging around in complexity thinking and complex systems that it made sense. I'm just going to read you a quote that would have been on my slides. But it's from Dave Snowden, who's a very grumpy old man. But he does have some really interesting things to say about complex systems. And a quote that really stuck with me was when working with complexity, we can understand why things happen only in retrospect. Instructive patterns can emerge if leaders conduct experiments that are safe to fail. That's why instead of attempting to impose a course of action, leaders must patiently allow the path forward to reveal itself. They need to probe first, then sense, then respond. And to me, that has become a little bit of a mantra in the way that I approach systems change and the struggles that I continue to have with it. So one of the things that we do at the UNIS Center is to have a look at how we can achieve really big systems challenges. Matt, you're a you're a miracle worker. So I'm up to the next slide. Thank you. So one of the things that we do at the UNIS Center is have a look at how we can approach really big challenges like the SDGs and set them up as missions with lots of cross sector stakeholders, lots of impact projects, lots of experiments that all have a theory of change for what we now call an impact map. And in doing this, I have an internal struggle because I know that we are aiming for systems level change, but that we're not going to be able to see that systems level change unless we experiment a lot at that bottom level, the impact project. So we need to be able to zoom in to that level at the project level, experiment really rapidly and learn and get a sense. And I'll come back to this because that sense making and the strength of that sense I think is critical if we're really going to make systems change. Thanks, Matt, on to the next slide. Just to give you a bit of a sense of what I mean when I say I'm working in complex areas. A lot of the work that we're doing here is related to employment and employment might sound like it's a quite a simple area, but actually it's part of multiple systems and a lot of the people that we work with are people who've been long term unemployed. And so I've faced in the last few years just a constant struggle that my funders and also the people that I work with, they're very much into solid outcomes and answers. And so oftentimes the outcome that I'm told that people want is a job. And yet I know from spending lots of time with job seekers that they've got all of these complex things that they need to navigate, especially if they've been out of work for a very long time. Okay, next slide please. So what often happens is that it pushes us to start to pursue transactional outcomes. So as a designer, I'm often faced with things like this, you know, coming into a designer program which is supposed to be, which is funded to be an employment program for people who've been disengaged for over five years. And the board member says to me, I want to see this program achieve real outcomes. And at the end of the day, it's only jobs that count. And so, you know, we're pushed into designing things that actually are going to lead to transactional outcomes, person plus job equals outcome. Next please. So what we know also from work, exploring what the actual outcomes are for people is that that turns them into cyclical, precariously employed people, underemployed people. So people cycle between unemployment, precarious employment and underemployment. And figures that are starting to come out from most employment programs now are saying that between 40 and 60% of long-term unemployed people, when they enter these services, end up churning back through them within a 12-month period. Next please. What we'd like to do is try and design for transformational outcomes. So what we know from the signals is that that means we need to focus on quality jobs with living wages and pathways to address other in outcomes. Next please. And we also know that this quality is actually good for everyone. So it's good for the employers as well, because spending time with employers leads us to realise that their costs of constantly having to re-recruit people when the people that they have employed leave actually costs them in productivity and also financial costs. Yes please, Matt. So in my attempt to try and become much more humble in what we're designing for and what we're evaluating, I've come to think about how might we aim for right outputs. And I've become actually much more fascinated by outputs that will then lead us to right size outcomes. So I've moved right away from thinking that I can design straightaway for outcomes and much more interested in how to create the right outputs. What I'm also interested in is how I actually end up convincing funders and boards and stakeholders that aiming for outputs and smaller outcomes like increasing employability rather than heading for a job is more likely to lead to quality overall in outcomes and it's actually more honest and I can be much more honest with the people I'm working with if I focus on things like employability rather than jobs. Next please. I have to tell you Ingrid your 10 minutes is up. Okay. I'll be one second. So even employability though can be really hard to measure. So this is pre-release a little figure of how we're working with a social enterprise that's focused on employing really long-term disadvantaged job seekers. And what we're doing here is mapping the enterprise to see where in the enterprise they actually can spend time and what kinds of employability skills and then those the people who are actually working in there have helped us to design technology that can track people so that we can see what's going on and last but not least the last slide is what that's leading me to think about is how can we look for signals rather than silver bullets. So having outputs at experimental level and then clustering those outputs from within and across the experiments to give a sense of what the signal strength is about how likely those clusters of outputs are to lead to positive outcomes. That's it. Thank you. Thank you so much Ingrid that's filled me with lots of thoughts. Thank you so much. So going back to the humble output and being more humble. Okay with that I'm going to go straight to Shani. Thanks Jess and thanks Ingrid for an absolutely brilliant provocation. So if you're based in the East Coast like me you've probably been sitting at your computer for about six hours now and for that reason I decided not to bring slides with me you're going to have to make do with my face. I will be talking about power today. So when I was invited to speak on this panel I had to have to admit I thought it was really hard to speak about trends and design and evaluation without acknowledging the COVID size elephant in the room. In a lot of ways COVID has both disrupted and catalyzed existing trends it's highlighted what is and is working in our social systems at this point in time. In particular it's shed a lot of light on social connectivity and community engagement especially in the public health and health promotion space. For those of you based in Melbourne like me or in Victoria or around Australia to be honest it's hard to forget the lockdown of public housing estates in North Melbourne and Flamington earlier this year. That intervention was put forth as a much needed response to COVID however its execution has been widely criticised from the provision of culturally inappropriate food to the absence of translated information offered to residents at the outset of the lockdown. It's safe to say that no matter where you sit on the spectrum of politics that public housing lockdown has not been recognised and will probably not be recognised as an example of a well-designed intervention. For me it's also raised this recurring issue of power in our practice, power in design but also power in evaluation. Power is and should continue to be a key consideration for every good evaluator and designer. We must always acknowledge the role we play when we bring an evaluation or a design process to a community. We must also acknowledge the implications of our processes for those who engage with them. So this time last year I spoke at a design conference about how designers might meaningfully share power in co-design. The presentation was a co-facilitation between Clear Horizon and Flamington Works, a community-led programme supporting jobs and small business growth for young people and women living in the Flamington housing community. We'd spent a year designing their women's co-design project together, a project that sought to address barriers to accessing employment for migrant women living in Flamington. Now a year later I find myself questioning the very content of that presentation specifically the ability for designers and evaluators to truly share power in our process and so I guess in that regard similar to Ingrid this is very much a provocation for myself. What's made me so cynical you might ask? Other than spending the last six months locked in my house like every other Victorian it would be another co-design project that I've recently embarked upon. This project is situated in the drug and alcohol space and involves co-designing of the historically marginalised and criminalised cohort. Unlike the Flamington Works project in which conversations about power and the notion of power sharing came quite naturally to us as a design team this new project has been less straightforward. While we address power from the outset so in our plan and process by acknowledging each of our roles either as process designers, content knowledge experts or technical experts and while we co-developed clear ways of working that held community at the centre of our process tensions with power have continued to emerge and have honestly had implications for the delivery of our design process. The staff differences between each of these projects has caused me to reflect both on how we as evaluators and designers approach power in the work that we do but also what does this mean for how we respond to and design for in the face of unprecedented social challenges like COVID where we're at right now. So to begin with power is relational it is not something that's owned or given and therefore it really cannot be shared. Instead power is developed through relationships and interactions relationships themselves being built on generations of history and cultural nuance. So relationships in and of themselves don't form or change overnight. Relational power also drones on pre-existing power currencies. So by power currencies I'm referring to the value that others place in the assets that you bring to the table. An example of such a power currency is lived experience. Lived experience can help to build productive and positive relationships by creating shared realities that we can work within. So when I talk about those shared realities they're not exclusive to design they're very much apparent in evaluation as well. If I go back to my example of Flemington works my lived experience as a first generation migrant and a woman of colour made my power as a process facilitator more palatable to the women that I was there to co-design with. Rather than sharing power they were gracious and accepting of the power that I held as a process facilitator in the very same way that I was accepting and celebrated the power that they held as our content knowledge experts in that particular project. However lived experience cannot be tokenised either you have it or you don't. And in the context of my most recent project my lack of lived experience in the drug and alcohol space did not thought well for my role as a process facilitator. I lack the same footholds to build the relationships that I was able to build in Flemington works project. So in this situation addressing power required a deconstruction of power in and of itself. It meant that I had to share some of my power currencies including to some extent my whole of my design process. Why might that be? Well there is power in process as a power that we all bring to the table as evaluated and designers irrespective of who we design with and where our values lie. What does this tell us then about addressing power? Well addressing power through relational mechanisms is only as good as the power currencies that you bring to the table and the power currencies that are held by others at that table. Relevant lived experience can make all the difference by creating the shared reality and neutralising all the various currencies that exist in each and every one of our banks. The second point I wanted to make was about how we actually address power in practice. So if power is relational and that indicates that we cannot share power what do we actually do when we say that we correct the power or that we share power in our evaluation and our design. Again I'm going to reflect internally and I'm going to look back at the Flemington Women's Project. In that particular project we invested our efforts into addressing power at two levels. So the first level was between Clay Horizons, Flemington Works and nine project support officers who were recruited from the community in the Flemington Housing Estate to deliver this project with us. The second level of power was between the project support officers and the broader Flemington community, recognising that nine individuals cannot represent the diverse wide-ranging views of a community rich in different ethnic and cultural backgrounds. So between Clay Horizon and Flemington Works, we co-developed practice principles to create a shared culture of working amongst ourselves and between us. We also created easy to use discovery tools so that the project support officers could conduct their own social research and replace traditional data synthesis and analysis mechanisms with visual methods and storytelling. Between the project support officers and the broader community, we embedded community feedback mechanisms so we had regular feedback loops to ensure that community voice could be fed back into a process along the entire process and make sure that we were better represented community in the work that we did. However, in doing so, in implying these methods that I've just discussed, what we in fact did was we framed power sharing through the lens of accessibility, improving opportunities at every turn for community to engage in a pre-existing human-centered design process. In reality, what we've done is substituted power with accessibility, where accessibility is just one aspect of power in all its rich complexity. Accessibility can improve the ability to engage in a process, but it does not guarantee the ability to exert influence on that same process. Where you have mechanisms in place to enable access to a process and to influence the process, such as having feedback mechanisms and practice principles like I described, a person can still only really influence the process as far as they understand the theory that underpins the work. Theory, which is often rooted in Western schools of thinking, and therefore are not universally understood. So by improving accessibility, we might be able to co-design an outcome, but the possibility of co-designing the process itself is limited at best. As such, accessibility has little to contribute to the dissemination of power. So thinking forward to a post-COVID future of rebuilding social connectivity and improving community engagement, to me personally, two things are very clear. The first is that power is relational. It cannot be shared, rather it's addressed through building meaningful relationships. And nothing builds relationships better than shared realities. To do so, you need genuine and relevant lived experience. The second is that as evaluators and as designers, we're at risk of conflating accessibility and power. They are far from the same thing. I just want to end this provocation by saying, like COVID, I'm just highlighting what is and working in our system. I was once told that the biggest barrier to change is the fear of defending and being told that you're doing a saying the wrong thing. To make change, we need to acknowledge that we're likely to continue saying and doing the wrong thing, even if we have the best of intentions. The only way forward really is to keep reflecting and adapting how we do what we do. Thank you so much, Charlie, that was really powerful. And I can feel the threads of humility coming through both presentations today, very reflective of your own practice. And it's hard to say the right thing too, after something like that. But I'm just going to keep going. I'm just going to keep going and invite Chris and then we're going to pull it all together. And I would say to Ingrid and Chris and Shani that you will have a chance to ask each other questions at the end of this as well. I will let you connect with each other. But Chris, over to you, Chris van Stone. Thank you, thank you. I thought I might spend the first few minutes of this reflecting on quietly on what Shani just shared with us. Thank you, Shani. Thank you, Ingrid. That's really rich. Coming to you here from Unceded Garnerland in Adelaide, I want to pay my respects to Garner Elders past, present and emerging. I work at an organisation called the Australian Centre for Social Innovation. And something we believe a lot in there is people-powered innovation. So really following on that perhaps less eloquently from what Shani's team of power. And we support a lot of different kinds of organisations to run participatory processes or projects or in some cases design 10-year long initiatives that are in some way, in many ways, participatory. So for example, we've recently been working with people who have just come out of prison on what used to be called a forensic mental health service. This is with Ballarat's community health. And creating a space for mental health professionals and people with that lived experience to come together and negotiate redesigns around that service, including changing its name from something that isn't forensic mental health service. Sometimes it's working, very often we're working with government. Sometimes that's around in more of a sort of commissioning space. So for example, we recently worked with families and people living with dementia to design a new living facility for people with severe dementia with the South Australian government and to design the specification for that. And we've recently been working with a number of towns that were affected in the summer bushfires to design a long-term initiative that would be community led. But also interestingly, the initiative itself was the whole governance over that initiative that's philanthropically funded is participatory. So it's participatory, it's a number of levels with a majority community board in that case, making a community member more of making decisions about the sort of shape and change of that. As Engrid was saying, they hopefully learn from those experiments over time and change the shape of that whole initiative year by year. So we did this because we think it can work really well, but as Shani's very well described, power is a big thing in this and this is sometimes hard. It can be hard and it's particularly clear when you're working with institutions where there are hierarchies and folks that used to making decisions and ministers or chief executives or folks like that are used to having final say or final tweak on something. So whilst we've had, you know, we're proud of lots of the work that we've done. We're also gutted about some pieces of work we've done that started off at one level in a very participatory way with a beautiful participatory way to go forward. But that was sort of ignored or if you were being generous, it was lost in reorganizations of government as it shifted between departments. And now they might be doing what they would have done anyway. And so I was wondering evaluators if you could help us with this. Because a lot of this power stuff is complex, as Shani was describing, and it is somewhat invisible or at least invisible to those who might be doing power over very often. But it is something that's happening in the moment, although sometimes that's behind closed doors, or sometimes that happens after the project is finished. And my simple provocation is, could we create ways of making this visible to all of us in real time? Could we get a little tap on our Apple Watch when we're overstepping the mark on power? Could we, and listening to Shani's thinking there, could we make it really explicit what aspects of power we might be sharing in a particular project or process? And what aspects we're not? And what might be the next step for us to learn how to grow into as a system, as an organization, as an individual? And could evaluators, acknowledging what Ingrid was saying about the need, you can only see what works a long time afterwards in this work, but could there be some real timish feedback on the power dynamic in particular projects? And could we really bring that to the fore so that we can reflect on what we're all doing or not doing and what we're helpfully doing and helpfully doing as we go through the work in the moment? That is it. I've held a challenger as Chris, I love it. So my head is buzzing and I'm sure people who are listening must be feeling the same with all of the things we've heard. I just like to thank the speakers for their time, but I'm also going to thank them properly at the end, but I'd really like to try and pull the threads together a bit, but I'm going to ask you to help me, everybody, everybody. So this is, we're going to do what's called a slam chat, right? What are you hearing? What common things are you hearing across the three presentations? If you could all type and I'll press together. Yeah, so don't press. Don't worry if you get it wrong. Just type something, chat slam. Yep. We're going to type something in. What commonalities do you hear across these three provocations? What do you hear? So tap, tap, tap, and then we're going to press together. I'm going to say some are coming through already. Go now then. Let's just go with the flow. Press go. Lots of, let's, oh wow, this is, yeah. So a lot about complexity, lift experience being talked about a lot in this, in this session. Power, power dynamics, complexity, relationships, difficulty of making judgment, power, power, complexity, power is complex. Ah, who said that? That brings it all together, doesn't it? Power is complex. Yeah, humility. Yeah, power is complex and we need to be, have, have, be humble about how we address it. And I love the idea of having a little watched. I've got that, you know that thing that people advertise at the moment where it stops you from slouching? I feel like maybe we need one of those like, you know, whenever we realize we're, we're abusing power. But now we're going to move into the final session here, which we've just got 10 minutes left for questions. But while you're thinking of your questions for the panel, I'm wondering whether the panel members have got any questions for each other. So I'll invite first of all the panel members to say if you wanted to ask any, each other anything while we collect questions from the floor. So if you're an audience member, feel free to type your, your question into the cat chat function. But if you are a panel member, verbally, verbally speak up. Do you have a question for each other? I have a question for Shana, actually. Yeah, go. I wonder, Shana, if you might just say a little bit more about what you saw as some of the people, the aspects or dimensions of power, accessibility being just one of them, what might be some of the other ones on your list? Yeah, look, thanks, Chris. It's a really good question. And I think it's something I've been grappling with because as I pointed out a year ago, I thought power sharing, I thought very differently of power sharing is what I do now. So I think accessibility is important. And this might sound a bit close to accessibility, but I think information is important to, and you can argue that accessibility provides people with access to information. But I think that that isn't necessarily always the case, because you're coming, if you are designing accessibility, you're designing a pathway to enter into your particular process. You're, you know, somebody recently used the analogy of roads. You're creating a shorter path. You're digging a shortcut through the grass to get to that next destination, as opposed to actually building that trail together. So I think information can change a lot of things. And I noticed a comment from Victoria as well about where the lived experience needs to be a precondition for proper genuine power, sharing or whatever you want to call it. And I would argue that it's not, it's about putting everyone's cards on the table and truly understanding what power currencies are at play, and then the implications of those power currencies for a process, which ties into your comment around evaluators, what can you do with power? And it's about evaluating power. Understanding what everyone's bringing to the table, but then what that means moving forward in a process as well. It's acknowledging that we can't share power in the way that we'd like to, and that happens organically and intrinsically as a result of the way we interact with one another, as opposed to our intention entering into that certain space. I don't need to answer your question, Chris, but that's sort of where my thinking is at. Thank you. Thanks so much, Shani. And did you have a question for anybody in grid? Yeah, I'm going to go for Shani as well. I hope that's okay, Shani. You talked a little bit about how you've become a little bit jaded by co-design and the power. I wonder if you have thought about what you would do differently in the next co-design project that you get involved in, or has it really turned you off this idea of co-design? Because I think power is something that is often discussed in co-design. And yeah, it does turn some people away from the process. Yeah, great question, Ingrid. And I think I should start by pressing that by saying I'm very much one of the people that just described who's an evaluator slowly turning into a designer. So I'm trying to have the best of both worlds. But I'd start by saying I've definitely not been turned off of co-design or design as a result of power at play. I think for me, I am often concerned about buzzwords. My background was originally called for sustainability and sustainability more generally. And if that isn't a buzzword, I don't know what is. So I have genuine fears about co-design becoming that next buzzword. And while I'm not elitist about jargon or language in any sense of the word, I do recognize that there is a danger to framing both evaluation and design as a co-develop process if it is in fact part of the Patreon nature. So it's about calling a state of state as opposed to anything else. I definitely think that co-design is something we should and will continue to work towards. But it's about having that reflective practice of really embedded into what we do, both in that room and then a year later, and being able to look at it and work out what we need to learn from and what we need to develop. That's great. Thank you. And I think we need more evaluators in the design space. So bring it on, Shani. And I love the idea that we should have a go at looking at how power plays out as evaluators. Don't you as well that provocation that Chris has given us or invitation maybe even when we're evaluating a design? That's another way evaluators and designers can work together is where you're actually evaluating the design process, which is I think it's an underrated beastly process design in evaluation. And I know that it can bring a lot of, you can de-risk design a bit, which is governments really like the idea of this, that you've got an evaluator there to look at what's going on and reveal. But one of the things we should really be looking at is power and how we do that well is something maybe a beautiful idea from Chris. I wouldn't mind having a look at some of the questions that have come from the chat function. Matt, is there something you'd like to pull up or something that you'd like to ask? Yeah, there was one from Deborah just earlier to Ingrid just on the concept of outputs that you mentioned and focusing on outputs. If you could just speak to that a little bit more. Yeah, sure. So I think, I mean, I have a fairly traditional view of what outputs are about. They're easily measurable immediate consequences of what's happened. And I guess from a design perspective, what I've noticed is that we're not very ambitious with our outputs and the way that we structure outputs. And we could actually be more, I'm going to, more singier, which is not a word, but I'm German. So I'm allowed to make up words. So for example, if I think about, you know, that last piece of work that we were involved in, the outputs that government funders were wanting was how many people showed up to the employment service? What a boring output. I mean, that's, that's like, how, how many cows are in a paddock? What does it actually tell us? So how can we get crunchier about those outputs? And one of the ways that we thought was, okay, well, if we knew where people spend time when they actually get to the service, and that's why we did that, you know, the mapping of where people actually spend time. And then we linked that to what are they actually doing? And what's their interpretation of how that builds capabilities that could lead to employability, that cluster of who's turning off how often, where are they turning up? And what are they doing? That cluster could give us strong signals about whether or not they are, you know, moving into greater levels of employability. And then if we add their own voice into that, to say, we know, what did you get out of today? And where you spent time, we've got like four outputs that together could give us a stronger signal about whether or not we're designing something that's leading towards employability. Your challenges are on what we evaluate because outputs, of course, are used as a measure of success. So the very measures of success need to be challenged in complex settings. Yes, absolutely. Yeah. We are really running out of time here. And I think it's been a fantastic start to the design and evaluation sprint. And I really, really would like to thank our panelists for giving their time today and their thoughts. So thank you so much to Ingrid, Chris and Shani. And maybe a little clap for everybody for giving them a time. And just so thank you so much. And before we dive off, we've just got a few words from Matt about the rest of the week and about the SIG. SIG, if you'd just like to stay on for a couple more minutes. And thank you, everybody, for participating today. Over to you, Matt. Great. Thanks, Jess. So just in terms of what's on for the rest of the week. So tomorrow we've got a session with a co-design facilitator, Christian Jewel, who's going to be running us through some of the processes that he uses to grapple with exactly what some people have been talking about. How do you practically deal with situations of power dynamics? And how do you engage people in that conversation? So he's going to step people through a bit of a process and a toolkit that he uses. So a bit more of a hands-on approach. On day three of our session, Jess, Chris and Shani, who you've all heard from today, are going to be running through a bit of a discussion around designers and evaluators working together for scale. So getting into that question of scaling and scalability. And then on day four, lucky last, I'll be running through some of these concepts that Ingrid's gone through about systems and systems thinking. But in particular, again, trying to sort of keep a balance between really insightful conversations and some tools or processes. So stepping through a bit of an exploration of mapping systems. But what that means actually from a people perspective rather than conceptually or conceptual mind mapping. So again, a bit of a practical application. And then at the end of the week, like I said, we'll have a survey that goes out. Basically, want to hear from you what you got out of it so that we know what we can do better for next time. And then ideas for next year as well. Yeah, I think besides that, I think we can call it quits. Oh, look at that. 30 seconds early. You can leave your, oh, yes, what I'll do, someone's just focus, I'll put the link to the SIG just in the chat, just while people are kind of starting to leave. But if you want to leave your comments or thoughts in the chat, you can. But otherwise, you will be free to go. Sorry, Kara, I'm specifically doing this for you. Come and join us for the rest of the week if you're not already registered and have a great afternoon. Bye bye everybody. Thanks to you very much. Thank you so much. Thank you.