 OK, so let's get started. Welcome again to everybody. I'm pleased that you're here. And I'm basically I'm like the person between you and your deserved weekend. So I'm trying to make this session as interesting as possible. And I also invite you very warmly to participate in the session and ask questions. So that I can accommodate what I say and what I present to your needs as well. So I'm here for you within the next one and a half hours. And so you please make the best use of this time as well. So before I introduce myself and the organization, I just would like to give you just an overview of what to expect from this session. And I'm just trying again to share my presentation with you because obviously that didn't work right now. So I'm trying to double click on that that you don't only see the starting slide. OK, so here we go. So will you be an evaluation expert after the session? I'm sorry you will be not. Not even I would consider myself being an evaluation expert because that's really a big task and you always learn on the run and on the certain project. So don't expect to have something ready to start with when we end the session. What I would like you to do or what I would like you to take away during the session is that you rather be able to ask appropriate questions on your journey towards a more impact oriented work than having already some fixed answers because it would be dangerous if I gave you like the fixed answer right now because everything you do when you look at the outcomes and the impact of your work will be very specific to what you're doing and in what environment you're working. So the thing I would like to do is just to provide you with a certain mindset and what I said, certain questions, how you and hopefully your team as well can address the idea of how to achieve more impact and impact orientation in the daily work of your organizations or your program. So basically what is impact orientation? That's not a term that has like a solid definition that works for everyone. So I would like to provide you with the definition we're working with and subsequently with the definition of what is impact orientation. So what kind of management idea are we looking at when we talk about impact orientation? Why is that important for you? Why should you bother? And to deal with all this monitoring and evaluation stuff, it is really cumbersome. It's a lot of hard work. And so you should have a good reason why you really, why you deal with that. And I'm trying to provide you with some ideas why for nonprofit organizations as well as museums, it might be a helpful approach to look into this. There's benefits of impact orientation. And then we are changing over to a little bit more practical orientated part because I want to show you what are the different steps within a program management circle you have to address in an impact-oriented journey. And one of the first things you have to do, of course, is to know where do you want to go or where do you want your program to end up or what do you want to achieve for your target group? So setting objectives with a focus on social impact would be one of the first things you have to do with your team to get started. And then I will give you a very quick run through tools or instruments, however you want to call it, to measure. And I put this measure in inverted commas. And I will tell you later why I did that because we are not very happy with the term measure because it suggests a certain approach to something what is really difficult to measure actually. But so these are the different points I would like to cover in the presentation. And before I start, I just would like to introduce where I'm from and the organization I'm from and who am I probably as well because you have to deal with me for the next more than one hour. So who is the person who's talking to you? So I'm working for an organization that is based in Berlin. It's called FINEO. We are a nonprofit organization ourselves. We do so and our aim is to strengthen civil society in all the different aspects that they are by encouraging nonprofit organizations and their programs to work in a more impact-oriented manner. We do that by providing them with certain tools and certain help. I come to that in a minute. And at the same time, we try to showcase the success from the social sector because we think beside politics and economy, the civic sector, social sector, how you want to call it, it's like the important force and the important power that shapes society. And quite often, it's a little bit neglected. And people who do amazingly good work in this sector, sometimes are not that able to show the results they do with their work. And we try to change that by highlighting best practice by examples, by highlighting impact that is generated by nonprofit organizations in order to just put more of a spotlight and emphasis on the benefits we get from the social sector. So we do that by what I already said. We create and distribute knowledge. So what we do exactly for nonprofit organizations, and I will provide you with a couple of links at the end of the presentation. We write handbooks. We have a learning website and so on and so on for nonprofit organizations who are interested to improve their monitoring and evaluation, their fundraising skills, and so on and so on. So we do that for free and hope to be able to support nonprofit organizations through that. Then what I also already said, we try to feature best practice examples in the social sector, not only because we want to highlight certain organizations, but because we also think that talking about best practice is a good way to improve the work of the social sector in general. We do agenda setting as well, especially when it comes to transparency. We think like talking about what's going on in the social sector when it comes to financials, when it comes to governance, but also when it comes to the achievement of nonprofit organizations is very crucial for the advancement of the social sector. And so we do agenda setting by publicizing research studies and so on. And the last but not the least thing we're doing is we work with so-called social investors. So these are people and institutions that support the social sector. So that can be like wealthy individuals, but mainly it's foundations or it's companies who do good with their corporate citizenship agenda. And we help them to co-work with the nonprofits they are supporting in a way that both of them benefit the most. OK, so and me, just about me, who am I? So I'm working for FINEO for almost 10 years now. That's impressive, even for myself. And before I worked actually in the field of political education, that's a bit what Leonat Schneeding talked about yesterday. That was the kind of work I was doing, working with school kids and working on issues like democracy, understanding the European integration process and so on. So I'm a little bit familiar with some of the programs that are run by museums at the moment. And I know how it feels like to do this work on the ground. So and now, but now I'm like a specialist for impact orientation, for organizational development, for monitoring and evaluation. Yes, and I'm trying my best to give you as much of my knowledge as possible. And please, sometimes just try to drain everything out of my brain you can by asking questions. I tell you what I think is important, but probably your questions are even more important. And just don't hesitate to share with me. OK, so just for the technical aspect, I see that there are already three people who want to join the panel. What I would like to do is that I proceed a little bit in my presentation. And then when we come to the first part of discussion, then I would invite you. I think that's the best. Otherwise, things get a little bit confused. OK, so what is social impact? What is impact orientation? Some dry definitions are waiting for you now at the moment. But I think it's really important that people are on the same page when they talk about social impact. Because when you just, and that's just like the first little exercise, I would ask you to do, when you sit back for just a couple of seconds and leave you alone and let you think a little bit, and close your eyes and think, what is the organization I'm working for? What is it here for? What is our purpose? Why do I get up every morning? What is it that makes me feel engaged to what my organization is doing? What do I value? And what is most important for me? Just think about that for a second. If you want to, you can type it in the chat, but you don't have to do that. Just try to reflect a little bit. So I would suggest, and I put a question mark there actually, but I would think that as you're working in the field of museums, and that's part of the civic sector, it's most probably because you do that, because you want to achieve some impact with what you're doing, not knowing already what impact is, but you want to do something. You want to do something good for your audience, for your target population, probably for your community. As a museum, you probably have a lot of different tasks on your table. At the same time, you have to work in preservation. You have to manage exhibitions and so on. But you do that having a goal in mind. And my impression would be that this goal is that you want to do something good or something valuable for society. And when we now look at it and think about what do we mean by social impact, I would just ask you, first of all, to be careful about not mixing up terminology. It's a bit annoying to have to talk about terminology so much. But the thing is, in the field of monitoring and evaluation and in the social sector, people talk about social impact a lot. And interestingly, a lot of people mean different things by that. They all go in the same direction. They try to say the same, but it's like same, same, and different. It's still there like the nuances that matter. And they matter specifically if you have to be precise and communicating about what you understand or what your definition of social impact is. For example, if you have to fill in some application forms for a grant or something, always be careful what is there behind, what the meaning behind the different terminology is. Because different institutions may use the words outcome, output, result, impact in a little bit different way. That doesn't matter. And that does not change the concept. It just changes the way people talk about. And you want to make sure that you're on the same page and you talk about the same thing. And what I'm doing now is I'm presenting you with the so-called input, output, outcome, impact methodology. That's a logic model we're working with. There are other logic models as well that are slightly different. But I think that gives you a clear or a pretty good idea what we're talking about using this different terminology. OK, and there you have. And I hope you enlarged your presentation. Otherwise, it might be a little bit difficult to see what is written in this logic model. So you see it's basically kind of like a flow chart. So it suggests there's like a timely development in time of certain aspects. And what we do is that we start with inputs. That means what is it we put into our organization? Or what is it we put into our program to make things happen? That can be the team who is working for the program. That can be the money that goes into the program. That can be educational material. That can be like in a museum. That could be like a special exhibition targeted for a certain audience. You're running your program with. So it's usually it's everything that you what you invest into a project that can be it can be collaboration as well. At what points do we work together with other project partners? So that's all that goes into the project. And that's what it's usually that's called input. And what does input do? It produces output. So it produces like the first things you do within a project. It's like the offerings you do for your target group. Like you build an exhibition and you open that. You offer a training course for teachers how to talk with their kids about arts. You write a report about a certain subject you're dealing with or your museum is dealing with at the moment. You're providing informational material. You're doing workshops for kids. You're doing probably activities for elderly people or whoever. And so this are like first of all like this is the first stage of outputs you have. So that's what you that's what you do. And so the second part of outputs and you see you know like the logic model you know it's going down there a little bit or you know it like kind forms little columns you can you can put that in whatever you know you can also make it a very long flow chart. We just did it like that for visual reasons otherwise we wouldn't fit that on a page. So and so like the second step of outputs is the people you reach. So it's your target population and it's not only your target population it's actually the beneficiaries. So it's the people who come to your museum. It's the people who attend your seminars. It's the people who visit your website particularly now during during the lockdown. It's the people who read through your materials. It's the people who are well benefit from what you're what you're doing and it's very very necessary to look into that. But at that point we are not talking about like results or outcomes yet. We're still talking about output because actually people came and used your services but we cannot talk about what what happened with them up up to now. So we do that in a in a next step when we talk about like the satisfaction of your of your beneficiaries. So the question is are the people happy with what you're doing? Are the people like content with what you're doing? Do they do they value it? Do they think that it was worse while spending spending their time? And so it's basically what you what you do with when you do satisfaction surveys you ask people where you're happy with our service but when you look at it from like an impact oriented point of view at that point unfortunately you have not reached like impact or outcome at the moment. You're still you're still on the level that you reached a certain a certain kind of output. People are happy but you cannot at the moment you cannot quite tell whether there's something like a like a longer lasting effects on the people. If you want to look at that you move to the next step and that is outcomes. So outcomes are the results on the target group level. So that's what your visitors what the kids in the school groups who come to your museum what they do and what they learn. So you look at that and when you look at outcomes you always ask yourself what difference do we want to make in this certain group of beneficiaries? How do we want this people to? Well, you know, it sounds a bit weird but actually that is the question. How do we want to be this people different after they experienced our exhibition or whatever it is then they were before? Do we want them to know more about a certain subject? Do we want them to have developed a certain attitude towards a certain subject? Do we want them to be more confident for example in what they're doing and how they're dealing with things? Do we want them to be a more inclusive part of society by being included in this kind of activity? So what does change in the single person's life who got through your program or who benefited from what you're doing? So did they learn something? Do they feel better? Do they have a different attitude towards something? And so what does that mean for people's everyday life? Do they take what they have learned into what they do in their everyday life? Is it only like a short term? Is it just a short term intervention or is it something that lasts for a little bit longer? And these are the questions that are asked by the impact orientation. You know, we have like a short term, we have short term results and we have longer terms results and the further the results are wave in a timely manner the more difficult of course it is to look at them and to analyze them. So and then when we continue looking at the logic model we in the last step, we come to the so-called impact in this definition, what defines a change or hopefully positive change on a societal level. That does not mean that a small museum has to contribute to like the society in the world as a whole but what changes in your community in your community as like a small scale society? What are the differences your organization can do on that level? And having said that, I think it's important to acknowledge that we as organizations who try to achieve impact or who try to towards results and impact. We're not working on a clean slate. We're not there on our own. We are always working in a certain environment and we have to deal with the challenges in this environment and we have to adapt what we're doing accordingly in order to reach our goals and objectives. And when you look at this circle, you see in the left-hand corner, you have the problems or the challenges that are there in society and that kind of like drip down on those problems and the needs your different target groups have. And with your vision and your goals of your organization, you're trying to address these needs and you do that by running programs, by running exhibitions and so on and so on. And so that's what you can do and what you're responsible for. And the way you do it will kind of affect the way, it really helps the beneficiaries and the participants. So, but what happens to the results on the long run, it's really hard for you to, well, to prove or to see because the further away your target group moves from what you did as a program, the more of other factors come into play. So it will be really difficult for you to make sure that with what you did, you really achieved this and that change within society. But what you can do is that you by thinking about like logic models and so on, you can kind of like a cause, you can kind of knit like a cause and relationship and say, you know, by what we are doing or what we did, we think we are contributing to this and that kind of social change because what we see is that we achieved or we achieved results in our beneficiaries and we only can or we hope for, we're more or less confident that that also is part of, you know, of being, or that, how would you say that, you know, it kind of like drips or it contributes to social impact on a community level. So what I was trying to tell you, you know, that it's like the area you are, you know, you with your work can take like 100% responsibility for. So we did that, so therefore we achieved that. It's pretty limited yet, but so your idea should be, is always to be on the look or to the watch out to how do we contribute to civil society, to our community, to society, to society as a whole because that will be what you frequently will be asked for and that's what is so difficult for most of the social organizations to say something about well, to say something about or to give proof. Just stop here, just for a very brief second and ask whether they're already, whether they're already questions on that. So, and I give it a try and let people into the, does it work? No, it doesn't. And let people onto the panel. So the first is Sophia. So I see what happens. Hello, nothing actually, okay. So I try with the other people who are, so I'm trying to invite people on the panel. So as soon as I click on that, they just seem to disappear. I'm sorry for that. I don't want to, I don't want to make you disappear. Okay, so in, I'm sorry. So this doesn't seem to, this does not seem, so either I'm too stupid to work it out or it does not work out. I don't know. So what I suggest, I'm sorry for that, is that if you have any questions, please type it into the chat and so we can deal with that and we make sure that nothing gets lost, okay? So after this dry theory, and I warned you, it's a bit dry stuff, but it's really important to go through that and have a good understanding for that. So I would like to look a little bit and I'm really interested in your opinion on that. I would, having listened to what people talked about yesterday, oh, there's a question coming in. I take that question, it's about long-term measuring, impact means long-term measuring. Who does that and with which resources when a project funding is over, how do you deal with that? Okay, so I grabbed this question first of all. So I think, and that's one of the challenges, I think, well, most of the social organizations have but organizations who are dealing with what I call, I'm not sure whether that's the appropriate English term but what we call like short-term pedagogy. So most of the time you will see your visitors or the kids you're dealing with only for a very short time. They come with their school, they do the program, probably there's like preparation phase and probably there's like a recap phase afterwards but usually you will not work with this people for a very long time. That can be different in the social sector when people really deal face to face with social workers, say every week or something who deal on issues like social problems, housing and so on. So like the connection of the organization to their beneficiaries might be a bit closer than it is in the work of arts and culture. Still don't despair. I mean, what you can do as soon as you have the impression that you established a program that is running on a little bit longer term and you're like working with target group or you're working with beneficiaries, for example, who came from the same community or even the same school or something like that. What you can do is that you buy, asking the people who have been in your program, you ask them just again, I say like half a year later, a year later, something like that, you know? And so you can kind of track what kind of learnings stayed on probably what other changes, what other changes there was, you know? You always can, when you ask, when you work with participants a little bit closer, that does not work with someone who's walking through your museum one time at least. But for example, when you work on a longer program with kids or young people or elderly people or whoever, you always can pick out a couple of people and make sure that that's a representative panel you're forming and asking them whether you can contact them again after a certain time, you know? That you say, okay, we would like to find out how people, where people ended up who were doing our programs, probably kids who were at the brink between school and working life or whatever. We want to know that, could we please contact you in, I don't know, in a couple of months, any year or something like that? I mean, there's always the risk that you lose these people. So not only don't only agree on that with just one person or two persons, but try to make sure that you have a higher number. But I mean, that's one of the things you can do. Otherwise, I mean, of course, it's really difficult to do on the level of outcome and impact to talk about like a long-run development in this case. So, okay, hang on, I have to measuring impact culture. Yeah, so methodologies and tools come later, just wait a little bit. And what about methods to assess causal impact? So you can assess the causal effect of the project or policy. Okay, so we have, that's one of the things I was about to mention later. We have this experimental approaches. So for people who are not familiar with that, you, in the so-called randomized control trials. So these are evaluations that like a really measure impact that you have a control group. So that's a group that is not participating in the program and you have a group that is participating in the group. And so after that, or while during the program, you measure what happens with both groups and you try to compare these two groups. And then you can, well, take, talk about like the causal effects of the program was a little bit more of a certainty because you say, okay, this group, obviously was treated with our program and they showed this effect. And the other people who did not participate in our program don't show this, don't show this effect. So it's pretty likely that this effects are caused by our program. So this is, I mean, that's like the high, the very high standard of evaluation and very, very few times this is done. It's a very costly method of evaluation. Sometimes it's done in development work, you know, so like the really large programs ran in countries, sometimes do randomized control trials. Honestly, I don't know about the example in like the work of arts and museums that that happened, but I might be wrong, but I haven't seen that. So, but what you can do, you know, if you, I mean, with some certainty, you want to link your program with the effects you see. I mean, one thing is that by setting up the tools in a way that you ask your participants, what do they, what they think about, you know, what, how much they think that the effects of the program really created the effects on them, you know? Because I think they are pretty able to figure out, you know, what caused effects with them and what didn't, you know? I mean, they might say, well, during that time, I learned this and that, and I definitely did do learn that via this program. But they also could say, well, you know, actually, yes, I learned that, but I mean, the program was helpful, but I also attended, I don't know, another program or some of my friends helped me or I just got some input in school about this and that. And so, you know, by figuring out that and asking your participants, you know, how they judge the influence of a certain program, you can get a little bit closer to, you can get a little bit closer to, well, I wouldn't call it proof, but being a little bit more certain about what, you know, the causes and effects in your program. What you also can do is that you ask other people, you know, people who probably have the comparison. For example, if you have kids running, you run a museum's educational project for kids in a school. Ask the teacher what differences he sees in this kids compared to some of his other kids he's teaching who do not attend this program. You know, that also should give you some, well, evidence or something close to evidence that helps you with that. Could I answer the question? If not, just feel free to continue asking. Okay. Okay, the Phineon navigator considers those designs anything specific for museums and art projects. You know what I think about that. So I think there are specific, well, some things specific to the work of museums and art projects. Still, I think the questions you ask and the frameworks you use, of course, you have to adapt them properly here and there, but that's the content. You fill them in and I will try to show you later what I mean by that, you know. Still are, I guess, more or less the same. I don't think that there's an evaluation or like an approach of impact orientation that is specifically and only designed for, excuse me, and only designed for arts organizations, the frameworks we suggest, and I mean, it's not only us, it's like just what the world of evaluation suggests that frameworks are pretty flexible to be filled with different content. And that can be like, that can be social projects, that can be cultural projects, that can be educational projects because it's the matter it is, what questions you ask within the framework. And you do that, of course, you do that specifically to your certain program and also to your certain societal or the certain needs in your community, you know. You fill that very individually with what you want to achieve and what are the needs you're addressing. But I come to that, I come to that later. Okay, so when you look into the chat, so there are people who share some practical, some best practice from the Oakland Museum of California. And so one thing what I did is at the end of the presentation and you do get the presentation, I have like two pages with resources and further reading and in there, you will find a lot of best practice examples. You find a lot of tools and so on. So that's just for further reading. So I had to end. So I have another person popping up on the panel. Oh, okay. And so, yeah, hi, no, sorry, you're on the panel. No, I'm really sorry. So, okay, sorry, sorry for that. So, okay, so let's continue. We covered already quite some things I was about to say later on. So what are the challenges of social impact in the work of museums or in the work of art organizations? Don't worry, Victoria. So actually we should have people on the panel. So actually I should encourage people to participate on the panel so that we're not only read from you so that we also hear from you. But anyway, I just continue on. So why is social impact in the work of museums so challenging? And is it more challenging than it is in other social organizations? And I honestly, I don't really know because I have the impression to whatever area in social work or cultural work I talk to everyone claims that it's specifically difficult for them. So I think there are challenges absolutely that have to be addressed. And one thing is what we talked about earlier, you know, you compared to other social programs most probably do not deal with your participants for a very long time. And that's definitely a challenge for evaluation and tracking the results of your work. And you just have to try to do your best and you know, find proxies to get as close as possible to learn the things you want to learn. And you also, I mean, but that's the case for every social organization, you know, you work in so many different communities and environments, you have so many different approaches and you have so many different subjects you're working on. You have natural science museums, you have art museums, you have history museums, you have museums that talk about a certain, I don't know, certain people in their history. You have museums that talk about, you know, how technical things developed over time. You have something like memorial sites. I mean, they could, in my opinion, this is a kind of museum as well, we should focus on. And so there are so many different issues and so many different fields you're working in. And then at the same time, you have so many different target audiences. You don't only have, I mean, you have the people who usually come to your place and who usually visit your museum, then you probably have people whose idea is not to come to your place because not because they want to, but just because it's, you know, it doesn't come to their mind that they should, that they could go to a museum for a change or something like that. So you have, I mean, you kind of, you have the target group you're already or you're working with and then you have a target group. You probably don't even know what actually their needs and interests are. So that is very, very challenging because you kind of have to build an interest for your cause. You have to build an interest for the arts and everything that is around that. When you do political education, you have to build an awareness where that is important. And you have to make a stand and communicate why you think that you're serving in need. Your target group probably does not realize yet that they have a need for that. So that's, I think that's really interesting. And I, you know, that came to me yesterday more and more clearly that it's, you know, you're not like an organization that's, for example, you know, providing shelter for homeless, you know, the need is very obvious. You know, for the people themselves, they're homeless, they need shelter or for someone who is hungry, they need food. But for someone who is not engaged in, well, say like cultural discussions or political education, they don't necessarily feel the necessity to do that. So you're kind of like you're building a market and you do that by looking very carefully at what the needs in your certain community are. And I only can invite you to what you're doing right now that you, that you, the conversation you started, that you continue, that you do, I mean, what just happened in the chat, you know, you share best practice. You share, you also share worst practice, by the way, you know, you tell each other what did work in my organization, what didn't work, what approaches did we use, probably other people, other people can contribute as well. So it's like learning, it's learning on the run. At the same time, I think they're already, you know, when someone in the chat yesterday provided a link to like the museum, the museums of impact in Finland. So that is what I understood that they're developing a framework on what measuring or managing the social impact of museums. So these are things that are worth to look out for. And these are things, by the way, that I have, you know, in this resource list I give you at the end of the presentation, you know, there's a lot of this in there. So, you know, it's worse to look around and it's worse of, you know, not only looking at the first three things you find in Google, but also digging a little bit deeper and especially also to use the resources, you know, organizations like Nemo and, you know, all the organizations, all this umbrella organizations you're working with provide you. So when, you know, when I did my research, I'm not a specialist on museums, but when I did my research for this presentation, I found that there's quite a lot, you know. The thing is, you know, the framework itself does not help. It needs to be implied as well. And that's a process that is not really, it's not really easy. But, you know, as we have to proceed on a little bit, I'm talking too much and I just check the question and answers again. Ah, okay. Thank you, Peir, for the link again. I think that's, I tried the website yesterday. It was not online or I had problems with the computer, but I think that's definitely something we all should watch out for. Okay, so I have to continue on because time runs. So one thing I would, and you know, I think I use this slide a lot. And I actually, I think it's the most important slide in the whole presentation because it actually should give you, or it gives you an idea why to bother with impact orientation. Why should we do that? You know, you have so many things on your table, what you have to do. Why do you have to think about impact, you know? And we think that they're basically they're two main areas in which impact orientation can help you. One is the area of to prove what you're doing. You know, you know that pretty clearly, you know? You have to write reports. You have to talk to your board. You have to talk to the CEO of the museum and so on. And you have to give evidence about what you're doing and what you achieve is what you're doing. And when you deal with impact orientation and when you try to collect data on the impact or the results you're achieving, it will be much easier for you to enter this kind of conversation because you will have, well, you have data, you will have qualitative quantitative data that helps you communicating and underlining your cause. And that is all so very, very helpful for fundraising because, you know, more and more donors ask for, you know, to what end do you do these things? It's nice that you do that, but to what end, what are the results? Why are you doing that? And the better you're able to answer these questions, well, the more successful that's, I'm not trying to promise something, but I think the more successful you will be in your fundraising and in your outreach activities. Okay, and then the other thing, and I think that is, well, in my point of view and in Finiu's point of view as well, it's probably even the more important aspect of this too. And use your impact-oriented approach to improve your work. You know, it's nice that you can provide facts and figures about what you're doing, but if you're not using this information to reflect on what you're doing to, at points where it's necessary, probably change a little bit on what you're doing and always learn and get better then impact orientation really, I would say it kind of like fails on the halfway because that's what impact orientation is about. It's about keeping what you want to achieve in mind. And if you do not achieve that, don't despair, but think again and think again, well, what happened? Probably we had the wrong assumptions or probably our program did not run as planned or there were so many side effects that affected the results of the program. So this and that happened. And when you do that, when you can always keep your impact in mind, that's the only way you will be able to, well, to learn and to be able to provide better services for your target audiences on the long run. Okay, so I have another guest on the panel and Sarah, I just invite you, I hope it's okay, I hope. Thank you. Hi Sarah. Oh, I'm sorry, I just, I thought I had to do this. I'm just listening to what you're saying since the beginning. No, you don't have to do that, but if you have some ideas or comments. Sorry, I'm going out. We are very happy to share that with you. Everything is fine. Okay, so I move on because I promised you some more, some more tools and some more tools and helpful and helpful things. So one thing I want to mention and you can have a look at that when you, well, first of all, when you look at the presentation, but then also look at the social impact framework we provide on our website. I give you, well, the link comes up at the end of the presentation. So that is basically, that's a management cycle, just a normal management cycle that what you know from project management, you plan, you carry things out and then you look back and you reflect how did we do and then you plan again and you start all over again. And in the impact oriented, in the impact, okay, I take a sip of water and start again. So when you do project management with a focus on impact orientation, it's very important that you focus on the challenges and needs of your target audience or your community. And we talked about that before, you might wonder that the needs of this target group, they don't jump directly into your face. You will have to figure out by using, I don't know, interviews, inquiries, focus groups and so on. You probably have to dig deeper into what your community really needs. So in order to build a project that serves the need and then consequently and then consequently reaches the desired results. And what the desired results are, that's basically what you and your team and within your organization and with the participation of stakeholders, what you set in the so-called project objectives. And I jump to this slide now and so there's one slide you can read through later that's questions that help you to evaluate or to figure out what are the needs in your community and the target group. And when you now get to setting your project objectives, so like the question, what do you want to reach with your project? We suggest to turn back to what we saw before. You remember this logic impact model flow chart and graphic wise you can do that however you want. What we did is that we kind of flipped that thing over and we turned it into like a, it looks like a little staircase or a little ladder suggesting that there's one step after the other that can be reached or should be reached but you can do that however you want. But what is interesting or what I think is helpful is that for every step of this logic model you ask the appropriate question in order to set the goals or the objectives you want to achieve. And what you do, you don't start or you should not start at the bottom at number one and start with, oh, these are the things we're doing anyway but you would start at the top, that's number seven and we saw in the flow chart graphic that number seven that's the changes on community level or in society level. So you start asking what do I want to achieve or what do I want to change in society and in order to do so and then you make your way quick your way slowly down step by step staircase if I want to achieve this change in my community, for example, I want to be my community more inclusive or I want to be my community to participate or engage more into political process or something like that. You always ask the question in order to achieve that what do I, what does, what does, what things have to happen before and before you have changed in community you most probably have, most probably it's, it's tricky. I'm not talking about advocacy here at the moment I'm talking about like programs that usually target people like single people in sense of participants. So, but if you want to achieve community change you also have to achieve changes in the life of individuals and so you think what are the changes in, what are the changes in the lives of individuals I want to achieve and in order to do that what does this person need to learn and in order to learn that what kind of service do I have to provide. So that's the way you work and I wrote down this questions in more detail and so that's the way you kind of think about your theory of change in order to, in order to reach the goal that is probably very far away and that societal change and we will probably never be 100% sure about how far we were able to contribute with our project to the actual change but we can by looking at what happens on the steps on the way we can kind of build and that was one of the questions before we can kind of build like a causal chain between what we do and what happens on a more abstract level. So when you plan your project and you set your goals or objectives however you want to call it, you kind of think backwards and then you, I mean, of course, then it should be like a process where you have your team involved it should be a process where you should have stakeholders involved and it definitely should be a process where when you're working with the community you should have people from the community involved. I think that's a very important criteria for quality, the quality of programs. Well, I mean, for the whole, not only for the planning phase but also for the other phase that you have always, you know you kind of speak with your beneficiaries or your participants and you always, you not only welcome their feedback but you also take their feedback into as something you really acknowledge as a important source of information and you really deal with that. On the next slide and I hop through that I just gave you a few ideas how to write down project objectives in order to make them easier to evaluate later. You heard that a thousand times before I can, I think I can skip that if you want to have a look at it again you can go back to the slide. And then I would like to provide you with some, and these are not fixed answers but just some ideas about what does impact and outcomes in the arts and cultural domain mean. Some people always, you know people ask for are there certain sets of indicators or certain sets of goals we can use for our work and for our measurement. And I always say, well, actually there of course there are indicators that are more or less commonly used but it's not a good idea to take that one to one and apply that to your organization. You always should think about your goals yourself and what that means for your evaluation or your work in your specific context. But still so what the, I think it's from the it's the Arts Council in Great Britain I think they're doing a pretty good and interesting work when it comes to self-evaluation they have a tool on self-evaluation. And so what they suggest, I mean it's a suggestion you always can elaborate on that or extend that or say, well, that's not for us but they recommend like four areas in which arts organizations have an influence on society. And so I'm not going into that into detail but so they define like four areas where arts organizations create an impact and that can be in society that can be on an individual basis when it comes to health and well-being that can be in education that can be as well on a personal level as on a societal level, I think and it also can be on the economical level. Of course, I mean arts organizations and museums are also players that generate income and so on and who have employees and so on so that can play a role as well. But to zoom into that and just to have a couple of ideas about what other people use as indicators in the literature section I added a couple of other organizations that provide so-called indicator sets for the area of arts. It can be helpful but don't take it just like one to one and use it without reconsidering whether that is really something that works for you. Because then you end up with something you need to measure and you did not really work towards this result that would be pretty unfortunate to say at least. Okay and now tools and methods to track, to measure results. So again, I put this measure in inverted commas and that's why I talked about that, why we do that because measuring impact means like in a strict sense it means doing a randomized control trials and that's not what we recommend and I mean it's a very high standard but it's really not applicable for most of the organizations. Okay, but how do you find the right tools and methods you can use and I mean there's a lot out there most of the things you will see again and again so probably you will think oh, a questionnaire again but that's about it. But how do you figure out how what tools are appropriate and I suggest and you definitely do not have to be able to read that but what I suggest, what we find extremely helpful and when I do evaluation I use that for my work as well to use an evaluation plan and doing that keeping in mind what we said before I just quickly jump into the slide again to remind you and to ask yourself what kind of information do I need for what purpose? What kind of information do I need in order to be able to improve my project? What kind of information do I have to have to provide for my stakeholders internal or external? You know, when I write my reports for the grant makers or whatever, what are the facts and figures I need to provide? What kind of information do I need to figure out whether my project is just up to date with the need in the community? And keeping that in mind, you put that into the data collection plan or you can put it on a whiteboard wherever and but just collect your thoughts on that and then ask yourself what kind of information is that I need in order to communicate with my stakeholders who of them need facts and figures, just heart numbers who of them rather need a more qualitative approach so probably they want stories, they want pictures, they want whatever and at what point do I need particularly qualitative data to make sense with the numbers I got, you know, because that's the thing about and I'm, I don't have the time to elaborate on that too much but you know, you have quantitative methods and qualitative methods and usually, I mean, I can oversimplify that, you know, that quantitative methods provide you with numbers and qualitative methods provide you with more information and you know, in order to make sense to the numbers, you know, why did the number of our visitors increase or decrease or why did 70% of the students 70% of the students and this and that program said they liked it very much or did not like it so much, you know, you will, if you have the bare figure, you will not be able to figure that out, you know, we in, you will always will need some stories, some comments, some information you get from interviews and so on to make sense in order to, you know, in order to make sense with, you know, with the sheer numbers you have and then again, yeah, so that's just what I just say, you know, you always will need different angles or different perspectives. You look at your project so and the more reliable you data needs to be, you know, the more effort you have to put into a more rigorous approach of data collection. So if you just want to know whether people were happy with a, I don't know, with a picnic in the park today, it's okay to ask two or three people and ask them to tell you their story. If you want to know whether you, whether you scale your educational project in your museum for the next, I don't know, for the next five years or to another museum or something like that, you might want to have a little bit more evidence, you know, you want to have not only some, you know, some people who said something but you want to have a more rigorous approach in data and data collection. So that means you have, what most of the time, it means that you ask more people, you ask them in a more structured manner. It can also mean that you need someone from external, you know, to lead this evaluation because you, I mean, self-evaluation is a good thing, but, you know, having an external evaluator sometimes adds a little bit of, well, you know, well, it's not trust, but you know, it's sometimes, you know, people just want an external evaluator from a neutral perspective to look at programs, so that might be the case. But at the same time, of course, the costs and the resources you put into this endeavor, of course, of course, rise as well. So you always think clearly about what you want and what you really need, you know? And in this, you know, looking back at this data collection plan, you will see, after a while, you know, when you figure out, we do not really do something with this kind of information and it's really cumbersome to evaluate that or to collect this data, don't do it. You know, only collect data when, you know, when it is to a certain end, when you need it to report or when you need it to improve your project. Otherwise, it's just a waste of resources. Okay, so that's, I mean, you can, later you can read through that if you want to, but that's just, again, you know, what are the advantages and disadvantages of qualitative and quantitative methods. So, and now what I did now, and I see we're running out of time I'm sorry, I'm sorry, that's, I put down a couple of methods you probably, you probably already know. So it's, I mean, it's about questionnaires, it's about focus groups, it's about interviews, it's about collecting or collecting of anecdotes, you know, and so in there, like on the slides, you see what are the advantages and what are the disadvantages and basically I covered most of that. You know, the more reliable the data has to be, the more standardized the test needs or the tool needs to be, you're using. So I'm not going through all of them, but one thing I would like to draw your attention to because I mean, you probably say, well, questionnaires we do and interviews, I know how to do that. And focus groups, well, we probably do that as well. But I would like to draw your attention to one link I put into the recommended reading section, that's about tiny tools. So you do not always have to have like a fully fledged questionnaire or this and that, you know, when you have kids in a program or, you know, people in the program, you also can use really little tools that help you getting feedback from your audience. You know, that's about, I don't know, sticking little dots on the wall to like, did you like the program today or did you not like the program today, you know? I had a program, you know, kids also, you know, they had this little balls, you know, with smiling faces on it and not so smiling faces on it. And they also, you know, they could put this into a bucket, you know, when they were about to leave the room. I mean, that's a tiny tool. I mean, then you can do something like quick, quick interviews that are not so time consuming to do. And in this publication about tiny tools, you find a lot of that and you also find a lot of that, that is really participative, you know? And I think that's very important for when you want to, when you want to engage your community. Use a tool that is, that really engages the people. You can, I mean, you also can do a quest, you also can do a questionnaire, but it really gives the, I think it's important to give the people the impression that their opinion really matters and that you really values them. And that can sometimes that is more the case when you know you use a more personal contact and more personal communication than when you just hand out, I don't know when you just hand out questionnaires, but still I'm not saying that you should not do that, but you also should always should emphasize that that is something you're not only doing because you want to check on those people, you know, or you know, you want to know something, but you always should tell them that with their answers and the time they put into filling and this questionnaire participating in the interview or whatever, they really help to make this program or this office better, not only for them, but also for the rest of the community. And I always find that the acceptance of people, you know, participating in evaluation increases enormously when you tell them, you know, that it's, you know, it's not about them, it's not only about them and, you know, like checking on them, but it's also because their voice really, their voice really matters. Yeah, so that was about it. I talked a lot and I hope that there's some more questions for the last couple of minutes. Was there something very clear, something very unclear? Should I, I mean, I can again, you know, I can focus a little bit on the different tools if you want to, do you think now that impact orientation is too cumbersome to even think about that? So I, well, I don't see any new questions. The question is whether that's because there are no questions or that's because the website doesn't show me. Okay, here. Okay, so the question is any indicative questions is any indicative questionnaires, any direction for controlled group trials? Probably that's now a little bit stupid to say, but actually my direction for a controlled group trial would be seek expert advice. First of all, make sure why you want to do that and then ask someone who really knows about that and then get consultation, really. I mean, because it's honestly, I mean really the very few of controlled group trials, I've seen, that's just big, that's just really, it's a lot of work. And by definition, you can't do that on your, I mean, when you are, I assume you're working within a museum or a nonprofit organization. By definition, you could not do that yourself because just the quality, you know, you have to have someone neutral from outside who would run that. And so yes, share, seek advice and think thrice whether you really want to do that. You know, I'm not talking against that. I just have the impression in most organizations, something like that would be really, really overwhelming. And I see, just to implement really small evaluation tools is already such a big endeavor for most organizations. So why not? And actually, that's one of the suggestions I forgot to make earlier, start small. Don't think, oh my God, now we're going to evaluate our whole museum and we are going to work impact oriented from the basement to the fifth floor, something like that, start small. Just pick a small, it's a small task. Take a small unit of your organization and let them start and let them share the experience and let them learn what works first before you engage and before you engage everybody. So that's what I suggest. Also, I think it's a process that you cannot only do from what, neither from the bottom up or the top down. You know, if like the management does not want you to shift your focus on social impact, then it will be difficult for you to do so. And at the same time, if the people who work in the organization are not convinced that that is something that is good for them and the organization, you will never be able to implement that. Okay. A lot of information here. I'm sorry. The PowerPoint will be available. I just click briefly. Okay, here, I mean here, you will get the PowerPoint. The social impact navigator is our resource that you can print. And we also have that online. So everything I said, you'll find there in better English and more interactive and in a better and you can go through that in your own time where basically the whole impact-oriented management circle is there and you can download a lot of material there. You can download templates and questionnaires and I don't know what. So that's probably a resource to go to. And then the resources I talked about about the different things I talked about, they will be there as well. It's all except one. It's all online and you can work your way through there. So I think you're busy for the next, till the next conference you will need the time to digest.