 Yn ymweld. Yn ymweld, ymweld. Yn ymweld, o'r bodi'n ffordd. O'r gyflawni, oherwydd yma, yng nghymru. A ydych chi'n gwybod yw'r gweithio? Yn ymweld, mae'n gweithio. Mae'r gweithio'r gweithio. Mae'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio. Mae'n ddweud yn fwy o'r gweithio. Mae'n ddweud o'r gweithio'n gweithio. Mae'n ddweud yma. Mae'n ddweud o'r gweithio. Mae'n ddweud o'n ddweud o'ch gweithio'n gweithio. Mae'n mwy o'n ddweud o'ch dealladio, a phan o'r gwisig y casio. Mae'n ddweud o'ch clywed o'r gweithio o bobl. Dwi'n ddweud o'r geth, yr hoffa o'n gweithio cyfeinio gweithio. Dwi'n ddweud o'r gweithio. So mwy o'r bod yna'r ffroedd. Roedd yn gweithio. Roedd fyddnod o'r gwlaeddu'i gwneud o'r ei ddeuill, I've only been with my society for about six months. I started in October and I should say that I've come from a political science background. I really knew to a lot of the kind of really digital techie kind of stuff. I don't know how to code. I don't know the difference between general and javascript but what I do know is that citizen engagement, direct democracy, transparency, accountability, these things are more present concerns than ever. My research experience is in trying to make sense of what's happening, why it's happening and what can be done to change things, hopefully for the better. So, why do you research in this field? Why are we at my society even bothering with research, aside from the fact we've got a lovely grant to do it? Well, one of the key questions we need to ask is what we're doing, even with doing it all and you know, echoing what Tom kind of said here. Many of us in this room I think are quite evangelical about our medium. We're very, very sold on what we're doing. We believe it's a good thing but simply believing something is good is not the same as it actually being good facts. So, we need to be able to show that what we're doing can value, that it's actually making some sort of difference in the real world. Otherwise, what's the point in doing that? And in order to do that, we need to take a measured and actually quite dispassionate approach to figuring things out. We have to put our biases aside, collect evidence and look at what the evidence is telling us whether that's what we really want to see or not. But why research impact specifically, this is a very specific comment we're talking about. Impact, we're not just talking about the very broad things you could research, we're focusing on impact. There are many other kinds of things that we could choose to look at. So, why is it so important for us to measure this above all else? Research and user experience, you could argue, is just as important. We don't do enough research there and we know we should. We could also focus on researching success or advice of our marketing efforts. No marketing, no users after all. But I believe we all know that ultimately all these things are secondary to figuring out whether what we're doing is actually beneficial. What's the point of having a better user experience if the outcome of using a tool is nothing much at all? What's the point of getting more users if our tools are actually not effective or not providing them with the things they need to solve their issues? So what we need to know above and beyond anything else in my opinion is whether the current users of our tools are getting benefits from them. What sort of benefits are they? Private, public, short-term, long-term, ephemeral or profound? Wouldn't it be great if they were profound? So I don't have long, so this is what I want to say to you today. First up, why impact research in civic tech is so tough, it's really hard, it's not easy. Impact is hard to measure, it's a fluid concept, it can be looked at in a multitude of different ways. We tend to get caught up in the digital aspects I think of what we're doing, the idea of kind of the internet. But in essence, if we kind of boil it down to really what we're looking at, it's a social science problem. We have to use the limited and often lacking tools available in our social science toolbox to actually conduct our research. Even our definition of impact are open to interpretation. Impact is defined in the dictionary as a marked effect or influence. We need to consider that our work impacts upon many different constituencies. Individuals, citizens, governments, decision makers, funders and other civil society organisations. Getting the data is really hard. Even just getting information that you can actually analyse, it's actually really difficult. We've got things like surveys, analytics, AB testing, interviews to gather data. Often the data we really need is often kind of locked away somewhere or not even able to be collected. By this I mean that it can be in the hands of people who won't or can't share it with researchers like myself and like many of you in this room. We know that sometimes our data will actually be skewed by certain populations being more open to taking surveys for instance or making themselves available for interview. We can try to correct for this but we'll never get a flawless full picture. Everything we do is always going to be open to interpretation because of these factors. The impact of civic tech can actually be more subtle and organic. It sometimes might not even manifest until a significant time after our tools have been utilised. It's impossible to record and quantify all of the kind of ripples that our tools create. So, we've thought about this. You'll be pleased to hear. We're going to do experiments. We're not writing long theoretical pieces. We're going to get our hands dirty. We're going to try and pick up facts. We're going to try and refrain from imposing our opinions and preconceived beliefs on those. We're very much looking at partnership. We want to look at tools that are being used around the world, not just the ones that we at my society run here in the UK. We're looking to be open about our results. We will of course be looking to publish in peer review journals to add weight, to add legitimacy to our findings but we know that this takes a really long time. Those academics in the room will know that... five minutes, okay. The academics in the room, you'll know how difficult and how long it takes to get findings published in the journal. We want to share our preliminary findings with you as we go on our blog. We're not scared of finding out things that are negative. Again, echoing what Tom said earlier. If we find that our tools are really not making a difference or they're having a negative effect, we'll still share those results. Perhaps maybe sit down and have a long, hard think about what we're doing and how we can change for the better. So, we are looking at impact using two axes. On the one way we're looking at time, so before people use a tool, during and afterwards. We're also looking at actors, so individual users. We're looking at civic technologists and the kind of social, political context that these things exist within. We're doing this because impact occurs at multiple points in a user journey and it's shaped by all of these different factors. So, we've developed a strategy based on a Hannah's Matrix model to timeframe these points of impact. So, I've been told to brace yourself for a scary table. Here it is. Don't strain your eyes to see this. There's a copy of this in the research strategy which is in your delegate pack. So, if you're at the back, don't strain your eyes. But this is our framework for researching impact. We are basically planning on using these boxes to prioritise and frame our research activities. We'll do multiple research activities in each box, focusing on different relevant research questions relating to that point of impact. So, we want to fill these boxes with knowledge about impact. At the moment all we have in these boxes are questions. We want to put answers into these boxes. So, just as an example, this is box C3 and there we are planning on conducting a study in Latin America, looking at how governments over there are reacting and responding to the advent of civic technology. Are they integrating with it? Are they trying to subvert it in any way? What's the impact that civic technology is having upon the kind of institutional government structural environment in Latin America? So, that's just one example of one experiment that's going to happen down there in that box. There will be many, hopefully. And as I said, we're looking to work with partners across the world. So, if you have sites, ideally with some traffic, that we can work with you as part of this big research journey, then please do get in touch with us because we would love to hear from you and love to work with you. So, just to kind of tie this up, when it's all over, we expect to be able to provide intelligence on things like costs, on things like impact measures, whether impact in specific areas is positive, negative, neutral, and whether, hopefully, there's value in our work. Finally, I just want to say that we're unlikely to develop any grand unified theory of impact if that's what you're hoping for from today. I don't think you're going to walk away having got that. The very nature of impact is so slippery. It's very difficult to completely pin it down using the tools at our disposal. So, what we will have, hopefully, is a broad understanding of where impact can and does occur, what that impact looks like and how best to collect the evidence to demonstrate it. We hope this information will be of benefit to the whole civic tech community and will inform good decision making in your own organisation. We're not looking to keep this to ourselves. We're not going to be putting our arms around at homework. We want to share this with all of you because we want this to be of benefit to everyone.