 I argued with Rebecca and others that there should be at least some point today where you could simply pick each other's brains in a big group because there's so many clever people here. My only role here is I'm going to chair and although there is a big theme of democracy going on today I want you to know that I will be fascistic on the point of good questions. The goal here is that people think of questions that they themselves really personally care about, about research and your own faith, your own needs, your own organisations. There shouldn't be sort of theoretical questions, these should be like things I need to solve. As a quote, a friend of mine for anybody who doesn't know in this situation, a question is an interrogative statement that ends with a question mark and I will jump on people who appear to be making speeches or other than questions. I have put only some seeds of ideas here. These are merely to sort of wet your appetites. I hope that over the course of 30, 40 minutes we might maybe get through seven, eight kind of good, interesting little mini debates and I am not proposing to answer any of these questions. My lips will be zipped, it is for you across the room to put your hands up and go I have a view, I have an answer, I have a theory. I am just timekeeping and I am a chair. Would anyone like to begin with any question? I see one immediately at the back, Gemma, do you want to give that to the friend? Do you want to do an intro as well? Fran Perron, I am the founder and director of the IndieProTrust and we fund lots of civic tech projects. My question is about marketing and impact. We see lots of civic tech projects fail not because they weren't good but because no one knew they existed. For anyone running a project, what is the problem in applying for marketing budget? Is it that you think funders won't give you the money or you don't think it is your responsibility to market and how can we look at that from a research perspective? The cue now is for anyone who would like to go and answer it. Please give one word intro when you are saying anything. Thank you for the question. Eric Barrett from Jumpzart, Georgia. Jumpzart, Georgia, the country. That is a great question. Our experience is that when we have requested funds specifically for social media advertising which seems to be the most effective way of reaching our target audiences, we get turned out. And in addition to that, when we request funds for midterm evaluations of impact in order to augment a project halfway through, that often gets pulled out as unnecessary when we think it is vital not to mention including post project evaluations as well. With Civic Tech we often find that the monitoring and evaluation seems to be a take on for donors that is unnecessary. And I would like to continue posing a question in regards to that. How do we speak to donors to communicate to them the importance of adding evaluation and impact into the projects? Just sticking on the first question and building the second. Anyone who would like to speak to Oscar? Oscar from Mexico. So I think we have realized that it depends who you are talking to. So if your tool is for a niche, like you are giving tools to other organizations, you probably just need to communicate effectively between that group. Otherwise, what we found out is that we just sort of bootstrap all our communications and we hired someone specifically for that because we figured out that was very important. It doesn't come via the funding. We haven't got to that part, but I think it would be really interesting to talk about that. Any more on this one? We'll do a couple more answers and then move on to the next question. I'm here from Belgium. We often or almost always work together with media to establish this and we see a huge impact on the different projects, so cooperate with big media players. So that for us is an important ingredient to get right. So do you want to pass it to Sean and then to the left? Sean from NSMS. Advertising for us ends up being one part of a larger problem, which is getting earned income from the work and explaining earned income as separate from or in complement to kind of the grant-funded work that we normally do. For us, getting more users is great, but getting more users in a way that validates an increasing flow of revenue makes a much bigger difference. So I would say that advertising is one part of it, because it would also be a facilitated business planning process where the activities that the organization is undertaking also drive a greater stream of revenue and less ongoing dependence on donors. Hi, I'm Anna from Poland. I see two points where we sometimes should change our philosophy about promoting our projects. First thing is like we make a lot of work to open data and to give tools to the citizens, and citizens usually are just lazy and they do not use these tools. We also need to show some interesting conclusions, examples, and analyze these data and show just conclusions. And the second is also need some change of thinking of our donors. It's like we always want to have good results on our website. Like we want traffic on our website, and we don't make work to publish something on media, and we do not count these people who read or who see our publications in media. And this also is because our donors want us to show the traffic on the website, not somewhere other places. I'm going to take my chair's property and ask for a new question on something else. Catherine? I wrote it down, so I'd be very concise. So I've been hearing this morning in Luke, where's Luke from Australia? You highlighted this tension between academic research models and the sort of iterative data-driven model that you saw in use in the GDS. So I just think that's an interesting tension. My question is, how do we develop robust methodologies that are both rigorous but also timely and agile and can be implemented into our current work streams instead of running alongside? An academic time scale doesn't work for the kind of work that we do. Any answers? So, there was one here, and then one over there. I think this is a really great question, and although I certainly don't have most of the answers on this, I think some of it would be making these tools scalable. So even if on an academic time scale, the development of it might be quite long-term, once it's actually implemented in 30 different projects or something, the actual implementation could be very short and work on the time scale of an organisation. So if those tools are somewhat standardised, the development process won't be quite so harsh for the actual project. And then there's someone over there. One answer to your question, Catherine, is that you get Jonathan to do it because he can do anything in SPSS in about two and a half minutes. Hi, my name's Martin from Brighton. And one way is sort of an evaluation framework to look at testing models right from the beginning. And so I think the World Bank recently supported a guide on this, a sort of practical guide, which I've got some leaflets found, which is how to integrate research and evaluation from the beginning. And I think it's a gentleman from Georgia about actually doing that. And if you integrate it into the sort of product development stage, it's more difficult for a funder to cut that because it's sort of integral to the whole initiative. And I think that's sort of a practical technique, depending on the openness to the funder about how open they are to sort of emanate. But I think right from the beginning is the critical thing, so that it doesn't get cut, so you can do that sort of iterative development. Next up is William at the back, but I want to point out the gender balance in the answer is bad. Not for the first time, I apologise for the gender. William Perrin from Indigo Trust didn't talk about local. We developed, and we faced this problem in evaluating a series of interventions in community websites for the Carnegie UK Trust, the UK branch of the American Plants Breathe organisation. And one thing I was determined to do was not to try and reduce it just to data. The impact, societal impact, has a very strong qualitative component that you can only really get down to by going out and interviewing people about what they think the impact of the project has been. That doesn't always fit with some people's lust for data, but on many of these projects, it's absolutely bonkers just to take a data-driven approach, because there isn't enough, you cannot really measure societal impact through data learning except in very rare circumstances. So we constructed an evaluation framework that was heavily qual and quant for the funders that they were very pleased with, because they bought the argument that these are relatively small projects. Quantitative work is actually quite bogus on sample sizes this small. I'm looking for a hand that doesn't have mail pattern folding. Jessica. Jessica from Zelendo. I would like to comment from a practitioner perspective. I think all the academia coming out to us many times don't factor in that they are the least of our priorities. And so if they want to ask to consider them in our work, then they should also factor us in right from when they are writing. Say, is this a partnership that you'd like to be part of? We want to do this for a number of years so that also we can actually see the return on investment for the time that you're putting in. A lot of times I get university students and I actually turn them down. I usually tell them, if I can't see what your work is going to have, impact is going to have on my work, no, I refuse interviews. I'm not doing any research to, doing research in their team, I would really love to hear from you. So if there's anyone, you know, join in now. I'll take a microphone. My name is Lisa, I work at GDS. I think it's really important that we don't conflate kind of social research with design research. There are different kinds of research. They require different kinds of methodologies. At GDS, the vast majority of the work that we're doing at the moment is design research. So we're working with citizens to try to make services that work better for them. That's really different to measuring impact across a whole population. So you use very different tools. You can measure impact across a whole population in an agile way. But you can certainly do design research in an agile way. So you need to think about what actually is the research question and what is the right kind of research, and then you can think about whether or not you should get an academic to do a larger scale piece, or whether you should get a researcher to work closely with your team to help you improve the thing that you're making really quickly. Thank you very much. Is there a new question from the room? No, no, no. How about gentlemen right back? Yes, James. Hi, James and Katie of Open North. So my question is about methodology. So a frequent methodology is you have a good control group, an experimental group, and hopefully at the end of it, the experimental group is more engaged than they were before. But I'm wondering about some of the influences on that control group that might make them less engaged, which isn't typically measurable within the time frame of the experiment. Like Chris Kase was mentioning earlier, once the money in politics is actively disengaging people. And so the question is about the net impact of our civic tech. If more people are becoming disengaged than we're capable of re-engaging through tools, well shouldn't we be measuring the net and not just the incremental difference on our experimental groups? Is that a good question? Yeah, like how would you design that as your methodology? A methodology question. There could be problems with the control group. There could be problems with the control group. I see the issue that you are you're raising. I think the point is to measure your experimental group at multiple points in time and to also assess that to see if they are changing because your control group may be influenced by other factors, but your experimental group should be affected by the technology that they are being exposed to. Does that answer your question? Yeah, probably. Since that may be flummoxed by the question, so I might move on to the onto a new one. So, thanks, Hand. I have a quiet voice. So I just wanted to pick up on the engagement side, if I understood that right, because I think in a way it sort of reads back to what I wanted about the conference as I was preparing to do talks this morning, which is our definitions around civic and engagement and participation and this came up in the session in the other room. And I wondered sort of off the back of your question to ask another question, which is really what are people understanding and conceptualising defining civic as being? So I think it's interesting to say that we're not participating and are we looking at sort of trying to get them to participate using tools? Yeah, that's one way of looking at it, but the second way of looking at it might be are we going to the right places to where people are trying to engage and express themselves as citizens? So other aspects that might be more constructive in some ways, like volunteering, participating in civic life in other ways. So I wonder, yeah, it's a question about definition, what do other people think? Any strong views? A room full of civic technologists with no views on the meaning of civic. This is a phone shell. Well, I'll say that what I think we put too much focus on is often implicit in the way we talk about this work is that civic equals voting and that that is the most important or the most central act and if people aren't voting they're somehow not simply engaged. And I think that's a real failing research. I bet if you looked at the indicators in all of Shelly's studies one of the major ones that we're looking at is, you know, going up or down is voting. And it's just, it's maybe necessary, but insufficient. Particularly, you know, how do we judge this work in non-democracies? Do we just ignore non-democracies altogether when it comes to civic participation? So that isn't an answer, but it's what it isn't. Right. At this point, in other contexts such as Latin America, Asia, Africa civic technology is not about voting. I mean, it's mostly about engagement with public services. It's mostly about engagement with other kind of actors rather than politicians. So I think that I just quickly come back to this because if the usual definition in American context of civic tech is voting there isn't which will kind of make the point to broaden this definition to other kind of interactions with this thing, or with the ornament or so on. Any more views on that? Or any totally... Can I respond on it? Tell a room of civic people. I feel a bit cheated. I want some more definitions. But my tentative sort of thoughts on it are that it's to me involvement of people in the decisions that affect their lives and those decisions are sort of related to an institution. Sometimes that's government. Sometimes you express it through voting. That happens once every four or five years if you're lucky enough to live in a democracy. So maybe we ought to be looking at other ways that people express themselves and trying to sort of define this a bit better. Hi, I think just based on that definition one thing we've started observing in this in Kenya is when that element of how it affects your life as related to an institution then is reduced to a personal thing. We're like well, I can't think of this huge number that is about taxes that are being misappropriated or whatever. So it comes back down to self-preservation. So it's that, and I think it goes back to what someone raised earlier about opting out altogether. Because in the grander scheme of it being about institutions that metric becomes problematic. Or when just people want to beam their thoughts but not be tied down to saying this is in reference to trying to get an institution to be held accountable. The disillusionment that plays in that especially from the citizen side is really interesting. I think it's partly why there isn't a standardized definition. Because it's very contextual and it rolls with different events and different moods I guess. At least from my observation. I'll just start talking about that. I'll read it because it's a definition. Digital citizen engagement is defined as the use of new media digital information and communication technologies to create or enhance the communication channels that facilitate the interaction between citizens and governments or the private sector. I think the private sector is an important addition to civic engagement. Identity venture here, the definition that is going to stick is actually one that will be made by people in this room who do good work. That's the way that we learn to use words and so we can't resolve this now but I'll have like a couple more and then definitely a new question. So James and then a very sensitive question. So the definition is current to the previous comment about whether opting out should be tools that help people opt out of a political system should be considered because right now most civic tech tools are giving people more information about the current system they're living in allowing them to engage with that system better but if say like nextdoor.com lets help people engage with their neighbors more so instead of relying on their government doing things better for them maybe they just set up these like local communities to help each other but is nextdoor considered civic tech if the definition is about empowering people to effective institutions? And then Chris should move on because this is an endless conversation. Absolutely. So just to say that you need to really differentiate back to Catherine's point about there's part of civic activity that is about trying to influence the decisions that other people are making about your life but the other piece that I think is growing much more quickly is civic tech around actually doing things or becoming an actor yourself about becoming a problem solver not trying to influence people to do things on your behalf but the real tools that have been created that have been incredibly successful have been about empowering people to become civic actors themselves not allowing other people to make better decisions on their behalf. So I mean that definition I think is sort of only describes really the one side which is using these tools to influence other people to make decisions and there's a whole other part of this work that is about empowering people to become actors in and of themselves. Next year there will be three rival research conferences on the same day with three rival recognition. So a different question from somebody. So there you go. I'm a design researcher at the Royal College of Art and we recently did a project where we used a community mapping platform to engage people in local issues and found that what we found was that it didn't really work as an alternative to traditional engagement methods but needed to be part of a wider engagement process so still using meetings and face-to-face conversations and workshops. So I wondered whether people have examples of how other civic tech has been integrated into more traditional community engagement processes. Some of this might be repeating a bit from my actual talk this morning but I think that we've also really really found that it's the offline things that matter and so we found that sometimes for example technology can be used to amplify the voice of people that aren't offline. So quite a few of the groups we work with they'll get testimonies from the communities that they're working with and they'll hold interviews and meetings with them. But it might be that they for example can use social media to amplify the message that it reaches politicians or it might be that they can collect all of the reports coming in through a particular reporting platform and then they can use that to negotiate with political leaders and say okay there's evidence here and I completely agree with you I think we've found again and again that it's the offline things that matter but it's actually the tech that's kind of amplifying the message enabling information to be exchanged at a lower cost and faster at a greater scale but it is really this offline work that matters and often that's individual relationships which is quite hard to quantify but often it is they've met somebody in government who is interested in this work and is willing to talk to them about things and again I think it's about showing government how you can actually benefit them or it might be government might be others but sometimes saying look we've found out that through using this platform actually one of your subcontractors isn't working too well and you can now go to them or you know we can actually see that citizens are interested in this what are you going to do about it so it's often kind of finding what's interesting for them is that Ed and then on their trust who are just exported to a sticky world so yeah it's interesting they're trying to one of the ways they're trying to use their platform is part of the planning process for local authorities and I came away thinking actually I'm not sure the incentive is currently strong enough from the institution they have to use some consultation but they're not really helped to account enough to my point of view what the quality of that so yes sticky world can help them do that kind of consultation better but not a lot of them are made to care about the quality of the work of the hospital so I think there's a gap there which they're trying to try to fill and that's going to be a problem which I'd say lots of different scenarios will map the institutions not having the incentive to really engage with the quality of the work I'm going to pull this slightly closer to research especially because we're almost out of time and what I would really like to know is whether anyone has any questions about any specifically about any research project that they think that they will be doing might be doing in the next six months where they just would like some advice from other people Hello my name is Becky Hogg I am producing a piece of research that I hope to publish in October which is a piece of academic research I would call it grey literature and I'm interested in talking to anyone who has a story to tell about the impact of scale of open government data please find me if you don't see me directly that doesn't need a long answer there were some more hands over here sorry Hello Simon Lindgren I'm a professor of sociology do research on digital activism and things like that I think it's really inspiring to hear the talks this morning because they were about the need for knowledge about the actual impact of such initiatives and I say from the perspective of internet research there's a huge gap between the real world and the theories that guess it's going to be awesome or it's going to be a catastrophe with these technologies so basically to make this into a question I would say also with Shelley's talk I think it shows that the shortcomings of those studies have to do with the fact that there's a gap between what actually happens and the researchers doing this guess work so in my research group we tried to bridge this actually so I saw your little buffet of questions there if anyone would like to partner on doing studies we're actually looking for cases to follow and to study so you can look up my research group my publications and so on and I'd be happy to do discussions as well so to make this into a question do people feel from the perspective of the civic technology projects that this gap exists as well sorry can you put your hands around for the back so Martin and then the lady in the far corner yeah that'll be time okay so I have the feeling that doing interactive user experience design that integrating interviews during the design process it's very important I wanted to know if you share that feeling and I wanted to invite you all to include this process in your research and in your project because I see many people are talking about doing research separate from development I don't think it's very I don't know this is one of the common themes in this session and then the last question hi I'm Sirupi and I work on something called Kallivali in Kathmandu in Nepal and we're in the initial stages of setting up a research project that works on what are the infrastructure issues say in a really small locality right and how do people complain about it so you have your citizens you have your neighbourhood associations and then you have the ward offices so how do people complain and how does the information flow so what are the channels for the flow and who are the different actors involved so if anyone has any sort of methodological advice on how to set up this research that would be very valuable and then the next step we want to add to that is once you know what those networks already are where again something like crowdsourcing or civic type or any other sort of digital non-digital technology help that process so I suppose there's two pieces to it and any advice on any of those pieces would be appreciated do you want to say anything about sort of what comes next um so now it's lunchtime yay can't be that announcement