 Best of it, all of it. Okay hello yeah am I okay uh so we just taste what consensus uh no no sorry we just know what consensus tastes like and uh now we should we have the consensus in our stomach and now we need to have consensus in our mind and brain so now uh I'm going to I'm going to have the sessions uh divided and in the order of the veteran process from stage four to stage stage upon stage one to stage four hold on okay this is the uh front to start with I is it kind of too loud okay so before we go into stage one I still need to remind you the outline of the process and maybe some tools so that you can understand and give a have a glance at what we will how we use these tools and this is the process as a reminder from stage one to stage four and also the online opinion collection and consultation meeting so for this session after the introduction of the tool uh presented by doshana and shuyang and then then we will have a um an exercise one as a topic poll yeah so this is the what will happen uh so we have many tools uh usually uh generally are our open source and we have three concepts behind these tools so this I mean sorry it might look too small okay okay this is a graph uh drawn by shuyang and you can see that uh from issue let's say you can see the proposal stage from issue qualified issue then stakeholder discovery we use haphaz but it's like uh used as a way to document everything on the internet so remember uh whenever you whatever you do on vtel or whatever you do on any civic participation platform you better uh record it online so that the participants can keep track of what's happening okay and then uh like we also use like slash share and um say it and youtube and discourse also it's all the way it's also the tools that we use for documentation and uh yeah you can have these uh so we have those materials on hackforters so later you can look into details how we divide these stages and use what tools for each stages for each stage and then uh these are the concept behind each tool they are interpretation facilitation and documentation and for interpretation uh it's trying to bridge the gap I mean the communication gap between different various stakeholders because people are from different backgrounds and they have different stories and experiences so we need to interpret the terminologies, jargons and vocabularies in various context in various contexts and also facilitation uh it's to facilitate in the engagement and deliberation uh to from from conflicts to consensus so interpretation is to get the idea across to each other while the facilitation is the the art of collecting opinions from the public and as I mentioned earlier uh the documentation is to also highlight the importance of transparency and accountability so it's really useful if you have documented everything on the internet because it's like we do on the hack folder we have the materials online so you can also always refer to those materials just by clicks and links yeah so uh the idea is the transparency because also the process and also the data allows participants to understand the decisions taken through the VTARM process so and also uh strengthens and builds trust among stakeholders and also among like facilitator between the facilitator and stakeholders so uh the transparency is the core and key idea behind these three concepts so okay uh this will be uh the stage one uh the proposal so um to do the proposal uh like uh like the NCI case firstly we need to develop a good proposal so it could be a like a top down and also bottom up a proposal like the uber uber case uber x case and also the bottom up proposal like the NCI case um so here um our our rule is that if you want to submit a proposal you need to participate in the mini hackathon and you got a submitted I mean like in person and then you can document it on hackpads so we have the recording on the internet and people that participants that cannot join that specific mini hackathon can then read the the transcripts later on hackpad so first of all we need to have a proposal and then uh the participants need to come up with the good name and title and also it has to be discussed with the community contributors at mini hackathon so take non NCI case for example um we try to translate uh non consensual and also the the term pornography in Chinese because it's really not that uh instinct uh this instinctive instinctive in Chinese so um in terms of non consensual we have the so if you read Chinese we have the translation down there well if it's literally translated it could be dissemination of one's bodily intimate images against one will but it's it's really long and it's not that actually not that friendly well personally I feel it's not that friendly for everyone yeah but we have to be specific and to be politically right so uh to translate the non consensual and and the pornography word but pornography is somehow maybe not that positive itself so later we use intimate images uh and also the Ministry of Justice uh suggests that they want to not limit it the limiting the idea of pornography so uh they wish they could use bodily intimate images instead of pornography so but that's in in the translation of Chinese so this is uh just want to demonstrate that we need to come up with a good name or title so that uh when you have that title on the web page on vTaiwan or whatever platform you have the title itself will be very important and critical for any kind of stakeholders to have a first glance of the issue yes because the title will be like the attention grabber so it's really important to have a good name and title and then um after you have a good title then at mini hackathon you also need to discuss this topic with the participants and you need to narrow down the scope of the topic I know that any kinds of issues may be very I mean the scope may be very broad but uh we need to focus on specific sometimes if we have so many participants from different backgrounds and from all walks of life then we need really need to narrow down the scope sometime so that uh people get to be focused on specific or the top issue or the core issue of a huge topic and also ensure the scope is fit for the purpose and for some case like NCI case also we need to try to uh figure out a strategy like the legislation strategy because it depends on different issues and sometimes the I mean the controversy among stakeholders I mean it depends and at different level so uh you need to maybe sometimes map out the strategy for for the later for the later process and also uh this is really important you've got to find out the competent authority of a specific topic so that it means that the competent authority is willing to take the responsibility of resolving the case and to to act so um it's very important that um maybe here like we like we do in VTHA when we have Audrey as an extension into the central government so that we can ask or to suggest a specific competent authority take the I mean own the issue yeah but so this is how we connect to the government and before we find out the competent authority or before the company authority said this case is isn't really can be initiated so finding a competent authority and the authorities also willing to to take on the issue will be like a prerequisite of the stage one yes so um this uh we use hackpad as a document documentation tool to to record everything especially the it's like a meeting having the meeting minute meeting minutes of mini hackathon at hackpad so this is the original screenshot of a proposal of NCI case that we have the name VTHA one and also mini hackathon in chinese and the dates of that meeting and also some and the content itself um and after after this those step six steps then we should need to choose the right tool for opinion collection so and this is a like a instruction of choosing a tool so if if the issue is not that clear and the structure is still unorganized then we can might use police and because police is much liberally open and the stakeholders or the participants get to post their own comments so it's a good tool to get an idea of what the atmosphere is like in terms of a specific issue and if the clear the air is clear enough then you can use discourse because it's a more organized uh like it's like a forum so you can have like a bulletin and you have the topic and the title and the like the issue and the sub issues and break it down into organized details so you can use discourse but it's just like the traditional forum but yeah this is for a more organized issue and but if you are not sure you can go traditional like a online questionnaire so you can use slide or a type form but type form is not open source right i guess so okay okay okay but you need to pay if you want to professional version uh yeah and um later we will use police as the experiment tool to to to do the trial trial because i guess it's the most unusual in the fun one so uh we will use it as a default tool to do to perform to perform the experiment and um after okay steps step 8th uh we can publish the represent to publish the presentations documents research paper made from research team if the competent authority have appointed a specific uh research team then uh the research team may have a presentation or a study or papers so you can post it on your platform and let the participants uh to understand uh what's the background or the legal uh backgrounds or some other uh information that they need they need to need to know so if we have any information uh made from research team or competent authority or even the proposal proposer himself or herself then we should post it posted on the platform so uh all the stakeholders and participants can have the access to these uh studies or presentations and they can easily understand what might be discussed later uh okay and then there are the tools uh here uh we have polis discourse uh slide on a type form and these four tools are the most uh used one on v taiwan and in here in polis uh i'll need shona to give us a brief introduction on polis and then after after shona's presentation we'll have suya to tell us to give us a brief introduction on other tools so shona okay can we ask questions about the the process before we jump into the tools or yeah totally you'll take me away to set up okay cool so okay my first question is on slide 9.6 which is find out the competent authority what if you don't find an authority that is willing to engage hello from my own experience i haven't had this this kind of issue before but for nci case uh originally because it was about the criminalization of the actor so we find that ministry of justice is the right competent authority for this case and they are willing to take the responsibility but um because it's uh a bottom up proposal so they don't really have pressure from the top uh top uh high ranking officials so i guess this is why it moves kind of slow i mean the process moves kind of slow but after uh like a year round um process then at this topics began began to expand lot i mean the the exchanges become broader so there are maybe more competent authority needed to participate in including like uh ministry of welfare and then health but um yeah well from my own experience they are all willing to take the responsibility because they do feel i mean they feel the need from from the participants especially those uh participating in mini hackathon so that's what the government will do if they do face and face the pressure from from the general public to agree you have anything to add i'll just add that it's a net reduction in risk if they participate early in the mini hackathon um because the process promise a net reduction in labor in time because it's crowdsourced but uh the threshold is not just about reducing the cost but also reducing risk uh and the idea of mini hackathon is that individual agencies they can participate on a individual basis without their minister's approval and it's usually seen as a risk reducing move because it's certainly better to participate at that point rather than you know when there's thousands of people on the street so yeah okay that's that well yes so what do you mean with research team in the slight like police like oh sometimes there is okay sometimes the uh the competent authority will have but if they originally have already wanted to bring up a proposal they will also have a like a funding for the research team and they will appoint a research team to help them do the research work so there will be like a ally of their their own the the ally of the competent authority so that's yeah that's appointed by the competent authority so and also um because they are most they are doing most of their research work so they are also uh obliged or they need to participate in mini hackathon and let them let them uh keep track um keep track of what's happening yeah i need to have a deeply involved with the mini hackathon and the participants sorry um i will do a quick run through of polis which is one of the the main tools that is used in the v taiwan process but before i do i wanted to sort of put up uh or share the business card of this incredible person that csn i met uh when we were in taiwan so this is lucha was a business card and she runs uh she's been running participatory um methods or researching and implementing them for a very long time um and i think this is this is sort of very important to remember um it says behind every technologist there is and should be much more than your technology behind technology there is a set of values informing us it's it's pursued um and sort of thinking about polis in the slide polis is just a tool it's a discovery tool that that is incredibly flexible and can be used um in many different contexts and processes and the processes that uh that people choose to implement it in is is really important uh which is something that's really wonderful about the v taiwan process i'm just going to quickly do pull up a couple of links so to show you what polis looks like um so this is the polis is a conversation platform and uh when participants go on the platform you can agree disagree or pass on sentiments or statements that you see and then if you feel your sentiment isn't reflected you can add it in and then the next set of people uh get get to vote on it um moderators tend to put in about 20 statements and then the conversation just takes off uh usually you there are conversations as large as you know a thousand statements mostly generated by participants so that's one of the nice things about the tool is that uh participants are actively creating um the questions or statements that go into the into the conversation this is an example this is just a a poll that's embedded into into a website the the labor party had used it to define political agenda so you see the statement come up agree disagree or pass and then your own statements this is a participant view i'm showing you uh what the algorithm does um once uh once all of the information is collected it's a giant matrix of agrees or disagrees and then a clustering algorithm to look at how people can be grouped based on on their opinions and the grouping is not done on any kind of demographic information it's just purely based on opinions um it's a conversation space that's mapped out and statements that people tend to agree on or disagree on um are closer together in that space and participants who tend to agree on the same things or disagree on the same things are also closer in that space and statements really don't matter it's the relationship between the the participants and how they feel about these various things or the various statements that come up um and then you can see varying number of clusters uh the algorithm also uh shows participant statements based on participation itself so if say i'm the hundredth person entering a conversation the statements that i will see with the highest probability are the ones that have are new in the conversation that haven't been answered before that we do not have any data on um you can imagine that this is useful when you have a thousand-odd statement conversation how do you how do you you know decide who gets to see what so those those are the statements with the highest chance of being seen the ones with the least priority are the ones that a lot of people have seen but they're passing on so it's sort of an indication that maybe it's not a clear statement maybe it's an irrelevant statement um but then by the time the 200th person has come in the statements that have been seen a lot are no longer being seen a lot have haven't been seen as much and so they bubble up again and so the conversation uh the statements keep moving based on participation um in the meanwhile there's uh while that is happening there's a real-time report being generated uh where all of the data is visualized i'm pulling up a report um from a conversation run by the canadian heritage department you could see the summary statistics they were trying to craft licensing around um around visual arts um questions that the statement is is sort of very broad the topic is on the visual arts marketplace they solicited information from artists sellers of art and then opened it out to everyone else and got their sentiments on um on the on the art industry the summary statistics are up here and then you can see the snapshot of the entire conversation each dot here is is a statement um and you move from the most divisive statements to the ones with the most consensus um and so you can just move across a conversation and get a snapshot of the entire conversation um it's sort of useful to to see this many times because we don't we don't realize how much we actually agree on and and that's a good place to start a conversation uh so Joe and Stephanie who are here ran a conversation in Bowling Creed um which I could pull up and they uh it was just tell us the problems uh in our town and there have been a lot of contentions in the in the town around social issues but it was really nice to see uh for the town for they actually agreed on a lot and so it can be a good place to start saying hey we agree let's talk about these things and then move to the uh places where we have disagreements the there are a number of different calculations that you can you can look at just to majority the usual majority calculation over 60 percent felt x over 70 percent felt x um or y and uh but what is what is neat is you can also look at uh how opinions fall between different groups so once the clustering is done you could see the opinions that are representative of a particular group of people so for instance group A feels I'm just going to read out the statements from here the 50 commission model that galleries employ prevents artists from making a sustainable living uh dead artists don't need opportunities to be seen in cell work I do living artists do so these are these are sentiments that made this group um that were particularly important this group and separated them out uh from the other group there's no there's no fix in the number of clusters and the clusters uh are based on the data itself uh you can also put in what's called metadata statements or demographic statements so these are the statements here for instance I create artwork for sale or public exhibition I identify as a first nations or an in inward person I'm a trans person I'm an emerging artist you could get some idea of demographic information none of the demographic information is used in the clustering but it adds a nice layer to interpret data so for instance you we know that group A feels that dead artists don't need an opportunity to be seen they feel that 50 commission model of galleries employ prevents artists from uh making a sustainable living and and now we can go interpret that data by looking at the the metadata or the demographic information so if you look if we look here we see that most of group A are actually artists so they say they create art and say as you're like oh this is what artists feel um about things group C felt differently and group C didn't really they didn't seem to be involved in the art industry so it's a nice way to kind of interpret which groups of people feel uh feel similarly or feel particularly about certain things the other really neat thing uh you can look at whether a number of different views or snapshots of conversations where you could look at correlations of of data so for instance Bowling Green we saw that people who tended to believe that marijuana sentences need to be harsher and prison sentences for marijuana marijuana should be criminalized and prison sentences should be should be harsher also felt that refugees should learn english but felt differently about fairness ordinances so you could you could sort of see correlations between different different statements as well what's what's one of the really neat calculations here is what's called group informed consensus and group informed consensus is particularly useful when you have a minority group that feels strongly about something so usually when you look at majority opinion in these broad uh sort of swaths or 60 percent believed in in something if you have a minority population and even if you all very strongly believe um or feel very strongly about something you come out and force to vote you're sort of washed out by the majority opinion but the group informed consensus is calculated in such a way that across different groups you should see consensus across the the three different groups so even if a group is really small if that group strongly disagrees it's not a group informed consensus so these are just various ways in which uh to look at how how a population feels about a particular issue it can be a broader issue but it's it is actually really useful to have it on a narrow issue because you can further the conversation on that issue and then move that data into into in-person conversations this absolutely doesn't remove the need for having in-person conversations or or iterations things like that but it's a great discovery tool it's been used in various ways the civic assembly Bowling Green used it as a virtual town hall Taiwan uses the tool for crafting legislation in the discovery phase and so there are various various uses yep that's my rundown policy thank you um can i just ask why you call it a conversation as opposed to well i i feel like the word conversation has a has a particular meaning it's something that a few people do with each other it's it goes back and forth and this seems inadequate to describe what you're showing us and i'm also wondering how in the process of using policy maybe we'll get to this later the information that the you know this sort of snapshot of crowd opinion that it's creating how that gets communicated back to the whole group because that's where conversation you know it's sort of like then the group sees what it's currently thinking and then maybe people change how they think but to me the word conversation doesn't adequately map to this and i'm and if we're trying to tell other people use this it'll help you have a better conversation i don't know i mean maybe we just don't have the word in english for it so it's a little we've we've tried many different combinations of things and actually the way we've left it open to whoever whoever it is who's using it to use a term that they find most suitable so dynamic survey poll has been used a conversation because it is it is a back and forth in some way you could see what other people are are saying you can you can respond it's done in these bite-sized ways that people are agreeing, disagreeing and passing but it is the kind of information that you get out of some kind of focused group right as opposed to giving somebody a set of questions and then just having them respond to those set of questions and you're kind of blocked by what you what you don't know already so it's it's something of a hybrid i agree it's not this elaborate a conversation maybe brings to mind a much more essayist type or in the case of remesh the goal really is to have a conversation between a collective made up of many individuals and an individual or maybe another collective made up of many individuals yeah what i remember it's a it's like a ranking like part of their back and forth is that it like keeps comparing different things and ranking them in various ways but the information that goes in is not like the people are not writing an essay they're sort of doing something very similar so this it it is yeah i agree that is like you know it's not it's not what we intuitively think of as a conversation neither is it a survey neither is it a poll is like some kind of hybrid i guess don't dynamic survey would be would be some some part but there was a second question that then is that okay i guess that's all just just a further note on that i mean this is one of the questions we brought to polis and you know i think parol polis provides a partial answer but it's still something that could be developed further i'm there there's a since polis can extend across any amount of time you want someone doesn't have to be like getting a survey call at dinnertime and having to answer 20 questions some fixed period it has the potential to be a deliberative process in itself where people think about the quest the statements they responded to they come back to it after a few days they see new ones they've and there is that opportunity for deliberation within the context of the sort of duration of the process polis does feed back a certain amount of information about what's going on in the poll but it's it's quite complex and you know it's not it's not really oriented toward providing that kind of real-time understanding of how the deliberation is proceeding that was something where we you know we felt like it got a little bit of the way but could there's more opportunity there to develop yeah it definitely tends to be a bit a bit dense i guess let this comes back to some of the initial parts how do participants know what is being what is being said and interacting uh during the conversation that is that what you're is that what you're asking that the report is a bit dense and so how do people who are participating sort of get well certainly the report is is dense uh and people aren't you know people are not referring back to the report every time they engage with it in fact one of the one of our you know conclusions at in the course of our work was that probably people took a brief look at the report and didn't really invest the time in digesting it that it would have required to make effective use of it so one thing you did different was that you had a different visualization and this and this visualization actually gives people a sense of oh i'm the blue dot and i like as i state my opinion i move into this group with this group and there's a little bit more of a dynamic participatory feeling and so that that actually has this sort of oh i could see okay maybe i don't enter i can't see all of the the layers but i could see what the majority opinions are for the conversation i could see what the real group is feeling and this is as you're playing around with it so there's some amount of some amount of that yeah i think that that's right and we yeah we were we made choices that didn't really lead in that direction but that that's certainly an aid to understanding what the conversation dynamic looks like yeah i had a like i remember talking to shreyoung about it in in sort of discussing what the two is and one of the one is the sort of feedback that came out of this is seeing this seeing this view and having sort of your you know participation we represented this we made it feel less of a black box made it feel less of i put in my statement something turns out and tells me oh these are the top statements yes that's a little bit more dynamic yeah um related to this i'm wondering if anyone from the the taiwan team could write in about the experiment of additional people in the community helping participants in a poll write ever better consensus statements like that statements that gain so a role for quote unquote no buddies or people who are trying to help society have a difficult conversation what you may there's the opportunity for everybody you know a theoretical hundred percent to participate but there's also a way we're talking about adding facilitation into these difficult conversations and this if the poll were active we would actually see these opinion cluster shapes move and it's possible to see which statements are getting a very high rate of agreement across different groups and it's possible to read all the statements and i had heard of a previous experiment i don't know how long it was where actually people who were trying to help this conversation move along would closely read the statements and try to add in more finely tailored wording to get an ever ever higher percentage of consensus does that ring any bells yes it's very time intensive and and and it's so polis is introduced as a way to essentially save time so that the moderators can still do it in their part time right so that's the original value proposition is that in vtai when everybody participate in their spare time as volunteers and once it grows to more than a thousand people using normal forums become two-time intensive so that people won't actually put in so much time in doing this now polis came in and we said okay now we can scale to ubrex scale conversations so there was a far as i can recall two uh topics but it's not in the vtai one process where in the polis there is dedicated full-time staff looking at the comments try to moderate out duplicate ones try to put in more like a common value once uh and and try to get more consensus but it's extremely time intensive and the person doing it at least the one that i'm personally involved with uh billy lean who is not here but also a uh important be dismembering of zero member um personally took care of the polis on whether we use caning that is whipping uh to to punish drunken driving and and that is a petition uh actually the the the petition case was the most number of petitioners uh in in taiwan's history uh and and we use polis just because there's tens of thousands of people was very wildly different highly antagonistic um divisive toxic statements and and then uh billy personally went into every statement and tried to get some common value out of it so that it's become more highly functional but i think he suffered a lot of mental damage because of this process and took more than one week to recover so this is not suggested to any like new people do this process it is extremely um exhaustive even uh to to people who have to process essentially to all the different toxic and um radical and other factions and comma was think that reflects comma value the other thing uh it's cl did the same thing around the capital punishment right after a uh random killing case and again i think he suffered a lot of mental pressure um and trying to moderate a uh case where again um thousands or tens of thousands people joined and that's the two cases i remember but both are outside of the veto and process we try to not inflict that on our facilitators but it is it is really tricky in that sometimes you can have like the cases where like seven thousand people come on in like a couple of hours and then and then would you do right it really so well we we've built the algorithm is built to turn it out but then everything else slows down so you're basically also on the it becomes it becomes cumbersome cumbersome on like on the practical end also to move things around but i like one thing that that would address your question though is if after this like the data is taken and then somebody with either export knowledge or somebody with an understanding of the issue can pass through that data and then sort of like divided into myth fact you know infrastructure question things like that and then and then from there take that information and then it passes in various ways and then that can be moved through different trajectories and i think that could be like uh in between way and well um in the context of some of the toxic conversations that happen in the u.s um how without the moderator you know sort of that kind of tight touching how polis deals just innately with that level of toxicity trolling type response which you're likely to get in a in a u.s forum around like i'm talking the nra kind of conversations so they have there is the option to you could either do a very heavy moderation and only left statements in like once once you've looked at it or you could do the opposite where you can keep a very close eye on it and then remove statements as soon as you fly as soon as you flag it as as problematic and this uh joe and stephanie had to do the group had to do a bull and green not so much because it is incredibly toxic but personal information was mentioned in some cases and so as long as the moderators i think are on on point and watching it regularly it's possible to to do that the problem with with doing the opt-in part is that you end up like the heavy hand of moderation can end up being a censoring uh tool where you're like oh this is not this is not an important statement of this is it just can it can become a slippery slope and so it is nice to sort of let it be you know organically what what participants want to say and have to say about something hasn't been that said like people don't seem that nasty in general on this at least not so far um so for the data and the clustering how do you figure out uh the clusters if people agree with similar statements if so uh are you is it an automated decision-making process to uh determine the similarities of the statements yeah so it what what comes out of the other end i'm just going to pull up this river what comes out of the other end is this this giant kind of matrix of you have a participant and then all of the statements and whether they agreed or disagreed on the statements and then it's it's like a dimensionality reduction and the clustering happens and the clustering is really so this this might give like a good good imprint impression of the clustering so each of these numbers here are a statement and then statements that are people treated similarly are closer together in this space which is why you see a lot more kind of like a mess in the middle and then as you move outwards like away from this the center you see more and more polarizing statements so it's it is a it is a distance metric in some sense so it's like is my opinion closer to to your group or yours or is it like different enough that i form my own separate separate group now and it's always this this this sort of like a comparison and the clustering happens uh happens so it does it you can have different clusters and then it does like a confidence interval measure to just keep checking what is this statistically most valid thanks i just thought it was really cool the um idea of being able to think about the moderation in a sort of like range of scales from extreme to sort of loose touch and i'm wondering the if you're able to detect any patterns or extract any metadata about patterns of moderation and sort of start to help moderators understand if they're expressing bias or if there's any sort of heuristic that they're consciously or unconsciously using and how that influences the sort of trajectory of consensus this is a whole other platform but uh it is so the we we do it in a at least i try to we do it in a more qualitative way right like this is not so some part of like setting up this experiment a process is really working with someone or with with a group who wants to use it to set up their process but that one like because that one group is way ahead in that like you've you've tested out for a very long time but most most of the time when people like are testing that this out in the beginning one of us will totally step into their process and be like oh this you know like from past experience this may not be the best way to do it on or that there's no there's no actual this is your measure of bias kind of thing but there is a we strongly suggest like we can't force anyone to do anything but there has been many like we strongly strongly suggest you do blah so i just want to know that in for the civic assembly in bowling green we really let our local partners lead in the moderation process and there's no way to for us to really understand the nuance between two statements about you know the traffic circle or something and they're like these are actually there's so many about the traffic circle so it's like there's actually two distinct ideas here there's no way we would have known that so our partners were the local newspaper and and a consultant with the with the local public university so they were really leading the process and and darshan out in you know consulted and and so did we yeah that's how we that's how we let it takes a task force it really does like instead of the people who know the technology that the experts and then the sort of like the person who can step back or the group who can step back and and be the the unbiased presence maybe so this isn't so much of a question but more perhaps like a statement or an opening for a conversation since i think what's really interesting is that we have a lot of people who have used it at different scales here and also maybe touching a little bit back to my cause point or mika's point excuse me about how to participants engage but and having worked with polis at least personally for myself what i find useful is this ability to sort of start to identify where the opinions are who those stakeholders might be what what information might not be clear enough what information like we might need to provide more resources around but it's definitely also very clear that there has to be some layer of digestion around this um even like the viz to some participants might be too confusing and so how do you how do you message that to people or how do you help them understand different parts of of the report so yeah i'm i guess i i open this up too for folks that are using it like what in what areas was it helpful and like where did it get you and then what did you need to do after that because i think that it's it's not like you're not going to use polis and expect to be like great i have a complete solution it's it's a tool also to that point it has polis hasn't sort of maximized its like sort of capability as a as an open source to read the more people who look at the data the more people who generate uh or who can contribute the the title of community you have the more you can dig into those things um but yeah definitely um but sorry the question was to you though we definitely could have used more time to translate and you know hopefully visualize the the results for the bowling ring community because after we ran the on the polis it was 10 days we had um uh a public forum a town hall uh where we had some local officials local and regional uh people come in and speak to the issues that were surfaced by the by the process um and because it's such a i think there were 2000 participants and 600 ish statements so it's just a lot of information to get through and it's it's nearly impossible to digest i think all at once so i think us having some more time to draw useful comparisons between uh statements that are that were you know surprising you know across the more conservative and uh progressive groups would have been helpful and i think it would have um they were just kind of subtle subtle and surprising things i came out of it that we could have drawn out that would have been useful to the conversation hello i i had a question with polis as an engagement vehicle what does did they help define what success should be when you run an engagement on the platform and the reason why i ask that is because if we look at these different ways of participating in government or society at large you know we always talk about voter turnout or even when you're putting out a survey you have response rate so how do you know i mean i think there are different types of numbers that have been brought out in terms of number of statements or number of people who participate in the conversation like does polis recommend a statistically significant part of the population do they help are you supposed to put an input in of the target market you're hoping to reach and then they help you calculate how much of that you accessed i'm just curious to know because sometimes i think something happening is seen as success but i'm curious to see whether that has been pushed further with this platform yeah so this is this is really important in that it is it is like sort of brings back to the layers around around the platform right i'm not so like sort of personally not so worried about numbers per se but diversity right so if you want to get all of the voices in a conversation you want to make sure that you are not just advertising or tapping into one one group right all of the people who need to be in the conversation of you who you want information from should be should be in the conversation and i think that is like almost before because it takes 10 minutes to set this up and you know release it but then there's all of these layers before okay what do you want to run the conversation on what is your objective is it to to solicit feedback on a particular issue is it a broader is it a broader tell me what your problems are in the town and then based on that you say i need to reach these groups how do i reach them the bowling green example was really good in that the partner organization there was a newspaper that's very widely read in the in the town and so they're you know they sent it out through their mailing list but they also solicited they went to church groups if i'm not mistaken they went they went to like places where like opium addiction like they went they went to like a number of different they made sure that they like they sent it through that international mailing lists and so those kind of making sure that you're you're going through the channels that are trying to reach people through all of the channels you have available i think is is really important because otherwise it's the same voice of the number the the methodology behind the outreach was really to target a representative swath of civic groups and and reach out to the leaders of those groups and and get them to basically you know unless they're in their communities but so outcome sometimes is much clearer if it leads to legislation than their boom but also otherwise it can sometimes get stuck in in a little bit of a loop because this is like like mentioned before there's there's a lot of information here how you turning through it how you're moving through the next step what's your what's your objective it's it's really important to stop it just to continue on that question and thread how has the v taiwan community done direct marketing or outreach to solicit a broader a base of engagement very case by case like in the uber x case what we care about again is diversity right so we settled on the url of the police poll but we did not publish it until all the stakeholder groups including the ministries the taxi companies the major few ones the unions of taxi drivers and uber we agree on a time of when do all the stakeholders receive the url and and then we send out the url early like me knew the url at the time and then at that's a great time everybody gets that url and they spread it through their line channels through their whatsapp channels or whatever channels and the importance here is that when you get in you both find that there are people who think just like you and there are like three quarters of people who don't think like you and both are important if you find a conversation that it's a echo chamber then your your participation or contribution is naturally lower because there is less chance to go across the aisle and find common values but if you start with a very diverse picture and people gradually converge to the middle there is a satisfying you know theme to it but that's because it's already a large controversy in the society in the main journalism there's you know media dedicated to to advocate for one particular position but in other cases where police is used like the ncii case it was seen as a very niche topic that maybe only three NGOs care about and so the main point here is amplifying to find people who have their own media to have their own brand on social media or whatever and try to use those channels to amplify this and diversity is again important but it's not so much about balance but about how can we go across different NGO and self-media civil media groups to make this into a more commonly aware topic so again it differs case by case but the most important thing is that every week the diversity need to increase rather than decrease I guess and just wondering about specific examples like did anyone you do print media did like postcards or flyers or billboards like how at what scale has the polis platform been advertised to the general audiences across the island in taiwan right so there's several things in v taiwan the main venues or vectors of spreading this is up to the initial stakeholder groups right so the stakeholder groups who participate earlier in the mini hackathon sorry in the agenda setting meetings they understand that they're responsible to find whatever appropriate venues they could be you know billboards or there could be QR codes there could be whatever that is most effective in their own mobilization camp but we don't interfere with that but we just become aware of that that that's the most important part there are other places where we use polis that's outside the v taiwan process like if it's in the e petition process there's some other way to reach to the petitioners we'll cover more on that tomorrow but in the v taiwan process it's mostly on the stakeholder groups to do their own mobilization and we don't actually we can't interfere with that as long as they they're fair like they get a url at some time the seed questions the initial statements into the polis system is co-determined by the stakeholders we we care about that balance but coming to the mobilization we don't instruct or prescribe specific mobilization methodologies just related to that actually when when tied to a physical event sometimes it can be really really effective like when the women's march for instance was tied to the giant women's march that happened and then you know you already have people coming up the march it's spread through those channels and then like if you're already mobilizing people for something and you're that you can piggyback on things and that can be useful so just a note on this since it was part of our lengthy discussions about what kind of community we were trying to reach you know I mean the thing we excluded at the outset because it would have been much more demanding than we had the time or resources to to commit to was constructing a representative panel that would have some statistical validity at the end of the day in terms of its representivity of the community a lot of a lot of polling methodologies rely on that for their for their authority it it's certainly something you could do in the context of a polis process but it's very demanding especially in a community that doesn't have kind of pre-built panels for for this sort of polling we opted not to do that and in fact sort of tore down all the barriers to participation that might have allowed for different kinds of demographic data we'd so ultimately we don't know exactly who responded to our poll we did do quite a bit of outreach but and could have done more but then doing more is never necessarily enough because you don't know what even then what your what groups are in or outside of your outreach strategy what we can say at the end of the day is that you know for some issues that were extremely divisive that appear to line up with fairly commonly understood political distinctions for example immigration turned out to be one of the most divisive issues in the survey groups that disagreed dramatically about immigration agreed nonetheless about a whole range of other issues so that's that's the kind of you know internal insight that isn't reliant on the construction of a representative sample and that was the kind of thing that we felt we could do given the time and resources we had available to us everyone hates traffic traffic paid of traffic ties us together as a as a species yes but traffic traffic is the most you know yes traffic and but you know there were some non-trivial commonalities too right but there was almost complete consensus on issues of government transparency and accountability on local development and zoning issues I mean things that you know form the basis of a very substantial consensus around you know issues over which the local government has some control people then disagree about immigration and you know drug sentencing and a variety of other things but it's that it's that kind of non-intuitive internal split that for me was the most interesting set of results in the poll quickly finish with our other alternative tools we use on this this very beginning stage when we create the the issue on VITAM's website so apart from using police we also have other tools such as I think the beginning we show that discourse and slido and type form and the way how we chose between how we choose between these four different tools it actually depends on the community of VITAM on itself and also depends on how much material we have already in our hands so for conversations or issues like in a very beginning stage and very controversial or just very unsure and unclear what they have we would choose police for sure and but it's also a decision by the community but for issues which is which has more materials already or more rough direction from the beginning we tend to choose discourse this kind of it's a online forum you can put different directions into different categories and you can have a description on top so the interface of the discourse look like this you can have oh you can draw you can have this part as a description of your issue you want to talk about and you can create a few different issues or different categories inside on this on this interface okay yeah and here you can also add your users so for example when we launch a case we oftentimes have a contact person who is in charge of answering all these questions or point people to answer all these questions on this forum already so be handy if you can just add this contact person and then they'll be like timely come to the forum and answer or reply so this is one yeah let's talk about that yeah and another tool we use is Slido well now quickly run Slido in like five minutes we're gonna like decide what topic we'll use for the rest of the workshop so Slido is an online survey tool I think but it has two main features one is for a question making questions another one is for polling so we'll be trying this one very very soon and another one is Typhoon I think Typhoon it's maybe many people heard of Typhoon already it's a unlike questionnaire system that is has very has very beautiful design so I think that the good thing about using Typhoon is to kind of just attract people to enjoy the process of answering all these questions so if we have an issue that has so many questions for people to answer from the beginning I would definitely choose Typhoon okay okay so we are going to move on to exercise one and we will take a poll and we will pick one question one topic one topic so please go to the hack folder there is a Slido link it's called Slido link here just right below opening slides there's a Slido link and then after you are in the Slido page then you get to choose get to choose which which topic you would like to discuss about but Abraith yeah is there a room or a code for the Slido not everyone has the hack folder right now is there like a code that we can get to oh I don't have a short URL yeah okay yeah just go to Slido.com and then enter the event code 0611 yes just to date 06611 okay what do you mean just go to the Slido.com and here you can enter your code event event code and enter 0611 and then you get to the page and there are two topics one is cyberbullying and the other one is data integration so and here I need you to pick one topic that interests you the most and here are sound inspiration questions for you like for cyberbullying you can think about what's the definition of cyberbullying and who can be counted as victims or the actors or potential victims and you can think we can or discussing about how we can prevent cyberbullying or should we regulate cyberbullying and also our social networks responsible for the rise of cyberbullying and how should teachers address cyberbullying these are just some questions for you to think about if we if cyberbullying is the topic that we're going to discuss about and also data integration we can think about what's the definition of open data or my data so what's the difference between open data and my data or should we combine data from disparate sources how do we protect privacy should we regulate data integration between public sectors and private sectors should the government offer data integration service for free or we should charge or should we who should be responsible for data breach these are also some questions for you to think about so so we have because we have limited time for preparation so we only have two questions for you please choose what interests you and we can okay so does that have anyone yet voted yet and also because when when you went to the registration table we have the color dots for you so you were secretly appointed as different stakeholders so these are the colors standing for and I don't have the orange orange color so I use white orange represents stands for citizen so if you are if you have the red color red color dots on your nam tag you will be the stakeholders of from the academics like scholar professor or sometimes lawyers or some experts expertise with yeah and the yellow dots means the business cooperates or private sectors and the green dots means the government and the orange dots means the citizen or civil society or the community contributors and the blue dot would mean the online participants so it would be the remote and later on in the in consultation meeting we will have you be seated according to the dots the color of the dots which means the different stakeholders so for now after we take the topic poll you might need to be someone else's shoes to think and to post comments according to the position you take all right okay and if you don't have a color dot we'll get you one oh okay looks like see us see us how many participants do we have for now how many participants do we have for now or we have uh 19 has everyone voted I don't know if you haven't voted you might be able to tip the vote yeah so I guess the topic will be data integration and this topic will be used for both day one and day two so tomorrow we will also use data on group integration as the topic to host the whole process to to lead the whole process for day two the landslide for data integration exit will suggest okay am I charging my phone so after the topic poll then we will go on to yeah well the exercise one will be like like um pretending that we are pretending that we have a like a mini hackathon so we decide which topic we'll like to discuss about for the later for on be Taiwan so yeah so we just finish our stage one proposal stage so we have selected the topic based on our consensus and then we move on to stage two yeah but I do I have did the work for you like the name the title of the topic or you also like to think about it better like data integration do you think it's a good title or title or you have different ideas um maybe like open data integration or some other issues some other name or topic that would you think suits the topic better than data integration yeah okay uh so just pretend a play like a play doing the play like that we just done we're done with stage one and then we move on to stage two which is the opinion opinion and we will do the online opinion collection and also we will use the polly as the default tool and for to do to do to initiate the online opinion collection uh step one is to set up a good guidance so if it's polis then uh we need to have a guidance on the welcome page you need to tell the participants or uh uh any online users to let them know that how to use this tool and we have an in Chinese in on be Taiwan we use the uh I translated into the English and we also we so the step the first one will be we always starting with I think like I think that our integration is bad idea like give me a symbol in short so like second uh every comment is independent so there's no need to reply others comments is so it's different from uh like the posts on Facebook that you get to reply others comments but in here you get just you can't just agree disagree or pass yeah some some of you might have already quite familiar with polis but there are some not I haven't used polis before so I'll just I'll still need to explain it and the third uh you need to tell the online user that post opinion separately so do not have too many points at one comments or in one statement so you can say that I think data integration should be a free service because I think that integration is really good but this is probably uh can be contained uh containing two different statements so you bet child um once the cost statements are too long then the other other online users cannot simply just agree or disagree so you have to give a sample you need to tell the online user it can be simple and one sentence should be planned and clear and also do not use question mark because it should be should not be a question it should be a positive or a negative statement instead of a question yeah so this is the guidance that we have on retail one to tell people how to use how to use the polis and also for other tools like polis discourse to slide on type form we need to have a introduction on welcome page so the introduction can be edited or written by the facilitator or by the editor or by the proposer those who are familiar with the topic yourself so we need to use the plan language so you have to be short so you can grab people's attention quickly so because in I mean in digital era people's attention are not really hard to keep and the content itself has to be friendly for all and like you can start it with a big news or controversial news like big events so people get to know the topic quickly and you have to explain the purpose and the use of the results so they can feel safe or somehow feel comfortable with what they are going to post or what they are going to maybe have some data recorded by the tool you use and also describe or what's the next process so they know that okay after this police survey then maybe there will be a mini hackathon or there will be a consultation meeting and they will know that their result will be just in van so they can know they can they understand that this is valuable and their opinions are valuable and for polis because it's open and we don't just I mean you have to have some seek comments to strike up the conversation otherwise people will just jump in the page and then they see nothing and maybe they probably don't know how to use it so they just exit and quit so you have some seek comments there and let them to press agree or disagree and they will somehow know how to play the game yeah so and here are the seek comments for NCII we are we were generally focusing on the definition of the what the definition of intimate images so you can see that we try to have many types of or possible potential scene or types of what bodily intimate images should be like and you can also say some plan just like like I said I think the government should be responsible for for the data breach okay that's it just plan it simple and step forth and this is just for discourse but well because we have we have that the authority in Taiwan they let them be obliged to reply within seven days on discourse so whenever there is a topic and people have posted comments and the editors think that okay this topic belongs to which component authority and then we tag and then we tag the component authority and they are they need to reply within seven days yeah so and each component authority has their own account on discourse so they they can't say that they didn't know they can say that they don't know and they will get notification so this is for discourse on vTaiwan and then step step five asks for contributions in the survey this is very important for all the tools because well you have already collected so many comments and you imagine that you put lots of efforts and you attract so many participants and you don't want them to just go away and you don't know who they are so in the survey you need to have some questions for them and ask whether the participants willing to publish their own opinions anonymously or not or and ask whether the participants willing to become an interviewee or they would like to participate in the consultation meeting or not and ask for some suggestions or some candidates or interview is that they may be familiar with like if and then you can expand your connections and also an opt-in or subscribing to the email from vTaiwan or other platform so that you can send more invitation of other issues through this list of emails but i know that gdpr because of gdpr so we might to be careful with the european citizen if you have collected their emails or names or other personal information and uh we believe that at least we need a month to run the online opinion collection so for all the tools we run for at least a month so we try to draw more attention during the month like you put like you have new release or you have some messages on some line on the networks or social medias or updates on social network and just tries to spread the link as much as possible and keep records by using screenshots for polis because well this is for the older version of polis that i kind of lost track of the origin and the specific dates so this is a reminder for me to have the screenshots for a specific date so that i understand the the formation or uh the changes between uh throughout different stages of polis and then after the online opinion collection ends you need to publish the reports and include in the raw report and the second hand report or the secondary reports because the report will be the material for upcoming mini hackathons and also consultation meeting so after you have the raw material you need to we upload it on vtawan immediately and the material will should be thoroughly discussed at mini hackathon and let the participants to figure out what's the next step should be either initiating one more round if the results is not clear enough or or we can't find a rough consensus by analyzing the report then we might need to initiate another round of online opinion collection or if we have reached a at a at to a certain level of rough consensus then we can move on to consultation meeting based on the report we had and remember to because we have already uh have a competent authority one or two or more plural then we need to submit the report including the raw or the secondary reports to the competent authority to let then also keep track of what's happening yeah so that's the stage two the the opinion stage and we can try just use the data integration and gashina yes she will initiate from scratch to open up a new poll this polly's page and then we will just use data and integration as the experiments to have to run through the stage two online opinion collection okay okay so uh the title should be data integration or anyone have any kinds of ideas or comments that we should have other kinds of title to instead of data integration or do you all agree that data integration is a great title well if the title is open data then stakeholders will not understand what what what this you are going to discuss like open data then but yeah sure it could be a title that kind of summarizes the problem with that stake a little bit better so maybe you want to narrow down the scope or you want just like the the legitimacy of data integration or something else oh yeah sure so if it helps think about it as this is the first thing that someone would see when they would come to your poll so ask yourself do you understand what data integration means if you were to get to a page and someone's like I want your opinions about data integration does that if folks want to yeah we can also yeah narrow down the scope like maybe have a more focused well I'm happy to also if people want to like popcorn out ideas I'm happy to write them so we can just like get some brainstorming about what folks thought when they were voting on data integration and what actually interested you about this so I'll I'll just write notes over here and folks can shout out I would love to see discussion around a municipal version of the general data protection regulation that was passed in the EU so hyper localized GDPR for those who don't know what that means it's more or less making sure that data protection is by done by design and by default so strategies for getting government to use open data standards just data portability for you know your consumer data what would it look like to have an api to move your data from one electronic medical record system to another or a different set of applications your credit data or something even just switching banks like the switching costs are almost impossibly high just because we can't port all of our payments over you know like so that kind of portability how to make data from different agencies or different governments more interoperable or useful by applying data standards or other techniques you sure I think from the perspective of a broader topic maybe the phrasing is data protection or data privacy rather than data integration data ethics I like that too okay I would also recommend using the term city data or public city data rather than open data because not everyone's familiar with that term all right so what the topics we have strategy for so almost in the bottom we've had a suggestion to rethink this as data protection or data ethics instead of data integration how to make data more interoperable within government data portability like apis or consumer data or should also tie into that that other question strategy for city data or public city data standards and municipal version of gdpr maybe there's like a seems like a great opportunity to whittle this thing down right so yeah it seems like we have two uh subsets one of them is about data protection privacy and ethics seems to be about how institutions government institutions can manage data in a way that is more useful for the public is that a uh a fair assessment yeah this privacy and then making it accessible I would do two big buckets I think so yeah should we should we go with one versus the other this seems like an opportunity for a poll this show of hands would be fine um quick question of have we agreed on the focus being at the municipal and city level because it seems like there's consensus there but if there is just stated overtly at the show of hands of folks are okay keeping this at the the city level answering to cordelia's question anyone not okay with it the data privacy though bears away from the data integration question that you first posed right does it does it subhuman data privacy is good data privacy is kind of broad so I mean in one context um data privacy could be I mean the general protection of privacy I mean like um I mean even including the physical space or virtual space or privacy I mean in American law it could be very broad I mean against the government or against the private sectors so in here I think we need to be careful of what data privacy is I mean it's a really broad term so yeah so uh our on vtai one data integration is uh to to facilitate the integration or sharing of data among public sectors departments or between the private sector in the public sectors so it's I mean the the public sector would be the necessary uh stakeholder of data exchange but whether it's with the stake private sector or it's with a amount different apartments or like federal and local so it's it's all already very broad but if it's data privacy then it would be even broader would you say that's two and four on the list here strategies for city data data standards for government and and or how to make data more interoperable yeah then we need to have a good name like a short name Mika you've got your hand up well just as one of the people who voted for cyberbullying I am more interested in spending the next 36 hours thinking about privacy than I am about data interoperability which is very dry and if you're not very technical may also be a little bit difficult to engage with whereas privacy feels like something we all can have an opinion on and the point is to see how you work through different opinions so I agree seems like a lot of motions for that do you have a question or a way that you I wonder if no one could say a little bit more about how it's a very specific policy question you're making it can you flesh that out a tiny bit more I think it helps to have a specific you know the way as I understand the v-taiwan process has worked there was very specific question and how should we regulate uber x so is there a how should new york city better protect the privacy of its residents is that a maybe how should cities protect citizen data or like the private like citizen data the privacy of their residents how can it help regulate or control an individual's digital privacy I like more along what mika was phrasing it you know how can new york city help ensure digital privacy protections of its you know residents I'm curious how the v-taiwan process works when after a poll and there's a clear majority for one thing and then people with microphones start saying something else and then people start deciding to do something that I'm gonna pass on that question for now because I think that we've seen in the v-taiwan process that the questions are brought to these meetings and they're discussed so we're kind of like trying to consensus around a question here in a different way than their process and I think Devin just to be honest I think we're just rephrase trying to rephrase a counter statement to see if we can have some clarity how about policing just treated as a survey we could I mean if you're looking for a narrower version of this question that might be kind of fun to engage with how about uh looking looking for that sweet spot between like what is this not what is the solution to this narrow problem but but a broader conversation on I think data privacy government use of data all all of these all of these things work at the city level yeah how does government better protect citizen data from you want to chime in since this is related to you too how to best use government so how can government best use data with that cover privacy and sharing of data is that why I think so I think it's a broad one broader one would that would that be a catch all bucket that that works for everyone how does government better utilize and protect data at the city level or citizen data yeah okay yes protection but both not just protect but also like utilize data and effectively okay we have how does the government ensure how can the government use and protect use use and protect yeah citizen data with that is that phrasing okay no you want want the word sorry our local resident data yes yes see yeah so maybe city government protect resident data um using protect resident data city government resident so not local so how how can city government use and protect resident residents data the data of residents residence data yeah so how can maybe how can how can yeah yeah yeah this is this is yeah this is exactly what we generally have a mini hackathon people just throwing their own opinions and yeah exactly what we have a mini hackathon and sometimes it takes around like more than two or three hours just all this confirmation again and again with also with some public officials yeah do people fight no no no no that's a yes are you happy Devin we review it's all blurry what does it say uh how can the city government use and protect resident data residents data yes all right thanks for making me the arbiter here um okay that we have that we can maybe just i mean if we could specify if we just want to pick a specific type of data that the city has like how can how can the city protect the data that people give it for the id and yc or how can the city it's very narrow i think something narrow maybe maybe that's too narrow maybe it's too narrow but i mean i think that if we don't do something where we specify then it's like what data does the city even have you know is a question that i think most people here i mean i i spend time thinking about this stuff i don't really know i agree the problem there though if you make it really narrow as you lose people who are not interested in that like specific okay you open the rabbit hole and all um yeah let's let's go with this we we can narrow it as as we go we go down um okay so this is this is what what a moderator view of of polis um looks like and um here's the topic you have you have a space for description you can either put it in here or embedded on on a larger website uh i think that might have an example somewhere um something like that i do on text and and then let's let's go through the settings first so visualization and then there are various ways in which you can you can get participants to engage so you can either require thank you you can you can require that that all participants sign in with facebook or or twitter so you get some kind of authentication information it's not usually advisable because that might sort of restrict uh some people so you can you can kind of leave it open um more there are a number of different moderation moderation settings okay do you want me to expand this here there um strict moderation means that it's an it's an opt-in version so all of the statements are not approved until so every single statement needs moderator approval or you can do the other way have it automatically go in and then have moderators remove it remove it out the authentication is well what i mentioned which do you do require them to have facebook or twitter uh log into facebook or twitter first before uh before participating so visualization is on prompts uh this one oh yeah i require out there thank you um okay and then there's a space to submit submit a comment um so does anyone want to come up with uh with a comment if we are seeding it with a couple of statements about how can the city government use and protect residents data yeah but we should so one i feel that yeah that's one way to put it so that they know it's important for people to know that it's your sentiment um that new york city it's also um NYC let's just it's also useful to to have to have the whole thing spelled out not the acronym so that anybody who wants to look it up or has that so yeah what do you want to say yeah actually that would be that would be a better um there this is also where devon this is where the what was devon's statement about about link was it link nyc was it something else sorry id ah id nyc okay so what was his i feel i forget well we'll wait for him to come back any other statements anyone wants to add this is just to see yeah he's here so yeah yeah oh i uh data is being collected on me by the city i know i know but i know would be so i know okay what people are saying all right i see so this is not so much an opinion it's almost it's like some kind of a metadata statement that people who know or don't know know so that we can mark it as marked as that um yeah all right i think the city cannot be trusted to protect my personal data yes okay um i'm just going to put these in for now and then you can uh what you can do is access this through the bitly so there's a and then submit vote and submit your own so it is it's it's up there it's bit dot ly slash v taiwan dash workshop so you go there you should see this essentially and so now what you can do if you have a statement and we didn't get to you if you scroll down a bit you can add your statement here and it'll yep i had a question about phrasing like what what's the implication of phrasing a statement like uh the city cannot be trusted versus the city can be trusted like i feel like that might have ramifications for um like people can be primed to respond in a certain way through the language yeah um so what kind of like considerations or is there any i mean it looks like the interface doesn't really offer much in terms of guidance on framing yeah it so it it doesn't it doesn't really matter but maybe positive is it sets a better tone do you have a positive is better right yeah so in in general it seems like positive is is the positive tone would be better which is why your seeded statements are are important because it kind of serves as a as the norms of the the the tone you set so when people come on and they see these statements are in a in a positive tone then they tend to also put in statements the positive tone the bitly is the taiwan dash workshop back there i have a quick question um is is the question or statement submission going to be open to the public without any sort of um like process for applying to be able to make a statement and if if not then how do you prevent people who are on there to like incite um discontent from going and doing that yeah so the the core setup is just you know open but you could put it behind a login screen or an authentication wall or something like that uh or you can whitelist people and then and then allow only those people to to access it but a lot of the times it is it is open and if somebody you know have you had problems where where particular participants have have caused trouble uh we only have collected some comments that have been uh that were too long so it's hard for the participant to to uh to press agree or disagree but we don't have i haven't had seen like discontent or other bad or yeah other bad words yeah and you can moderate statement by statement you don't need to to moderate up yeah yeah yeah here right good point okay where is the backstage here um so you can go in i'm showing you the ins insides of it this is the topic and then you can go into moderate and here are all of the statements um and you can you can moderate in or out these statements so i say except i think date on yankee span should be sold to private companies but not date on that or why not um so you can you can do it like i'm doing it is like just people say yeah um so if someone wrote something that you're kind of unsure about uh is there any way to ask them uh i know that there are people who are anonymous and some people are not uh is there another way of engaging in that conversation of saying hey what do you mean if people sign in using facebook or twitter then then there's a way to identify them and maybe further those conversations but if it's an on it's an anonymous user that's that's hard also it's not it's not credit so if somebody has a statement you can't you can't respond to that particular statement and say oh tell me more on this and then start a separate thing that's kind of where the past unsure comes into play like if you're just not sure what they meant in that moment then that becomes the past unsure and as darshan explained like if too many of those questions like if that question is uncertain for so many people then it's gonna fall below um the other way that the other thing that you can do is if you think there's a way to rephrase that statement into a way that's clearer where you're like oh if i was phrased like this it would be more understandable potentially then you could enter another statement if i'm unsure or past does that never come back to me again like will i lose that so once yeah once you you answer a statement it won't come back to you again yeah even when you come back to the platform unless you delete all of your cookies and all of that you'll have it i guess i'm not the only one who has a question on trolling um i'm wondering if that's a natural way to deal with if say like one percent of the people are in there to to cause this trouble with the polling if people are just passing on what they say if it just drops the bottom yeah if yeah exactly it drops to the very bottom if people are saying you can also collect location information and things like that sometimes do to filter depending on your need you can do a bunch of monitoring to see how you know maybe advertising is changing things stuff like that but yeah it's it's it's very straightforward to to set up and and participate in and as soon as seven people vote the report goes goes online and then how you choose to distribute the report is is up to the moderators so you can either make the link to the report just public that anyone can access it or look at it or you can you can keep that data internally and then digest and send out that information later can participants add demographic questions or or how does that get determined the participants can add add any questions you can on your end decide that you don't want them to to include any demographic statements and then just you know remove it out of the conversation but yours was a demographic statement and I could show you how to treat it you just click it as metadata and then they don't go into they don't go into the clustering so I walk for government would be and then it doesn't go into the clustering and it's treated as the separate metadata the moderators have a lot of flexibility in how they set things up I just noticed this comments oh I work for the government it's really good because we can see some comments that can ask it's sort of like a question asking for what what the online users are like which which kind of state force that they belong to so you can have some see comments like I I am a non-profit or I'm from non-profit and some sort of that kind of stakeholders then we can collect the data that then we can realize what the online user or those who have participate in the survey could be yeah yeah and if you have a broader question then you can ask the most specific questions within this and then later break it out for instance the questions that Devon brought up like you can you can add that in and then you know like follow a second line of conversation down that channel in a night question yeah any so we're going to leave you with some time to vote to add statements we'll keep track of things and make sure that we mark what's metadata as metadata and in a few minutes we'll have some snacks and then we'll get ready for the stakeholder consultation portion of this so it's a bit of a break but uh yeah engage with polis also just to make sure is the concept of the metadata question clear to people here yeah yeah so this is a metadata statement I'm an academic or a join so this this is an interesting question you want you want this is this is an example of a statement that you need to split into two you ask only one question at a time so when people say agree or disagree you know which part they're talking about so it would be I am an academic would be one statement I am a journalist would be a separate statement and so this also really illustrates like the importance of having a moderator for your polis poll conversation because you might want to like in time you're going to have to adjust these or fix them so it's not something that you just want to like leave out there and then not not watch yeah you want to watch continuously and you can split that yourself sorry can you split that statement yeah so you can remove a statement and then you can put in as two statements and the nice thing is the moderators and participant statements are treated equally it's not that you get special statements because you're a moderator so then it just all goes into the conversation and then it depends on how people are reacting to the statement how it gets toggled up or down so what you would do if you rejected it as the moderator you just go back and re add them as seed statements and they'll go back in so as a moderator like seed statements basically are the ones that will start it but also any questions that you want to add yeah yes you had did you end up rejecting it did I end up did you end up rejecting that question yes oh yeah add them back and then you could just go in here and what's the best protocol to deal with sentiments that are not framed in a way that like sentiments that are not for start but don't start with I think or whatever like is that the role of the moderators to weed out or is that something participants can sort of vote against by saying pass unsure so a combination it doesn't it doesn't have to be I feel you know something doesn't have to be phrased like that as long as it's clear that it's a single sentiment and it's it's your sentiment and that it's that's up to the moderator people have different codes of contacts people have different ways of dealing with things and so there's a fair bit of leeway there so you could take it out clean it out put it back one really good practice and is that if you moderate out something then you let people know you say these many statements were removed because of this this and this reason and so that actually increases the transparency and and the trust that that participants have in the moderators I was wondering if the the Taiwan team could let us know if the stakeholder groups agree on the seed statements before the poll is publicly released yeah and just relatedly can you elaborate on whether people share the same definitions for terms to or to what degree you know that to be the case because I think that's that's a piece of this question as well I mean it turns but yeah I don't know if we've gone over the I don't know if we define the O R I D in terms of the the O FATE objective and fact findings phase which would address David's question maybe you could say a few words about that which would have happened before what we're doing right now and for the online opinion collection we for the C comments we have to have those discussion about C comments at mini hackathon just right after the poll so so all the C comments are agreed upon by all the participants at this mini hackathon and as for O R I D method we encourage the people and the participants to have the C comments by by the definition of or in R which is the objective and reflective excuse um yeah because as we can see that these or in R are um more about the facts and also the filling and these are the like the source and the materials substantial materials for for our next stage like consultation meeting yeah so or in R we are what we focus on on at the at this opinion stage and all the C comments are agreed by the participants yeah but we yes uh we we uh encourage I mean we have the C comments for around like five at least five C comments will be great because one or two might be two that to feel so yeah five yes when you say people who come for the hackathon is it mostly citizens then or government to do all of the stakeholders to list point list this point come to the hackathon in a CI case we have mystery of justice at the mini hackathon for the C comments discussion so they do they did participate in this part and and for other cases I'm not I'm not that sure but it depends on the the competent authority whether they are able to join because uh when we have a like agenda for a specific on a specific day for mini hackathon we will post it on Slack and let the participants or the online user know that they can join which which many kind of hackathon they are interested in so for a specific issue they can get to know the date earlier and we all set the date so if the competent authority can not make it then we still have them hack headpad as the record as the recording for the competent authority yeah so also which date I mean we um which mini hackathon will be talking about what issue is also uh agreed and also voted by the uh by the participants where where do you do that voting is it online person there's some uh some I mean active participants will have a restriction page on KKTX and they will send out the link on Slack so they can have like in the next week or the two next next next weeks they will have a agenda and also set up a restriction page so let the participants get to know which which mini hackathon that will like to join yeah that's nice so are we going to have a break yeah so there's some snack behind so um so yes how many how many minutes do we have for the break a minute break okay it's just yeah yes is there a way to log in to pull this without Facebook or Twitter is it I don't know yes there is but you have yeah yeah it says optionally connect to see friends and people you know in the visualization and I have no other way okay to see to see friends and and you know other participants you it has to be social media because people don't have a a pollist social account you get a pollist account but it's not I know who my friends are and I can see how other people are voting if that's the case then what other so like what is pollist's data retention in regards to social media data do you know like just uh whatever is is public which is your name and followers for twitter I think something like that okay um yeah but we are encouraging everyone to have their own deploy so we own zero data of which there is a well we should talk about uh dockerization of polis and there's there's a few there's a few attempts that could use civic tech community help when it comes to packaging polis in a way that respects whatever data privacy preferences various communities have when you come to the back if you can bring your cups if you have them too um yeah The end of the day, I can have all of you guys tonight in a guy who's an annual general As you make your way back into the main space, please take a look at your name tags and note your dot color and move to the table with the according tent. Once again as you make your way into the space, please take a look at your name tag and if you do not have a dot on your name tag, please come see me, you're in the Researcher Professor's zone. Please take a look at your tag, note your color dot and make your way to the table with the appropriate tent. Are you blue? Yes, you're a remote participant. Actually, wait, no, no, no, no, no, right here. This is like a new name, so if you add it. Can we just move these up then? Yes, they need a table. Oh, no, no. This is remote, this is citizen. Sorry, what? Yeah, she's more, let's just put it here. Sorry, we're going to rearrange a little bit. So you want to sit on the other side of the U as well if you can. Sorry, we moved the orange to the one down. I think Hordelia is remote. Hordelia, will you start following this tag? Remote blue. Yes, with a blue tag. Does anyone not have a tag? Okay, okay, may I have your attention? May I have your attention please? Okay, so after the online opinion collection, so we have, we can imagine that, okay, we have run a polling survey for a month, and we have the raw data, and maybe it can be a round or several rounds of online opinion collection. So we have the raw data, the raw reports of online opinion collection generated on polling. And we can also, may have also had a secondary study or presentation on the discussion. So we can probably be made by facilitator or made by the competent authority. So we have these two materials and definitely have to submit them on VTIO and platform. And the mini hackathon, if the competent authority and the contributor things, they are ready. So we can go on and initiate the consultation meeting, which is at stage three. And this is, I will talk about the preparation for a consultation meeting. And so in the introduction, I said that the facilitator should participate in the VTIO process as early as possible. But if we haven't had a right facilitator in the earlier stage, then at least at this point, we need to find a right facilitator. And he or she maybe can be recommended or suggested someone recommended by the competent authority or by the contributors and mini hackathon. And we need to discuss with the competent authority and define the size and the scope of the issue. And this work actually has to be done again and again in a repeated way throughout the VTIO process. And we need to host a pre-meeting or also like a pre-training, which is important with the facilitator and the competent authority and mini hackathon to try to keep us each other posted. And so to also to let the contributors to understand like the facilitating style of the facilitator and also let the facilitator understand what the competent authority hope to discuss about. And the pre-meeting or the pre-training has to be at least one week prior to the consultation meeting at least so that we have enough time to prepare. And here are the things to do before consultation meeting. And this can be just a general to do for a meeting. So it's like round-down of the meeting. For example, like the task distribution, staff list and pay some flow of the meeting. And the agenda for promotion, like we can have press release. So like on VTIO, we have a pre-release from Executive Yuan. And also maybe prepare some posts, hopefully post on social networks. And we use KKTICS as the registration page for VTIO1. And also have a list of invited guests and participants. And like I mentioned earlier, the unwritten rules is only the invited guests and the registered. And the participants who have made contributions can attend consultation meeting. And the fourth would be the seating plan. And also if you have some funding, then you can have the attendance fee for the experts such as scholar, professors or lawyers, so on and so on. And also a selection of stakeholders agreed by the contributors at mini hackathon. And in the consultation meeting, we have a reprinted notice for the facilitator such as the background of the invited speakers and copies of standing seating plans and handouts for the facilitators as a material, as a reference. And we need to have an equipment list such as for doing a transcript. We will have a stenographer to write down all the speech statements. And also like wall banner and agenda poster, live streaming camera, restriction table, name tags, snacks and handouts. These are the general equipment for a meeting. And one nice thing about transparency and documentation is that as an editor, when I send out invitation letters or to draft a public post for a promotion, because all the materials such as presentation, documents and papers are all online. So all I do is just list our URL below. So it's quite convenient. And the invited guest will also see that you have all these materials online. So I think it's really good and shows a spirit of documentation and transparency. And about the setup, this would be more physical. And we have a standing agenda poster at the entrance of the meeting room. So this is the agenda poster of NCI case. And we have the title of the consultation meeting. So just to show you as an example. And we have the agenda of meeting table here. And I translate it into English. So the order of the meeting will be the reception and self-introduction of the whole participants. So they get to introduce themselves like 30 seconds or one minute per person. And after self-introduction, the facilitator can have its presentation to explain the process and also what we have been done on detail one on a specific issue. And then after facilitator's presentation, there will be a component authorities presentation. And then we will jump into a discussion led by all the participants. And after that, facilitator should come to a conclusion based on the comments that we have collected under the consultation meeting. And under the timetable, we have venue and the facilitator's name and title. So this is a giant agenda poster standing at the entrance of the meeting room. And this is a wall banner. We generally have it above the facilitator's head. So in here, we have a U-shaped table. So facilitator will sit at the bottom of the U and then just like where Tashana sit. And we will have the wall banner just right above her head. And it's huge. So it can be shown clearly on the left-hand channel. And here, did you see any difference? Because the upper one was the original one for NCI case. And the lower one was the edited one. So did you see any difference between these two wall banners? Maybe a little too small. Did you see a difference? The original one, this is a side story of NCI case. The original one, what generally speaking is it's not politically right because it used like a blue, color blues representing a male. Yeah, just shooting, how to say that, something. So the female with the color pink will be the victim. So I think this is really not right. Because we all source our design of wall banner to the printing company. So I called them and said this is not right. So we should use like a neutral color, which I think would be like purple. So then I said, okay, we change it to purple. And you can tell what gender victims or the actors would be. All right, so this is just a side story. So just maybe an example to say that we need to be careful of our materials. And especially when we outsource the design of the wall banner or in our poster to printing company. And this is the layout of the meeting room. So just like we had right now, we have a U-shaped table. And this is the projection screen. So also here we have projection screen. And the audio video crew and the camera will be right there. And the sitting plan. We have different stakeholders to be seated in different section. So in here, here's for the government. Oh, sorry. C.S. Sorry, the government should be in the researcher part. Oh, yeah, right. So, okay, now just give me your hand, please. Yeah, in place of just swap them. And here there is like academics. Oh, yeah, sorry. So you might need to exchange your, no, all right. And then civil society and the private sectors like corporate and business and companies. Why did you choose that particular order of arrangement, government, academic, private civil? No, no, it was wrong. No, no, no, in general when designing the room space. Audrey, it is what it is, I guess. So the reason, well, there's multiple reasons. The facilitator, actually there's usually more than one person in that seat. Usually we have a civil society facilitator and the minister in charge. It used to be a minister, Jacqueline Tsai, now it's me. And the minister is there actually just to give the opening blessing and the final synthesis and she's not supposed to say anything else. But the facilitator coming in from the civil society does the real facilitation. But it's important because then the academics and the government agencies, they're technically, they're invited by the political will of the minister. So it's important that the minister is sitting on the left-hand side, with the government and academic people invited by the government agencies. Well, they're still vetted and agreed by the civil society, but technically their invitation letter is from the government agency. So on the left-hand side is the minister and basically the people who are here because of the political will of the minister. And the right-hand side are basically civil society people and the private sector stakeholders reached by the process designed by civil society, representing the GovZero part of things. So that's like the atmosphere. And also because there's usually common language inside each group. So the need to translate and interpret domain-specific language, even the academic maybe expert on a separate domain, actually doesn't mean that they are experts on the domain of the private sector. So there's a lot of interpretation back and forth that need to go forward. So it's designed to be as simple as possible to have kind of cross-examination of issues when the same word doesn't mean the same thing to the two different parties. We can have two different cameras capture their respective faces and numberable expressions when going through. It's while having the blessing of the civil society GovZero facilitator as well as the minister. And it's pretty symbolic actually to just gradually fuse or link together people's general understanding on this meeting. So that's why we arranged a sitting plan like this. But I guess you can still adjust it a little bit. It used to be if there's participation from the youth counselors and legislators and so on, there would actually be a small table at the middle of it. And they are mostly as observers and not actually stakeholders, but they represent some other decision-making agency. So we'll put them into the middle of it as observers in the midst of it all. That's the general idea. Back to Evers. Okay. So who chooses? I think it was mentioned briefly, but I want to be sure. Who chooses the stakeholders in each of the civil society, private sector? So just very briefly, anyone who posted anything on Polish or discourse, that's not obviously a troll, gets an invitation to be present in the civil society and we never get overrun with requests. Most people just prefer to watch the live stream and type in the statements. So that's how civil society is basically self-elected. Private sectors were predetermined. They need to be actually in part of the mini hackathons where we identify the private sector stakeholders. It's generally agreed well before we even go to the Polish stage. The academics may be invited at the last moment. As long as there's some government agency that feel there are some factual issues that need to be cleared up by academic contributions, they have a fixed number like five or something, a number of seats there and each government agency can suggest a list of academics on an ordered priority list and the ministry will end up giving the final say on who to include, usually to encompass as much different fields as possible. And again, the government agencies are predetermined at a mini hackathon level. So those two are fixed and these two are fluid. That's the general shape. And also we have a virtual space in the air and so there's a chat room. There's a live stream chat room and the facilitator is supposed to, once in a while, bring in the issues channeling it into the face-to-face meeting. Yeah, so facilitator will always need to have a laptop or iPad around. So he or she can check the online opinions, by the way. Oh, yeah, I have the, okay. Yeah, so the facilitator will bring in the insightful or valuable opinions from the online chat room. So later, like we have on Zoom, we can have a, we have a remote online participants here. Then we can practice how it works. Yes, and the facilitator calls the shots throughout the whole meeting. That's another rule. So everyone should respect the facilitator and let the facilitator to control the pace and flow of the whole meeting. And on-site registration is not welcomed. So we get to, because we have the live streaming channel. So we will prepare it in advance for everyone to have a name tag on the live streaming. So whenever a camera is pointed at someone, we have a, like a name tag, like a name tag above the screen. So the online participants can immediately tell who the participant or who the speaker is. So on-site registration is not welcomed because it's not possible for us to immediately have a name tag for everyone on the live streaming channel. Okay, and the method, we call it the ORID method. It's a suggestions or recommendation approach for the facilitator to have a neutral mindset. So it's just a way of thinking. So now we call it ORID method. It's also called focus conversation method. So it divides people's comments into four stages from objective, reflected, interpretive to decisional. And you can see that objective represents facts and data. And I use VITA1 as example. So VITA1 is a project and it's a fact. And reflected will be emotions and feelings. So I think VITA1 is a great project. And it's my personal feelings, my personal perception. And interpretive represents opinions and values. So VITA1 is a project that should expand. So it's more like opinions. And the decision will be the conclusion or decisions or consensus. So we concluded that VITA1 expands within a month. So this decision should be based on the OR and R and I. So this is an example of how we use ORID method. And it's a really nice, like a rational thinking approach for the facilitator to be neutral. And to be neutral to listen to the participants' comments when hosting the live streaming consultation meeting. So we will move on to Exercise 3, which is an abstract consultation meeting. And since you already state according to the color of your dots. So CS, would you like to explain the rule? Yeah, so we're gonna abbreviate this a bit. And also you're assuming, you're gonna role play a little bit based on what your section is. So we're gonna ask you to do is to put on your citizen or your government hat and think about your position in regards to our polis statement or polis like question that we had. If you want to take a look at the report, it's a VITA1-report. That'll give you the full set of data that you have. So what you want to do as your group is we're gonna give you about five minutes. You can take a look at polis at the report and kind of come up with a position for your group here. Like what is your stance? Imagine that you're gonna come into the stakeholder meeting and you have a stance. So maybe the government's stance is, you know, they think that no matter what, they are going to have control of citizen data or resident data and they don't want residents to ever have an ability to share it. Even though, you know, they might see whatever polis is displaying, they might go contrary to that. So that might be your position. So as a team, spend about five minutes coming up with your position and you'll get about 30 seconds to one minute to be able to state your position and Audrey is going to facilitate a stakeholder conversation with all of our individual stakeholder views and using this polis data. So in the beginning of a consultation meeting, we're massively oversimplifying this, but people are given a polis report usually in the form of a slide share days before the consultation meeting that outlines two very simple things. First is the common values that people have despite their differences. And the second is that the openings to potential solutions that could work for everybody. So just using this polis as an example, it's clear that nobody really knows what data is being collected on them by the city. Only a very few fraction of people think that they know what data is being collected and this is irrespective of whether they're met fans or some other fans. That's the point of contention, obviously, but we're not getting there. And the second thing is that of the common value is the statement number five, which is, Resonance should have a clear accessible way to understand what data the city has and it's important as statement 14, important that residents know how their data is used and most importantly, who has access to these data. So these two statements are almost synonyms. And interestingly, a more reflective statement is that more polis network camera on the street by itself will not make anyone feel safer. And so that's another kind of opening. So if we just go by those four statements alone, a natural opening will be. So if people feel, you know, not safer about police network cameras, what will make people feel safer and how could the city provide the data in a way that the residents understand what they kind of data they have and who has access to the data that is being collected by police network camera. So one possible opening is just to focus on police network cameras and in which case this will be sent to the academics and the government people on the left hand side. So they have some time to prepare a presentation that outlines only the factual part. So not their personal reflections of how cameras make people safer or not, but what existing deployment of cameras are, what are the current rules in sharing the data that they collect, are they being used in statistical ways or some other ways to improve the public good or whatever. So usually the sequence will begin with the government making factual responses to those common values and statements so that everybody is on the same page. Now everybody can challenge them on those facts of course, but usually it starts with the government making presentations and the researchers will make some possible interpretations and openings very brief ones since they tend to go on and on to facilitate her will. Time control with the experts. So after this initial like 10 minutes in the usual collaboration meeting, then it's up to the stakeholders to share whatever they want to share and facilitators role is to gradually fuse into a common solutions that seems to have the consensus of everyone. So, but, well, you don't have a week to do the research. So we're going to massively abbreviate it and just pass the microphone around a little bit, right? So just people make short factual statements or mock factual statements from the government table. We'll give you like five minutes to talk amongst your group around how you want to prepare your statements. So just focus on those four statements since we don't have much time. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. All right, we're starting in one minute. All right. So let's get started. Do we need to do this whole clap if you can hear me again? Okay. Yeah, okay, clap if you can hear me. Clap twice if you can hear me. All right, that's such a nice trick. All right, it's so effective, yes. So, right, so this time we have 20 minutes rather than two hours, so we're going to abbreviate a lot. So anyway, so let's first, there's supposed to be a round of self-introduction and we assume that's already done. And then afterwards, yeah, why don't the city government respond to the Polish statements and share with us what's the current status on data protection and sharing of the city government. Speaking on behalf of the Economic Development Corporation, the city, first of all, want to just remind everyone that a great deal of personal data that the city collects is already available. People simply need to file a FOIL request under the city's open records law, and unless there's a very pertinent reason to withhold it, you can get that information. So clearly some of the comments from the public suggest that we have not done a good enough job of educating people about what they can already get access to. But in addition, with the concern that we want to express is that as people evaluate what is to be done, we need to further strengthen people's ability to manage their own data that we not impose onerous new regulations that might choke the tech industry in New York or all the innovation industries and cause them to leave the city and go elsewhere because we're coming up with things that are just impossible for developers to handle. My colleagues may have additional things. Hello, so I am working in the Department of Technology and Innovation, and I think that the statements that are, like the ones that got the majority of votes, vast majority of votes were through, one is a concern about understanding what data we are collecting from our residents and how are we using it, and we agree that we need more transparency and we're going to move forward and focus our efforts in this aspect of what was said during this exercise. But I think what's going to be harder is the other statement that says that residents should be able to access certain data sets for organizing but not raise privacy concerns. And I think that arises a lot of questions that we would like to hear the experts, the academic, civil society and private sector, what do you have to say about that? Because if you're saying that you want data to organize it means data to reach out and so how not to violate privacy concerns and at the same time empower people to get more connected to each other and to their community that has same beliefs and same opinions and will make benefit from understanding better who's with them and what. Okay, so the general idea is that you need more transparency. You agree with this general sentiment but how this transparency does not interfere with the privacy concerns is that something the academics may shed something on. So any reflections about this? Yes, we are from a think tank called Open Data and maybe different levels of data. And so from the polls it's clear that the majority of participants responded that they have high speed internet and they're old enough to vote. That means that we are only talking to a very specific group of people who have responded to the poll and our suggestion is that we have to reach out to more communities to diversify and be more inclusive of our data suggestions and legislations. We believe that from the poll that most people don't know that data is being collected on them by the city and therefore we want to pose two questions to the government and the citizen group. One is what is the role of government in educating its people about open data and what are the data that what is the data that the government is already collecting on citizens and from that we would love for the citizen group to respond with how do you feel about those data being collected what do you want to include or take away from the data that is being collected on you and from all of that we would like to suggest that there needs to be different levels of data privacy and openness for government citizen bodies and private sectors but we are here to talk about that with everyone else. But that's great. Any other points from the academics? Okay, right. So it seems that people are generally want a very transparent kind of metadata list of what data is currently being collected. Can I get a very quick factual check on do you already have this in the government, in the city government? We already have been working on that and it's a long process that is not yet concluded. And we're excited to roll out a series of educational brochures. Okay, that's great. So there are actually already brochures. Are they in like PDF form? Like are they accessible online or at the moment only in public libraries? Okay, so it's only in PDF form. So that's a fact. So that's our introductory briefing. So people from the private sector I'm sure that there's tech industry people here that are supposedly benefiting from the simple regulations do agree with the existing directions that the academics and the city officials are going and what are some of your hopes and fears looking at this current picture. Thank you for having us. Well as the economic engine fuel and job growth in one of the greatest cities on the planet, New York City businesses already face one of the highest tax rates in the nation. And we concur with our colleagues in the Economic Development Corporation that, you know, onerous regulation is not the answer. We believe that self-regulation in the private sector has worked and in fact we'd be willing to share some of these solutions with our government friends for a small fee. And we would hate for any additional costs of regulation to be inevitably passed on to the consumers from New York City. We concur with some of the statements said by the think tank about that as the private sector providing a lot of different services we would like kind of a special degree of access to a lot of private data so we can better serve residents of New York City. And yes, as my colleague mentioned earlier, we are, you know, the tech giants in the space. We have over 30 years of experience with big data, so we're really well equipped to provide services for the government if they need a more secure, if you know what I'm saying, way to protect their data. We have a little bit more experience in that department. Hi, thank you. I just want to say how important this discussion is and we we we believe since we have such a strong relationship with all of the citizens of New York City we're committed to a public private civic partnership to help design the systems that are going to work for the citizens of New York. That's great. So can I ask just one quick clarifying question when you say self-regulation what do you have in mind? Can you come up with one particular example of the so-called already established self-regulatory thing that's going on in the private sector? Well, any of you can come up with it doesn't have to be true. No, it is true. We love our customers and we are sure to present detailed privacy policies every time someone signs up for a service which they consent to. Right, so privacy policies and informed consent. Okay, right. So does this make sense to you? Are there any citizens here who want to ask something? Our demands are very simple. We want to know what data is being collected by whom and for what. We want it to be easy to access and easy to understand. And we don't necessarily have a ton of confidence that the city can protect us or necessarily utilize it in our favor so we're a bit weary. Right. Just adding to that also we also want to have the ability to know all those things and also have the ability to turn off what we'd like. Turning off as in storing or? Updating keeping the data up-to-date or up-to-out of processing. So both access to how it's currently being used and also either editing or opting out of further processing. Okay, yes. The sovereignty of being able they may need to collect our data for some particular purpose. The age verification location verification but we should have the sovereignty that that isn't used for marketing materials and that at any one particular time period we can have that shit deleted. Okay, which is well, send them to this. So any other inputs from our something? Okay, if not there's some online statements as well. So I'll just pick one for the interest of time. For the person who said that data collected by the city she tried figuring out how to access data on May and it was totally impossible because it already costs $150 per request and nearly 20 hours of work and so no one can really read through those detailed policies, right? So is it so quick fact check, is it actually true that one has to pay like $150 That's incorrect. All freedom of information requests are free. There is no cost to the resident when they make a request. So I have no idea where that $150 is coming from. By the way, I do want to add that our city is the safest city we recently rolled out an initiative to protect Yorkers from cyber war and you can all the city public Wi-Fi services now have had additional protections added and as a user there is a free app that we've made available to people that they can download which will tell them if while they're browsing the web using our Wi-Fi if they're visiting sites that are known sites containing malware. So that makes New York the safest city. We are doing everything possible to protect from cyber war. We're second to none. Thank you for your commitment. By the way, everything I just said was announced by our mayor a month ago. He literally said those things. Yes, it's very important to have that commitment. Sorry, I see some people raising hands about the FLIA or some other requests. When I filed my last FOIL request to be delivered via hard drive I still had to pay $0.10 per record that was from the Economic Development Corporation. I'm sorry, my mic isn't working right now. I can't. That's a future point of clarification we can follow up right after this meeting and I see that everybody really agrees on a more transparency on what data is being accessed to and used for the right to keep it up to date or opting out. Are there anyone right? Yes. Hi, NYPD. Just to remind folks that we are constantly protecting the citizens of New York that's our number one priority. Any issue involving privacy while we work on at the request of the mayor we have to recognize the realities of the situation of course we're a center of global terror as well by the data that we collect no one would want us to share I mean I could go through a variety of different contingencies and scenarios about what that would be but obviously our intentions are always to be as forthright as possible with New Yorkers and to work with the mayor and various city agencies and civil society organizations to ensure that we're doing the best job possible but we also have to be aware that this is a balance and that security comes first. Right, so you said that security is important as for both the application and for you know safeguarding the privacy concerns I think everybody here agrees with that is the underlying cybersecurity but I'm very happy that you're committed to it so about the access to the current users and editing and keeping data up to date as well I heard that the government is pretty okay on this regard even though technologically they may need some help and I understand that this multinational here already has something that they're willing to share so about this particular reflection about more transparency and more system control are there any other statements that's as to the current users no? Okay, right so and the other point that's being brought out was that of this conversation they're mostly from people with high speed internet access and we do need to include more diverse people into this conversation Does anyone here regardless of the sector have any experience in reaching out to people who do not have high speed internet access and educate them about their right to data and about policies of each and inclusivity yes In the business district that we participate in we do provide high speed internet access there is some advertising that pays that facilitates that service but we've allowed anyone who is a shop any person who comes shopping or just strolling in our district can access the high speed internet okay that's great and say the structure that you offer at no cost but some advertising to people yes so thank you for providing the option are there any issue or any thoughts around this kind of free basic internet paid by advertising access any thoughts around it if not we're just keeping it as one option to increase the diversity and access to public information so I think that's pretty much resolved and we don't have any time left either but that was just a very quick markup of a potential to contributed reflections to address each other's reflections to actionable like outcomes and it's not the highest quality facilitation I've done but usually you can come over something that could translate into future actions that could be tracked on part of the Taiwan platform especially if this is part of the public private citizen permission program as suggested and so yeah I think we still have like 10 more minutes so if yes are actionable items usually discussed during the facilitation session also so if people have made promises like if like there are contention points can we close this yeah so why are the poor always the most surveilled communities that is actually a great question I wish I have two more hours to go into that right yes typically how much time would be set aside for this period of discussion right so two hour at a minimum and usually it's from 7pm to 9pm on a weekday and that day that this time slot is very carefully chosen so that everybody around different sectors put in roughly the same effort and the venue is within the administration it's one of the designated multi-stay holder meeting room in the administration itself and so one of the reason that we don't allow registration is that there is armed guards to the outside of the administration building it's really seen as symbolic as this is taken by administration as one of the ministry or like cross ministry level meetings that has fighting power to all the agencies involved so the physical side itself is also very symbolic and the time slot which we tried various different configurations but a weekday usually Thursday at 7pm to 9pm that could be extended to 10pm or so is a balance between the effort that different sector people have to put in and also it doesn't compete with other more popular television shows that people have and so it ensure more online participation because it is turning into one of the more interesting television shows so thank you for asking but yeah we really had to compete with other life house and other popular live stream shows for people's attention any other yes and what is the outcome so the outcome is basically a set of factual corrections or factual clarification that's needed for example does each I request actually required and are there sometimes there are multiple ways there are ways that cost more there are ways that are free but the ways that are free are not known to people that's one possibility actually that we see usually in such discussions so usually the minister right on the spot because everything is kept in the transcript so we can annotate the transcript like in real time and meeting to ask each agency to post a follow up point by point respond to the factual the objective layer of things so that at least everybody learns a little bit more about what's currently factual and going on but may not be a widespread knowledge to people so that that always happens and the second thing is that we check people's feelings so for example it's very clear that people here have some very shared sentiments but the written form may not allow them to go into a lot more detail so that is being expanded as we can see here of what the hopes and fears and other emotions actually entail in terms of personal experience and some people share their story and things like that so that also lets the policy makers when they see similar reflections or feelings in the future they have a better understanding of the context where it's happening and also they have the contact of these people where they can contact like at any time really to serve as potential contributors to future meetings that's a reflective part now the interpretive part or the possible ideas for example a PPC partnership on bringing high speed accessibility to people who are not as well connected that's one possible actionable result of this meeting although it's kind of orthogonal to a regional topic what's much more likely to be actionable is to set up a transparency report-ish framework where the various sectors work together to let people know what data is currently being accessed and opted out on that portal so what's likely to result of that particular portion is collaborative workshops to co-determine the user experience of that particular data access portal so if the because it was as I understand the agency's volunteer already to build it the next thing for the ministry to do is actually to just secure budget for that part to happen and the seed input to the vendor doing that collaboration meeting will be the transcript of whatever people have said that particular part and I also see that the educational material currently only in paper form in brochure can actually do some expanding as well and that's another possible action item so right after the meeting in the next mini-hecoson each of these areas will be explored and each agency who are in favor of this idea will outline both the resource they have and the resource they need to make it happen and that will result in other actions that will be posted on the vtaiwan bullet so as I mentioned in the orid method the facts the feelings and ideas these three are taken care of to some degree in the vtaiwan consultation meeting but it's not the place where we do decisions so all the decision on parts are the second diamond in the design thinking happens after this meeting so this meeting is really just a little bit in the first diamond after this diverse polis stuff to try to census us things a little bit so that we can agree on common problem statement and things that could be solutions in the middle of the two diamonds and for example in this mock meeting we end up with three possible touch points and each of which will grow into its own second diamond afterwards but people here get notifications and invitations to those future meetings it's a long answer so can I it's fine okay so I just noticed in this session I noticed in this session the business and department officials kind of steered a lot of the conversation and do you find that that happens not necessarily between these two camps but do we find that like there are some groups that steer conversation and are there steps that facilitators take like either by balancing time or putting out provocations or something to get more balance first question second question is about like the the detail richness of the conversation I feel like if the outputs from this session are meant to be co-design prompts and so on for subsequent work you know in order to get a full sense of the landscape we kind of have to like define what we're talking about we're talking about medical data you know public health data you know are we talking about surveillance footage are we talking about IP addresses from hotspots you know like when we're talking about data writ large it's hard to get into the details so I wonder what kinds of things we work on in the co-design session after so that's why we always use reference classes if possible like police network camera right so that's one reference class that we can use as a motivating sample by no means exhaustive on the facilitated sessions and we then gradually branch out into other kinds of data and other kind of uses in maybe focus groups or co-design workshops after the consultation meeting so consultation meeting is really only meant for people to voice their concerns their hopes and fears and their general ideas and it's okay to not have consensus at the consultation meeting and with even you know the this parts where the facts are even disputed it's impossible to have a real discussion around that so one of the action item is just to clear up the facts now the facilitator does have to balance the time especially the civil society people may need a lot of interpretation and translation that's actually the job of the academics so supposedly the academics here are supposed to when people raise concerns of fears or whatever a facilitator should cue the academics to reframe their language into the context of what's being discussed and we don't get the time to do that here and also the online people tend to be very vocal so again at points where people are generally feeling pretty calm and comfortable and resolve things provocative questions from the online parts start to come in and then we do another round resolving the issues like the correlation between the poor and the surveillance that that's actually a very good provocative question that we can easily spend another 20 minutes on and to balance the time so yes that's one of the things the facilitator need to do and the facilitator really can use the online part of the chat room as a process that actually there's a lot of co-editors online that try to sort the relevance of the online conversation that are pertinent to particular areas the facilitator is doing that's the co-facilitating team usually we have two to three people doing that in real time on hackpad this is directly related to the workflow with this aspect so this is zoom right and do you have like a full fledged like professional zoom account I assume and then with that how does this zoom kind of like white boarding integrate with the hackpad this is your notes and then you said that there was somebody else sitting in the room and like transcribing yes so right so on the currently we use youtube live but yeah it could be more anything the layout is like this chat room here and whiteboard here and right and then the camera so usually when two sides are talking we see two people and with their respective name cards actually so right it's laid out this way on a typical youtube live channel but at the same time and it's pinned to the top usually the link to the hackpad so people who don't have you know earphones or people who are not visually inclined can follow exactly the same discussion but explicitly in the textual form only on hackpad and the hackpad will always start with a embedded youtube right so you can always click and go back here and then and people here just summarized the points that's being made in the face to face meeting and then there's a section of the incoming inquiries right so incoming questions really that is being copy pasted essentially from here and ignoring all the trolls into here and so this is actually what the facilitator sees usually in another tablet so both the recorded summaries so that they can copy some of this into the whiteboard here and as well as the more pertinent questions across the from the live streamed chat room that they can bring to a face to face provocation so you have two just for clarification you tend to have two tablets I assume that they're like Apple tablets with the Apple pen one that's for your whiteboard and then you have another one which is just the hackpad that's right that's right exactly so this one usually we use ipad well ipad pro but now ipad works too with apple pencil but before that we use a normal laptop or even large whiteboard with still stylus technologies but now we're mostly settled on apple pencil to the whiteboarding part and the hackpad is usually kept on another tablet and we usually use tablets because that doesn't create a distance between the facilitator and people because like it's not like it's not shielded I was kind of I unexpectedly experienced like the frustration of being in a meeting like this and feeling that like certain issues are getting passed over and then things just move on so like for example it seemed to me like from the polis the most contentious and one of the most important questions was about like surveillance cameras and we kind of like got around it and we're like okay time's up so I was wondering if you experienced like if you have any anything further about like that experience of people feeling like this process can't encompass what they're hoping to achieve and getting burnt out by this kind of deliberation right so keep in mind that we only spend maybe 15 minutes in a what should be a two hour process so in the two hour process it's usually there are touch points where we quickly summarize what's been discussed before and we ask around what people feel that are exactly the things that are being passed over and worth elaborating usually that's during such segmentation time that we bring a new batch of input from online people also so there's usually two to three synchronization points during the two hour meeting but still yeah there should there's always some topics that are not explored in depth enough in such meetings and usually what people do is that they type their points that still needs clarified and they get collected from the chat room here to the hackpad here and then the minister at the end of the conversation like suppose Averos is the minister she will read from the hackpad and say we understand that there are such this is like five points that still needs more elaboration on and we task the ministries within seven days to go back to the VTaiwan online forum and provide detailed explanation of these problems and open more collaborative meetings if needed be so usually the ministry's role really is to summarize what has been reached and to outline the openings that still needs clarification and so the host barrier is that this consultation meeting is at the middle of a VTaiwan process right so it's between the online divergence part and the second part of more focused either special interest group meetings focus group meetings or some other process so this is like a continuation point between two divergence parts so this is usually the spirit of the ministry try to convey in that there's always another future meeting that you will be all invited that is the spirit yes I want to go back to I think David something that he had raised about the breadth of issues that are covered and your invitations or who sits at the table because you have this lumped category of civil society which I'm assuming is a mix of citizens and groups or associations that represent special interests like how do you engage the private sector table here is because it's more of the private sector thing but in NCII it's actually large NGOs that sit here right so like more old power organizations are actually in this corner and the generalized civil society are mostly people who are not representing other people but themselves that's the that's the real distinction but yeah depending on topic we may label things differently but that's the real difference so people in this corner are people who think they represent other people so you don't have business then it's one or the other do you have no you can have both like in Uber it's both the union of taxi like the head of the union and Uber itself and some other taxi companies they're all in this corner like they all presume to speak for thousands of other people but the generalized civil society themselves are their family right how do you signal the conclusion of a meeting in a way that people feel good about right that's the hardest part so we always go over time in B-Town consultation meetings that's not a secret so people receive that it's from 7 p.m. to 9 p.m. but actually it's 9.30 we have one that goes almost to 10 p.m. so usually it ends with quite a few rounds of just simple recapping of what's going on and just assessing people's feelings of whether they feel it's good enough now if it's as diverse as this right the threshold is just you know people won't kill themselves because of this right but if it's more convergent it could be people are generally happy with it but not happy another fine consensus so that not one letter can be changed but generally happy with it so the facilitator will sense the divergence and verbalize this degree of divergence and check whether this verbalization like we can live with it it's okay with people here and then read out the action items tasked to the agency but that part is usually after confirming the people's general sentiments the reading out the actions is delegated to the minister but the people crowdsourcing the statements and so on they're usually already prepared maybe 80% of what the minister has to say of course she can add more things to it do you use like any of these like hand signals to gauge sentiment? we try that yeah but no not anymore so online chat rooms we use a lot of smileys and things like that but no in this kind of meetings so far because everybody can see everybody's face we find the more like hand gestures and things like that they're more equipped if people don't have at least 5 minutes to 10 minutes of their time to say things and also it works if people can't generally see each other's face but in such an arrangement they'll then need it thanks I just one thing I just want to thank you Audrey for giving me such a nice way of saying we are obviously over the time that we had it's the real deal here so respectfully if there are folks that do have to leave that is quite okay otherwise we're happy to take a few more questions and continue this discussion is there prime produce people? we should wrap up gradually but there was a question just building on that last one when you say recapping so that people are happy is it that the various stakeholders feel heard is it that they have an action point to be discussing the mini hackathon outlined in red what is that combination that you found works and how do you determine that? it's mostly an art there's a few rules of thumb and they're just meant to be taken as rules of thumb first that it's important to recap always the common values so if the polis surface something that people generally are their common fears their common hopes their common values that identify as important as important to go back to those and to read again for the people involved that their contribution makes those common values even more manifest and that tend to bring everybody to a higher function level and then the other rule of thumb is based on that we face the contentions and potential actions that could lead to resolution of those contentions that's also worth reading out loud and one trick is that to read exactly as the wording that's being kept on the on the whiteboard so that people feel respected for the particular words that they chose and if there are particular groups of people pockets of people who are not okay with that particular wording and so just alternative wording it's worth it to also repeat that in the recap so that you'll respect it and kept in the full transcript record that applies doubly actually to the online chat room so what I just demoed is actually a counter example because there's a bunch of very relevant online chat room statement that were not being read out out but if there is time those should totally be read out out any other final questions yes I'm curious what do you wish you could improve about this process there's tons of ways that this could improve so I think three main things the first thing is that this process what we found usually works only if people have first-hand experience that are broadly speaking compatible so that when people hear the same word like UberX they have some idea of what people are talking about so like for data integration before the consultation meeting the actual process actually goes through like three focus groups like five pre-meetings or whatever like that just to get each word and its glossary and definitions fixed in place so that everybody can know what we're talking about so that we don't you know go like this and instead of explore one specific branch but the time and resource needed for pre-meetings is infinite you can always do that infinitely and usually there's a time constraint like a consultation meeting really has to happen in two months or three months like because of legislative or other political forces at play so we can always do the preparation work better but the reality is that we always end up with some irregularities that are nevertheless taking a lot of time in the consultative meetings that could be avoided if we had to do more preparation meetings but it's okay I guess so that's the first thing is the common imagery the second thing is the facilitative method itself what we are doing this is kind of a whiteboarding exercise there's no common code for what is an action what is the fact and it's all up to the facilitators personal and that's where trying to improve and that will be tomorrow's workshop where we're trying to bring a shared code to these kind of processes and for the benefit of online participants especially that's very important and the third thing is to increase the regularity of going through this process the digital communication act that's going to be passed about two months from now explicitly said that all digital issues that concerns multiple ministries need to go through some process like this but before the act is passed it is somewhat selective and because of this selection we're kind of dependent on the goodwill of agency of following through those action items and so sometimes the agency if they don't have the resource they will sound very reluctant or defensive and that degrades the quality of the consultation meeting so that's the third thing that I think could improve so I guess that's it sorry one final question sure Taiwan obviously is a democracy administrations come and they go what is the what's your position as digital minister in the Taiwan's position in terms of like a long term democratic scene of Taiwan when an administration might come, a legislature might change, parties might change do you feel like you're in a position where you kind of have what would call bipartisan support and institutionalize this if yes how if not what's next under various contingencies right so the veto on part is not institutionalized at all at this point what's institutionalized is the national participation platform which is has like almost five million participants out of a 23 million country so it's a large number of participants but it's much less research-ish it's what all the regulations all the budget items and all the petitions can go through and that part is very well institutionalized there's a national regulation about it there's even a regulation about participation offices and there starts to be regulation about in each ministry how the participation offices are going to organize and so that part is the institutional arm that we're working with now v-taiwan itself especially when it concerns truly cross-sectoral multi-national like uber-ish things I think it really needs to be institutionalized but maybe not with the name v-taiwan that's why in the digital communication act we end up with the wording that says a open multi-stay holder process not necessarily initiated by the government with it on this wording so that it doesn't need to be v-taiwan or f-zero it could be the internet governance forum it could be any other community that satisfies this standard could still by law have a partnership relationship with the government so v-taiwan is not monopolizing this contact point with the government that's my personal standpoint but many other v-taiwan contributors differ with me on this opinion so take it with a grain of salt that's why we need a large debate on the future of v-taiwan after the digital communication act passes so thank you very much well I can tell you what my plan is after this and that is we have a very important decision to make whether or not we want to go to a Mexican style margaritas type bar establishment or more of a local beer establishment both have moderate prices enter within a half block of here I was hoping that we could slido this I don't know if we are prepared exactly for it in which case we could do an old-fashioned hand raise if people have strong ideas if not we can all just gather over here and make the decision in an ad hoc manner but I feel like I'm in a position where I want to say thank you everyone for coming and more information from the organizers will be coming forthwith which means very shortly in moments just one more thing there's actually a short questionnaire asking for your feedback on you know the process and what you learned what you found interesting we'll send out the link if you can take it it would be awesome and if you don't we'll hound you tomorrow if you're interested as well prime produce recently had a really neat cybernetics conference and there's some neat books out over in the registration area that are cybernetics related to also like organizational structures and participation that you're free to peruse through but yes as you tomorrow's going to be real fun we'll start at nine o'clock slash nine thirty absolutely hard like definitely starting there will be a reception beginning at six thirty tomorrow evening which you are all invited to invite other people to although not particularly on the internet in a way that's widely distributed but other people are you know there's over fifty sixty people have RSVP'd there will be alcohol for donation style sale which will be very useful and we are going to be in great shape so this is it's all going to be here starting six thirty and there's also food and there will also be non-alcoholic drinks as well absolutely MC thanks everybody it was just one thing for starting um do you know starting tomorrow um one more announcement for those that are here that are coming tomorrow tomorrow it's absolutely necessary