 Thanks. So, yeah, hopefully this is not very many slides, so the average length of time per slide may be higher than normal. Let's see how we go. So the basic idea is we're interested in the spread of beliefs. These could be any type of belief, but we are focusing on discrete beliefs, let's say, about questions of facts. So the beliefs you're allowed to have, really, in our model, essentially, you believe that something is true or it's false, or maybe you don't know. You could imagine many more ways of doing it. You could have probabilities as beliefs, all sorts of things, but for now we're looking at this discrete kind of model. I'll explain a little bit about where it comes from. So the obvious questions are any kind of diffusion model. When do you get, you know, you basically got three states of a node, three colors, if you think about it, when do you converge to some sort of unanimity as the diffusion process runs sufficiently long? And in a question like this, it's not enough to be unanimous. Ideally, you would, in fact, unanimously have the correct answer, or at least a strong majority of people would be getting the correct answer. And then a more general question is which models are going to explain the data? There's a huge number of models in the general area, and still an interesting question to me as to how you can differentiate between them based on experimental data. We've only started a little bit here. So I'll give you an idea of what we've done. Where it came from is that the topic of belief revision is very, very well known. In philosophy, logic, computer science, people doing databases and artificial intelligence, it's important to know what do you do when you have a consistent world view. And an extra piece of information comes in, some proposition, and makes the entire set of propositions you had inconsistent. And you then have to revise that in some way by throwing out something. And there are axiomatic methods for doing this from the 1980s, which have been well studied for individuals, how you would do that. And then the question that Patrick and some logician co-authors studied was what do you do when everyone is connected in some kind of social network and they can influence the beliefs of others in some sense? So they're basically their model. There are two different types of ways of changing beliefs. You can be convinced that P is true if there's sufficient evidence for it among your neighbors directly, or if you happen to believe not P before that and you don't have enough evidence to convince you of P being true, you can become undecided. There are various clear rules as to how you should do that. And originally, we were interested in this model and studying it. We did some simulations and we thought maybe we could do something analytically if we dreamed a bit more. It was quite a difficult looking model. Then we decided we really should actually see whether it has any relevance whatsoever to the problem that we're looking at. So we decided to try a lab experiment. That's really what this is about. And as Valerie came in, being the expert on the experimental behavioral side, so the basic picture is that we've got our computers in the lab and they're all linked up according to a topology that we choose and we chose two very different ones. One was a complete graph and one was a directed thing with very different node degrees, in-degrees and out-degrees. And of course, the subjects, the participants don't know what network they're connected to. They just know that they have some neighbors somewhere in the room, but they don't know which is which. And we gave them five questions one at a time. And for each question, each one had an objectively correct answer and they were allowed to choose one of three options. And it was to make it, we made it a multiple choice. One of the answers was the correct answer, one was some wrong answer and the third was don't know. And at each or at first, everyone answers. Then when they're all done, you get some feedback about your neighbors and you decide to update what you can. You have the opportunity to update. And they had ten iterations of each question. So the total number, we did this in several different sittings because of the size of the room, but we ended up with something over 50 participants, five questions, ten iterations, each one. So there's a reasonable amount of data out there. You could always like more. So what are some of the issues? There's a lot of actual tricks involving experimentation, which I had no clue about to start with. The first thing is, of course, you have to have the right facility and you've got to be able to recruit people from a pool. And that was all taken care of by the fact that we have this nice behavioral science lab in the business school that Valerie and other people are running. You have to get people to take the quest, the task seriously. I mean, they'd undergraduate pool, they come in, you're paying them of the order of $20 for an hour's work. It's very tempting probably for them just to look at their phone all the time and just keep pressing something like that. So we had to try to induce them to reveal their beliefs in some way. We did that by giving you more money if you got the correct answer. And to make sure you paid attention all the time, we paid them on a randomly chosen iteration rather than just the last one, in which case they fall asleep for nine iterations and then take it seriously. In order to do that, you've got to be a bit careful. You also don't want them just to put don't know every time because it's too hard to think about. So we tried to incentivize it so that you get more for being correct. You're expected payoff, if you put don't know, is better than randomly guessing the other answers, but it's still not nearly as good as thinking about the answer. So who knows whether the utility functions of the participants were sufficiently stimulated by those payments, but that's all we could do really. And that's a big problem in all these experiments. Here's an idea of the questions. There's a set of three questions in psychology called the cognitive reflection test of Frederick. And of course, to audience like this, they all look pretty easy, but apparently they're quite hard. So that was the first one, five machines, five minutes, five widgets, how long does it take 100 machines to make 100 widgets? And we just said is it more or less than 50? That was what you had to choose. This is a slightly more complicated to want to state. It's not that hard. If I'd had a picture, it would have been easier. You have four cars. This is a very old one. It's been used in like hundreds of papers because it's considered to be very difficult. Something like a quarter of the population can get this right given a reasonable length of time to think about. It's because conditional logic of implication seems to be difficult for a lot of people. So you've got four cars. You have a logical proposition that you claimed you have to test whether it's true or false and which cars you need to turn over. So those are the two of the questions. These are what we call, I suppose, the analytical questions. We also had three factual questions. One of them was this one. Sort of supposed to be aimed to some extent at the target audience a little bit. Here's one of these urban myth type of things which we thought we'd put in there. People might have heard of it. And this one, of course, was designed to be impossible to do. So we tried to induce people without telling them directly to think a bit about what other people might know. At least that was our idea. It was hopefully that subjects would think about the beliefs of others and realize that in this case it would be ridiculous for anyone to even make a guess. For this last one, because it was so ridiculous, we gave the answer privately to a couple of people and we told all the rest of them that we have given it. We have given it to them privately. So we expect probably some different behavior on these different questions. So just as a note, this Wason test turned out to be difficult. So even though they are all past university entrance somehow, they still found it difficult. They had something like two minutes to answer the first time and then they had 30 seconds each iteration to update. Which now that we have seen, well not many other papers in the area, but it actually seems reasonably generous that amount of time. But more than, you'll see the graphs in a minute. Anyway, it was consistent with the psychology literature that it was a hard one. We thought this fast and furious question most likely you either know it or you don't and you would know that you didn't know it. And you would guess very strongly that other people in the room, since they all look a bit like you, some of them would know it. That was our belief, but we weren't sure. And we made this impossible question, right, as we said. So what are the key findings? First one, crowds not always wise, right? So not a surprise probably, but correct information was not always aggregated by the group, even in ten iterations. I'll show you what I mean by that. I think one of the key things that is interesting is that there are very different, there seem to be quite different behavior between different types of questions. I'm not going to present very many graphs and things here. We have a preprint that we're almost finished with vast amounts of analysis in there and there's still a lot more that could be done. But seems to be a bit of a difference between these analytical ones where you can work it out yourself without any influence from anybody in principle and factual type questions where you know that you have no idea possibly about this piece of information. And I guess one of the findings we have is that anyone, if you're trying to model this, the fusion of beliefs, you probably need to take that into account, the type of question you're talking about. There's a clear asymmetry between don't know and the other answers. Now one speculation we have is that if you forced people to choose one or the other and didn't allow the don't know, you would get probably worse results in terms of aggregation of the right result. But we haven't yet done that control experiment so we need a bit more money before we can do that one. But even so in the complete model we have, we were forcing people to choose between three and we're allowing them the don't know. There's a clear asymmetry between them. It's not that hard to guess why that is but I'll show you in a minute. And we found that there seem to be three groups of subjects. I don't think we expected this. So there is, we looked at all the times where people changed their belief, their answer. And there seem to be people who require almost no influence from anyone in order to change. Some who essentially never change until everyone is against them. And those that seem to adopt this majority but it's kind of a clear distinct groups of subjects. And of course any social science experiment we found that rationality was violated. If you think about that last question you know that some people have been given the answer and you're pretty sure that everyone else knows they haven't been given the answer. Plus it's essentially impossible to guess. They should all write put don't know until they get the signal from one of their neighbors of the correct and then they should just copy that. But of course they didn't. Some people put the wrong answer down right from the start. So you get some sort of behavior like that. Here's the correctness. So this gives you an idea of what we mean. The second question. So that the rows are the particular four different time experiments we had on different days. And these are the questions. The five questions. And you can see this Waston test. It was below 50% or no more than 50% on the first iteration. And here it went it was worse at the end than it was at the beginning. So that's you know clearly not very wise crowd there. And here when the complete graph they just immediately went realize that a large majority of people had the wrong that had one answer and they just went with that. Right. There was no reason to change which is a little disturbing. This is the one with the widgets question. Make this many in this five and five minutes with five workers etc. And it's a bit variable but at least overall they well here was a bit strange. Overall that least the majority opinion was basically right by the end. And here of course much better is to expect this is the one with the black and white dots. All they had to do was wait. Right. They should be able to get the right answer. This great wall of China thing is a little bit confusing. Sort of in between hard to know exactly what was going on there yet. This I think is gives you a better idea as what's going on. This is the answer type frequency. So black means you didn't answer at all for whatever reason fell asleep. Red means you got it right. These are the five different questions aggregated over the four different experiments. Red means you got it right. Greeners don't know and blue is wrong. And you can see the way something a lot of people were wrong. But you can also see not that many people were undecided there. That was the thing that surprised surprising in a way but not you know since it's an analytical question you know that you can do it. It's just amazing how few people could do it. And here there's a lot more people waiting saying don't know this that's the complete the right behavior you would think and here as well. You'd expect a lot of people just to sit there and wait until they get the correct signal from those who do know. So there's a bit of a difference there. The don't know is very asymmetric. So this is the empirical cumulative distribution function. So here is the fraction of people of neighbors who were your color. And here is the probability of switching to that color. And so when you're red some people will switch to red with not very many neighbors who are red. And then there's a whole hunk of fair amount of them who jump around the point where 50 percent of their neighbors are red. And then there's some who hold out to here. Green is similar but the key thing is the yellow is very different because in order to become undecided it's not necessary that you have a lot of undecided neighbors. You just have to have conflicting evidence I would imagine right from different sides. So you're more likely to and here there's a lot of people who are undecided early on and it continued like that. Well that's an interesting feature. There's not we couldn't find very much. There's a huge literature of course in social network diffusion stuff but in this particular area we couldn't find much. We just found this one like last week I think after we've done all this stuff. It's quite similar in some ways. They have a they had three questions here from the cognitive reflection tests but they allowed free-form answers. They didn't have multiple choice answers which and this is the correctness. This is how many of them these are the the ranges over the different experiments. So and they use different topologies. So here they had several different networks. And you can see here the question one which probably it was one of those I can't remember the the one we had with the widgets. I don't know whether it was the first second or third and but it's one of their three. They have this set up that this you can see here that as the time they only had five trials and instead of ten five iterations each time but you can see that there's some sort of aggregation of information going on there. But it was a little bit weird I thought what they were doing because they had a free-form answer and so it's quite possible that there's the it's quite possible that most people were wrong but they all had different wrong answers and so the correct answer was the plurality winner. It's the most likely one and so that would get people copying it possibly just almost out by luck I'm not quite sure. So there's a few things to look at there but that is actually the most similar thing to our work that we've seen. Future work let me just go to just mention a couple of things we haven't investigated yet anything to do with the subjects the particular people. Each of them we can go through and see that some of them will have changed more often than others we can try to in terms of these basic threshold models where you switch you're based on if you get more than say 60% of people saying green then you turn green right and that 60% number is property of you possibly of the type of question that you have. We could do that kind of we could get bounds from the observed data we might be able to do some kind of statistical analysis. I think there's not enough data to really get into it in as much detail as we would like and any model in fact you try and fit and there's always going to be an issue there. However we did find that as far as we can work out our results are not inconsistent with the original model that we started with and therefore at least it's worth studying that so we reassured ourselves a bit on that. We haven't investigated the role of the topology in detail we only had two very different ones and we haven't really investigated in the case where the graph is very different so you've got some nodes with very high and very low degree you could see whether there's a difference there etc we've got all the data we could dig into all of this if we felt like it haven't got the yet. So I've got a question for the audience is can we do it more cheaply because you know these undergraduates are expensive I don't know you know it's it's our budget is in the sort of low thousands all right very low and so you don't get very many data points that way and I think I'll leave it at that because any questions of anything detailed you asked threatened to Valerie and the group around me who are guessing and they're all saying something different very quickly up to two or three iterations I think it was one person who asked me to go into it you didn't always get it right so just get everybody's fake notes on the other one so one thing you can look at is to what extent people pay more attention to if you're sitting up for 10 periods there might be some people who are switching back and forth they see my neighbors switch and I shouldn't really trust them I realize they don't really know the answer right so maybe I pay more attention if I'm sitting there and several of my neighbors are blinking on on and off and the other ones are just staying at the same answer then maybe I trust the ones who don't know yeah we didn't we didn't tell them which neighbor was doing what we just said you have seven neighbors three of them said this four two of them said that and two of them aren't decided and they can't see the series of then it starts to get really I mean interesting but yeah then you could start thinking about letting them choose their own network who they follow and stuff like that but no here we forced them to they didn't know anything other than they're in a room with a whole lot of people that look like them and they're connected to some of them presumably they could have worked out um in the complete graph when it said you know you have 29 neighbors or they probably could work out that's the whole room apart from that strategic behavior we haven't looked for it I don't think we would like to find it because I mean you have to have very strange you don't know who the people are that you are annoying by doing that and you're going to potentially lose money because you're getting paid for even random iterations you don't know whether that's the one thing so I mean I find it hard to believe there would be a big problem but I suppose we could look for it just sorry there is a recent check on that you could look for people who are told what the correct number of white spots and black spots wasn't seen but make sure that they did that number that would be a good start should we just follow up on that yeah so that there can be done yeah you can check all of that yeah just just want to just going back to this one thing I forgot to say here one of the fun that their big picture finding here was that the group gets smarter but the individuals don't get any smarter because they're not surprising because they only gave them 15 seconds for each iteration and the whole experiment was an hour but the two competing theories is that one is that by getting feedback from your neighbors saying your answer might be wrong it stimulates you to think better and the other one is that it just stimulates you to copy them and their finding was that they're pretty sure that it's it's mostly due to copying which is not surprising and you don't really have enough time to think about it again so that's good though because then it doesn't really matter what order you ask the questions and it seems so there there's always potentially problematic issues like that seem to go away least we're fairly happy that there's a lot of major problem in this case where they have the three or four answers were that were their names given the answers or were they just given the same statistics that you would because if you're given an answer you can check it right it says yeah I have to check I mean like all of these papers there's like a seven page glitzy thing and 30 page appendix and you dive you dive into that and you click on there spreadsheet and you see that yeah that actually the data they make available there's some missing data that they really should have kept if you want to reproduce the experiment properly they didn't put that in there but overall it's not bad but yeah I can't remember whether they did that I suspect they just said they showed all the answers they would have to show the answers otherwise it would be hard to heard on this one thing yeah but you they only gave them much less time than we did I mean 15 seconds is quite hard to check on some of those problems right but you're you're right yeah there's a lot more information getting sent back in that group so the six you can imagine if you felt marked if you're a risk neutral and you think you're more likely to be right than wrong yeah then six isn't really enough to know push it towards no answer right so you might want it might be interesting to see whether you get a lot more undecided yes I mean there's just so many knobs you can twiddle on this machine right so we can do the whole thing again with totally different qualities or as you say change that I mean you could it'll be really nice if you can get each subject and have the work out somehow the utility or at least you know like some pre-test and pay them in such a way to incentivize them to do that but yeah that would be interesting of course if you put it too high it's no one's going to bother answering yeah very hard to work out where to put that number yeah well it's not clear where whether it was in the middle it's just that a randomly chosen iteration so the idea was we wanted them to say for all you know this is the last iteration the counts so make it count and do whatever you think is going to give you the best payoff that's what we're hoping they would do I mean who knows how many people do that you want to be right at the end why look at any of the other rounds except the night one I mean if you're looking at other people you can't trust what other people are doing they could all be just waiting till the last yeah well I think from the point of view of good serving energy and we should just not do anything until the last iteration that was what we thought would happen if we didn't do that it's certainly true that whatever you do in terms of the payoff scheme can potentially influence the behavior quite a bit so it makes it so tricky I'm hoping that it will be nice to be able to do this on a much larger scale in some way or you suggested one thing possibility but yeah real life scale internet experiments I'm not sure it was like the Hager beliefs I mean it's really hard to say anything about them in the end it's observation because nobody gets to talk to anyone else about the beliefs and convince each other or anything it's just they're observing their answers so even the participants are not necessarily thinking that hard about so the reason we chose 10 was purely just for practical purposes right if we keep them longer too long and we have to pay them more these guys only use five iterations in their experiment it seemed to be reasonably a reasonable number but yeah I think you it could be I mean I don't know you look at these things these are just the aggregate we're not looking at each individual these are just the statistics of the whole group so question some of those questions you can see maybe hadn't completely converged although I'd be surprised if it looked much different if we ran it for 10 more then you've got the whole boredom issue as well taking people taking people to take it seriously if you do too many iterations I'm just gonna do this yeah well it's disengaging time for further thought and I'm saying it's a possibility that if you run waste not for other periods yeah so why is it this stubborn minority of people who persist in this this this strange action not yeah yeah now I don't know what this is going to get applied to if anything eventually right so who knows how whether 100 iterations or 10 is more realistic in terms of you know whatever potential application you may have I don't know but yeah it'll be nice to do them but again it's a resource issue