 All right, thank you everyone for joining today. I know it's a busy time across the university with exams and finals and all of that, so appreciate you joining us. We're really excited for this penultimate installment of our fall speaker series. We're lucky to be joined by Tali Stroud and Josh Tucker, who will be co-chair to the external academic team on the US 2020 Facebook and Instagram election study. Tali is also professor in the Department of Communication Studies and the School of Journalism and Media at UT Austin and the founding and current director of the Center for Media Engagement and a co-director at New Public. And Josh is a professor of politics, affiliate professor of Russian and Slavic studies, and affiliated professor of data science at NYU. He's the director of their Jordan Center for Advanced Study of Russia and the co-director of their Center for Social Media and Politics. So we're really lucky to have them here today discussing a pretty urgent topic from cutting-edge social science. So I hope you'll join us or join me in welcoming them. Thank you so much, and thank you for hosting us today. We're just so honored to be here and really appreciate the invitation and a chance to share a little bit of the research that we've been doing as part of the Facebook and Instagram US 2021 presidential election. Can people hear me okay in the back and online? Are we all good? Just a little bit higher. Louder, louder, okay. Better? Yes. Okay, great. Okay. Well, let's go ahead and get started by acknowledging that Josh and I are two representatives of a huge team of people. Research like this takes a lot of different people. And so we just wanted to, at the outset, acknowledge all of the collaborators and contributors to this work, both at a variety of different academic institutions and within the core meta team. What we'd like to do today is first talk about the collaboration with meta, the decisions that we made there and how we came to do this research. Then we'll tell you a little bit about the research design. We're gonna focus on two parts of the results here. There are so many results across all of these that we definitely won't do justice to every part of these, but we're gonna focus a bit today on the on-platform experiences on Facebook and Instagram, and then we'll hone in a bit on polarization. And finally, we'll talk a bit about what we can and cannot conclude from this sort of research. So to begin, let's talk a little bit about collaborating with meta. It's a very interesting position to be in where the subject of study is also your collaborator. And so we wanted to really think through how we were going to engage in this research and really be thoughtful about that from the outset. A bit of background to begin with, the partnership with meta began in early 2020. There was a core team of 17 researchers that collaborated to design the overall project. And we added additional academic team members to fill specific needs on the various papers. We worked with NORC from the University of Chicago, and they partnered for gathering the data on the study. And the scoop of the study, the idea of the study from the outset was that this would be examining the role and impact of Facebook and Instagram in the context of the US 2020 elections. If you're curious more about the process of how this came to be, we wrote out an FAQ. It's publicly posted, you can search for it there on Medium. And we just tried to really give the details about how this collaboration came to be and how we thought about it from the outset. So encourage you to look there if you'd like more information. So when we approached this, we thought to ourselves, we really need to think about the integrity provisions that we put in place prior to initiating this sort of a collaboration and doing this sort of research. And the first thing that we did was we pre-registered the study designs. So this was including all of the research that we were planning to do on a date and time-staped stamp to website prior to initiating it. These pre-registerations are also available after the studies are published and of course as part of the peer review process. And this was a way to ensure that at the outset, we were specifying what we were going to do, that everyone was in agreement, that before we knew any results, this was the research that we were going to do and what we were going to present in the eventual papers. Before we began the study, we were very adamant that we wanted to ensure that there was going to be no pre-publication approval. We didn't just want this to be a situation where our finding was potentially problematic for the company and they said they would have some right of refusal, that was not the case here. So Meadow was able to check the research that we were doing first for its privacy obligations to make sure we weren't violating any of those obligations for their legal obligations. And then also in terms of feasibility, so that we came to them and said, let's do this billion dollar project. They of course have the ability to say, oh, I'm sorry, that's not feasible at the time. So those were some of the provisions that we put in place in terms of review. The way this was structured was that we had a variety of different pre-registrations that were put in place and each one we specified academic lead authors and these lead authors had control rights. So the event of any disagreement about what would be included in the paper, whether it came from other academics on the team or whether it came from the medical collaborators, those academic core authors retained the final control rights and they had the final decision-making power to say this is what's included or excluded in the paper. We also put into place a project repertoire. So not only were we studying all of these people and what was happening, but we were also being studied because this was Mike Weijker from University of Wisconsin and he was essentially an ethnographer throughout this project determining how things went and observing the process. He also has published some of the results of his work alongside the project results that we'll share with you today. We also have the data that are publicly available. So once the projects have been published, the data are included on an archive at SOMAR and ICPSR that this is at the University of Michigan and this is for researchers who are interested in replicating or extending any of the findings. So if you find any of the data here of interest, really encourage you to check out and apply. You do have to have an IOV approval, no sorts of steps in place, but it's all there so encourage you to take a look at that if there's additional research that you would like to do to extend what we're sharing here today and in the papers. So the process by which the paper topics were chosen, we agreed at the outset that the overarching context here was Facebook and Instagram and the US 2020 election and then we specified four key areas of interest that would govern the research project and so the academic team identified political polarization as the first one, the second political participation including vote choice and turnout, the third was disinformation, misinformation, information, knowledge and perception and misperception so we kind of lumped all of those together and then the fourth and final one was attitudes and beliefs about democratic norms this included confidence in institutions as well. So with that broad framework we then said about how are we going to work as part of this collaboration in order to identify the research projects that we're going to conduct and so we chose particular topics of study and then assembled research teams around these topics or around these ideas. We had some groups that were focused on experimental sorts of studies and I'll tell you a little bit more about those in just a moment. We had some that were observational studies so looking at what was actually happening on the platform and then we had some that were a combination of both experimental and observational and you'll get a flavor for all three of these as we talk through some of the results that we've been sharing so far. The first four papers were published this summer in science and in nature. Really encourage you to take a look at the full papers as I mentioned at the outset we're only going to be able to kind of brush the surface of the findings here but they're robust supplemental appendices with lots of details there that I encourage you to read. Okay so what was the research design? Today what we're going to talk about is these first four studies and first we have some observational studies that we'll share with you today. The first is some analysis looking at ideological segregation on Facebook and we look at ideological segregation in two ways. The first is looking at the measure of segregation or how ideologically segregated the audience is with a maximum score here meaning that conservatives are only looking at conservatives news and liberals are only looking at liberal news to the other end of the extreme which would be that liberals and conservatives are both looking at the same sorts of outlets and URLs and then favorability which is looking at the political composition of who is looking at political news URLs on the platform. Zero being completely balanced negative being liberal and positive being conservative. And in this paper what we were able to look at is all political news URLs that circulated on the Facebook platform and were shared by 100 or more shared 100 or more times on the platform. So we'll tell you more about the results of that a little bit here. We also have an observational study where we were looking at like-minded exposure and here we're looking at how often people are encountering like-minded sources and I really want to emphasize the word sources here because we're not just looking at content so we're not looking at content that may express the same ideological view that one holds. We're looking at friends, pages and groups that share the ideological affiliation of the individual user and so this doesn't only include political content it includes any content that was posted by a like-minded user, a like-minded page or a like-minded group and the reason we did this was theoretical because it turns out that even non-political things can send very political cues. When you say I'm going in my hybrid car to shop at Whole Foods you may not be intending to indicate your partisanship but you may be sending that sort of a signal. So in this study what we're looking at is how often are people exposed to ideologically like-minded sources? The experimental treatments that we'll share today are threefold and this was a really exciting opportunity to test some of these proposals that people have made about how the platforms might be changed and their really theoretical ideas undergirding what exactly is it that could be done on these platforms and what effect would it have? So the first is altering the algorithm and so in this study if we replace the algorithmic feed the ranking done by the internal ranking algorithm in use of the time with a reverse chronological feed where people then just saw the most recent content that was posted first and could scroll down and see things in the order in which they were posted based on the time ordering. The second one was to think about virality and so first altering the algorithm second thinking about virality and here is where we blocked reshared content so when people's feed we altered the algorithm so content that had been reshared wasn't included was demoted severely unblocked in most cases in the feed and then the third and final one is where we were thinking about the idea of filter bubbles and echo chambers and so here what we did is we experimentally reduced content from politically like minded sources in people's feeds. So these are three alterations that we made to the content that people were seeing in their feeds and we were able to evaluate then what the consequences were. In order to recruit people to participate in the experiment we recruited people that were based in the US that were age 18 and above and monthly active users on the Facebook and Instagram platforms. We recruited people in both English and Spanish and participants gave their informed consent to participate in the experiment so they were notified that it's possible that enrolling a part of this that their feed would be altered in some way although we did not specify precisely the way in which anyone's feed would be altered. They participated in the study for a period of three months and approximately we have around 6,000 people per treatment around 14,000 under control. The data that we collected first our primary data source was a panel survey that was conducted in between August and March surrounding the 2020 election and this was the line partnership with NRC. In addition we appended all of the survey data with a log data that individuals consented to include from both their Facebook or from their Facebook and their or their Instagram experience. And so this included things like how much time they were spending on the platform or what they were engaging with. And then we had aggregated Facebook and Instagram log data. This was for the observational studies so that we could understand what was happening across all US users on the platform who are age 18 and above. We also have data on passive browser and app usage tracking. So for a subset of the participants they agreed to allow us to track what they were doing on their mobile devices and on their browsers so that we could then evaluate what was happening outside of the meta platforms that we were examining and look at how that changed as a consequence of the changes that we made to the algorithm. We also have data on public records including voter files and FPC contributions. We won't be talking about that today. We also have Twitter usage data for people that agree to share their Twitter handle with us and then we scrape data to associate with that. Okay, getting into the results I'm going to start sharing a little bit with what we learned about how people's on platform experiences were altered and I'm going to focus predominantly here on the experiments that we did. So the first experiment that we did that I'm going to share some results for is when we changed what was happening with reshares. So this is when we're blocking reshares. This is when we're thinking about what is it if we change the nature of our reality on the platform. And when we do this you can just see by taking a quick glance through here it's not as though what they saw in the feed changed dramatically if you compare the yellow bars to the blue bars. So it's not like they all love political content before and no one's off political content afterward. But there were really important shifts that if you look at the relative difference in terms of percentage it was pretty noteworthy how blocking reshares changed what people experienced on the platform. So when reshares content is blocked on Facebook users see less political content. They also see less content from cross cutting and like minded sources. So they're less likely to see content that was shared by cross cutting sources and like minded sources. And in fact see slightly more content from either moderate or mixed sources. And what we mean by that is either people that are going to those sources like they're part of a certain group or they like a certain page that they don't have strong partisan identities or it means that the people that are going there are both from the left and the right resulting in an average being right around the middle. So that sort of content is elevated when we block reshares. When we block reshares people see less content from untrustworthy sources and they see more uncivil content. So you get kind of an interesting mix here. The solution of course is okay if we just block reshares maybe we dampen down virality and then this will be something that will help people on the platform. But if we just look at the content that they see we might say okay maybe it's a good thing that they saw less content from untrustworthy sources but they also saw less political content and they saw more uncivil content. So a takeaway from this and several of the other studies is the way in which these seemingly simple changes to the algorithm all to what people see it's difficult to come down on the side because this is normatively good or this is normatively bad in terms of the consequence. Okay for the experiment in which we change the algorithm from the standard algorithmic prioritization of content to the chronological feed we were able to look at both Facebook and Instagram and this is instructive as well. So when we switch to a chronological feed users see more political content. So that changes. We're only able to look at cross cutting like minded and moderate sources on Facebook. We don't have access to that data on Instagram because the platform doesn't have an ideological classifier in the same way for Instagram. But when we look at Facebook we see less content from cross cutting and like minded sources and more content from moderate and mixed sources. We see more content from untrustworthy sources and we see on Facebook less uncivil content and contact with slow words. However on Instagram it's a bit different where we see a slight increase in uncivil content and contact with slow words. So I think the takeaway from this again is that aid it's not a normatively clean story. So we might say, oh that's great that people see more political content. Maybe they're gonna be able to think more about politics but they also see more content from untrustworthy sources. So it's a mixed bag here and the solution of oh let's just remove the algorithm. May have some things that we find normatively troubling. And the other takeaway is that these algorithms differ. Facebook and Instagram are different in terms of what they're privileging. We also see that when we alter the algorithm it changes how people allocate their time. And the place we saw this most clearly is when we switch from the algorithmic prioritization of content that Facebook uses by default to this chronological feed. People spent a lot less time on Facebook and a lot less time on Instagram when we switched their feed to chronological as opposed to ranked using the standard ranking algorithm. And then there were substitution effects over the subset of people that allowed us to look at their browsing behavior. We could see did they change their habits in terms of accessing other sites. And what we see is that on mobile those people who were switched to the chronological feed on Facebook increased their use of Instagram. And on their browser they increased their use of YouTube and Reddit. When they switched to chronological feed for Instagram on mobile they increased their use of YouTube and TikTok. So there is people alter their behavior on other sites when we change the algorithm on Facebook or on Instagram. They also change how they engage with content. And so this that I'm gonna show you here is from the like-minded study. When like-minded content is denoted people engage less with content from like-minded sources and from this info repeat offenders. So the first one had to make sense, right? If you're seeing less like-minded content you would anticipate that there's less engagement with like-minded content. The interesting part though is that when people see content from like-minded sources so provided that they see it which they're much less likely to do when we denote like-minded content but when they do see it they're actually more likely to engage with it. So it's almost as though in this setting where people don't have as much like-minded content and when they see it they're especially likely to say oh I'm gonna click on that, I'm gonna like that. They have an elevated probability of doing so. Which is kind of an interesting phenomenon if we think that algorithms are learning from people's behavior seeing less like-minded content seems to motivate them to engage with it more. Okay that's a broad brush on some of the platform experience results and I'm going to turn it over to Josh now to share something about the polarization results. Great. Thanks Tanya and thanks everybody so much. I want to echo Tanya and inviting us to be here today. It's a great pleasure to be able to present these results at first time which has made such an impact over the years about what the rest of us know about the internet and society and so it's been, it's a really honored to be here today. So as Tanya said I'm gonna take over and talk about polarization. And the reason I'm gonna focus on polarization is that as Tanya said, there is, as Tanya said, there is, there's a ton in these papers. Like and these are the kind of papers that have like eight page papers and then 200 page supplemental appendices. So you should really dig in and if there are particular questions you're working on like take a look at them to see there may be data that's in there and there's many more papers that are coming from the project that are gonna have in particular some of them will have a lot more observational data that we hope will be helpful for everyone to research. But when I first got asked to give a talk about what we had learned so far across the four papers, I thought I would focus on polarization because polarization is something that cuts across all of these particular papers. And so this is, I think we can, we learn a lot from looking across, more than we look at any of the individual papers from looking across the different papers. So let me remind you again, I'm gonna show you things about polarization now that come from both the observational studies as well as the experimental studies. And so let's begin with the observational studies. And just to reiterate, remind you again what Talia told you that we did in these observational studies. We have two different ways we can look at polarization in the observational studies. We can look at ideological segregation and exposure to news. So that is for news sources on the plot. There's links to news on the plot comment. They're primarily seen by Republicans, primarily seen by Democrats, or they seem to be mixed, or primarily seen by liberals, conservatives are mixed. And then to exposure from politically-liked-minded sources. So here's what we find. This chart gives a very condensed thing about what we have learned over time, we learned over time, about how much people are exposed, how much segregation there is in exposure to news on Facebook. And the key benchmark here to know is that there have been a button, and Talia told you about this statistic here zero would mean it's seen equally by liberals and conservatives. One would mean it's seen entirely by liberals or entirely by conservatives. So this is a measure of how segregated the audience is. This isn't telling you what direction it's segregated, but just how segregated it is. The key benchmark to know here is that prior to this particular, prior to us running this study on Facebook, sorry, I'm just gonna set the timer so I know what I'm talking about. Prior to us running this particular study on Facebook, people had done some studies of this on browsers. So looking at what people left to their own devices, what kind of segregation did that lead to? And the benchmark for the browsers was about 0.10. So there was some segregation in people's news consumption habits. But when we look at the audiences on Facebook and Instagram, it's much higher than that. So here, the yellow line here is looking at the domain level, and the blue line here is looking at the individual story level. So the takeaway from this is that there are pretty high levels of segregation in regard to news exposure on Facebook. Oops, sorry. Moreover, we have the ability to look at this segregation at different levels. And you can think about this. So if you think about people's exposure to news on the Facebook platform, there's kind of a number of different ways that you could think about the level of segregation. The first is what we call the potential audience. So the potential audience for the news is everyone for whom that news could have appeared in their feed. And so where would that be? Well, so if I'm on Facebook, I can get things into my feed from what my friends post, what pages I follow post, and what groups I follow post. So we can think about this as kind of like a decision of the individual. The individual decides who they're gonna follow, and that creates a level of segregation. And that's our lowest level of segregation when we look at domains, and it's about the lowest level of segregation when we look at the URLs. Then we think of the next step, which is what am I actually exposed to? That's the algorithm. That is clearly algorithmic decision because you have all this potential content, but at the next step, it's well, what is the platform choose to show you? And we see that at the domain level that goes up, there's higher levels of segregation once the algorithm gets involved. Interestingly, at the URL level, there's actually no difference there. It's about basically the same. And then there's engagement. Well, this is clearly the human part of the behavior. It shows up on my screen, what am I gonna engage with? So what am I gonna click on? What am I gonna like? What am I gonna do something? And in both of these cases, there's another big jump when we look at engagement. So the highest levels of segregation that we find are in terms of what people are choosing to engage with when they get there. So in terms of who likes the story, who reacts to a story, that's even more segregated than who's shown a story. And that's across the URLs are all the domains. So what we see here is that when we think about this question of segregation, it is a complex interaction of algorithmic components from the platform, right? Which is definitely happening when we go from here to here. Human components, which is definitely happening when we go from here. And then there's this first step, which I described as, well, that's kind of a human component in terms of who you choose to make your networks with. But of course, we know these platforms also suggest who should be in your networks. So this first level is actually a component of both algorithms and human choice. So this segregation is to take what's segregation there, second, it comes from this kind of complex intersection of algorithmic effects and human choice. The next big thing we learned from this is that it's actually asymmetrical. So it turns out there's lots of segregation out there. And there's segregation on the left. There's plenty of stuff that's seen primarily by the left and there's stuff that's primarily seen by the right. But it's not symmetrical. It actually turns out, this is especially clear when you look at the URLs, that there is just a corner of the Facebook ecosystem where things are seen only, where news is seen entirely or almost entirely by people on the right. And there's something like that on the left, but it's much, much smaller. So you see the sort of when we get into like 95% of content only coming from the left is almost dropped off to a small part. On the right, this goes up much more tired, which has some overlaps with work that's been done at Berkman Klein in the past looking at the television ecosystem. All right, but we also have really interesting data from the like-minded sources paper at the object. Yeah. Just to clarify, how was the news content defined? How is news content defined in terms of being, whether or not the news content is on the left or the right? Okay, so this, oh, how was you? Also, like, is this news or not? What is news? Yeah, so we had a, there's a classifier for what was civic news. It was like a classically, you know, like training machine learning algorithm. This is all about audience determination. So everything I'm told, thanks for clarifying this. Everything I'm telling you here in these, in this study, in the Gonzalez de Bayon study, is all about audience determination. So this is, we have estimates of people's, people whose ideology we're able to estimate on the platform. We aggregate up to all the people who saw, interacted with, were engaged to these kinds of things. All right. So now the second place that we have observational data has to do with the like-minded sources paper. This is the nine-handed all paper. And in the like-minded sources paper, we're able to look at how much content in people's feeds comes from people who, from sources that are politically like-minded. So if I'm conservative, how much am I seeing from conservative sources? Cross-cutting, if I'm conservative, how much am I seeing from liberal sources? Or moderate, mixed, if I'm, you know, how much am I seeing from mixed sources, regardless of whether I'm conservative or liberal? All right. So that's the set up here. These are cumulative distribution functions, which are kind of interesting. I'm going to kind of give you on the slide the sort of takeaway from this, which is that, and here's the sort of main headline feature here, which is that for the me and Facebook user, right, like, so that's the person who's, you know, right here on this study on the 50th percentile, for our media and Facebook user, they're seeing a majority of their content from politically like-minded sources. So just over 50% of the content. That's in contrast to the media and Facebook user is seeing about 15% of the content that they see on Facebook from cross-cutting sources. But it's important to remember, so Talia was talking about earlier about how we made this decision to look at all content and breaking it down based on the political congruence with the source of the content. When we actually run classifiers over how much of this is actually political information or political news, it's actually a small portion of it, slightly less than 7% in each cases. So most of what you're seeing isn't political content or political news, but it's still coming from politically like-minded sources. So that gets into, if you read Jamie Saddle's book, Freight Fronemies, like this whole concept of like, how much of the, I'm driving my electric vehicle to the food market, you know, how many other subtle cues are we getting in there? What is interesting though, is that extreme echo chambers don't seem that prevalent. So only about a fifth of the sample is getting 75% or more from politically like-minded sources. So four fifths of people on Facebook are seeing at least a quarter of their content are coming from non-politically like-minded sources. But that being said, getting a lot of content from cross-cutting political sources is also pretty rare. So only about a third of people get at least 25% of their content from cross-cutting sources. And as I said, the median person only gets 15%. All right, so when we look at the observational evidence on polarization, Facebook definitely shows signs of echo chambers and political segregation. And I've been involved with a bunch of empirical papers that have pushed back on the overwhelming kind of echo chamber narrative that's out there that's shown that these social media platforms are actually a bit less of an echo chamber than people originally thought. I will say me personally working on this project based on this observational data, I updated my prior of it in terms of how ideologically segregated I think Facebook is. A lot of news is primarily seen by those on the left or the right, but not both. And this is especially so for news seen by the right. New segregation is higher on Facebook than in previous studies of web browsing. And it's much higher from groups and pages. I didn't mean to show those results those are in the paper here. Similarly, people are exposed to much more content from politically like-minded sources. And there's this complex relationship that seems to be driving that between algorithms and human choice. All right, now what about the experiments? Here's the big punch line, right? So we had these three experiments getting at these three big fundamental concepts. I don't have to tell you this at first, I'm fine. In the study of internet, morality, engagement maximizing algorithms and echo chambers, right? Like these are big things that we look at and think about having, we as scientists are really interested in the causal impact. We think these are fundamental features of social media and what's different about social media, the ability of information to go viral, people being able to self-select into echo chambers and these engagement-activizing algorithms. And we were able to really precisely manipulate features of the platform that are tied to each of those things. Holly went through all of these with reshares, replacing the algorithm with chronological feed and depressing the amount of content that people get from like-minded sources. And the findings were contrary to what we expected to find. There was no impact of any of these experiences on affective polarization. Those of you who are like political scientists use this term for how much you dislike the other party or ideological polarization, exactly how polarized you are on issues. In neither case do we find this. I'll just roll you through where we can find these things. But like this is the algorithm to reverse chronological feed. And remember, this was being proposed as a policy adaptation, right? To deal with polarization. No impact on Facebook or Instagram. These are really big experiments. We had 23,000 people in it. We were precisely powered to be able to detect nulls of less than 0.03 standard deviation. So this is much higher than any other story. So when we say these were no effects, these are really precisely powered to be able to reject the fact that there was an effect here. This is what we ended up finding with the reshares when we look at affective polarization and issue polarization. Again, no effect. This is what we found in the like-minded paper. Again, affective polarization or having extreme ideological views. No impact on it. Okay. So what can and can't we conclude from these studies? All right. So there's a ton of caveats I want to give here, right? Like we did these studies for only, we can say only three months. Now as a political scientist, like most of what we know, a lot of what we know about political behavior and the causal impacts of political behavior was based on getting University of Michigan undergraduates in a lab for, you know, an hour and a half. So three months is long or it's short, depending on your interest. We thought three months was a really long time. But of course, you know, maybe this is a big thing. If it's, maybe we find different effects if we're taking people off, you know, changing these things for two years. We only did it in one country and the United States has a weird political system compared to the rest of the world. So we don't know how this would play out in multi-party systems or single-party authoritarian regimes. This was during a period of heavy political exposure, right? And so, and there was a reason it was during a period of heavy political exposure. The entire purpose of this study was that we walked out of 2016 when the country kind of collectively lost its mind about the impact of social media on politics with so many unanswered questions. And part of our motivation for doing this study was we wanted to make sure that policymakers, scholars, the press, the community, the public would have more information about what actually happened on social media in the context of 2020 than we did in 2016. But this means that these results I'm showing you, this is a time when people are getting political information on TV. They're getting it on the radio. They're getting it from their neighbors. They're much more likely to be talking about politics in the hallway than they are other periods of time. So we can ask the question of, well, what if we did the same study not in the middle of a campaign season? Why did it have a difference? You know, when we think about these kinds of experiments, we have to think, you know, if I was in the treatment group, I wouldn't have seen reshared content. But Talia might be my friend in real life and she might still be seeing reshared content. And so, we didn't shut off reshared content for everybody, right? And there were ethical reasons why you don't change, you know, for 99% of the platform and legal 1% holdout, which is a different way you could have designed these kinds of studies, but maybe that would be different if we did this. Again, we privileged scientific inquiry in these studies. That was the point of this. And we were, this was an opportunity we had never before had as external scholars to look at particular aspects of the platform experience because we could change those particular aspects of the platform experience. So we did one at a time in each study because we wanted to get, we wanted to look at morality, we wanted to look at echo chambers. But maybe we would have seen something different if we changed all of them as opposed to only changing one of them. And then, if you guys can make this go away, my last point, it was only one platform, right? And people live on lots of platforms, right? So, and even the studies that we did on Facebook and Instagram, those were separate studies. We, again, to be precise about what we were studying, we didn't enroll people in both the Facebook and Instagram study. But it's possible if you change the way people were exposed to the content on Facebook and Instagram and TikTok and Twitter, you might see something different. All right, so what can we conclude? We can conclude that algorithmic changes affect platform experiences. That's what Stalia has talked to you, talked to you a ton about this. We can conclude that Facebook has a good deal of ideological segregation. As I said a moment ago, a lot of news consumed by liberals are conservatives and this is driven more by pages in group than users and people are getting much more content from ideologically like-minded sources on these platforms than they are from cross-counting sources. This, to me, all seems incontroversial. What we also can conclude is that we have not found simple answers for complex problems. We would love to have been able to come to you here today at Harvard Law School and say, look, get them to turn off reshares before every election and it will make people hate each other less. Like, that would have been great. It would have been a really good outcome. We would have had a clear policy of sorts of shit. But what we found is, and we think we went after here, like the big things that people are speculating, writing about in the literature, that people are speculating about in the press, that policy makers talk about, that people testify before Congress about, morality, intention-adjusting algorithms, you know, echo chambers, right? We went after the big ones and none of them seemed to give us confidence that if we just changed this for a couple months before an election, it would have a different impact. And I think for my case, I come from a background and I'm a political scientist who's been studying political behavior for my whole career, which started like over there. And I think some of this part to me is like there's a little bit of like revenge of political behaviorist here because political behaviors will tell you it's really hard to shift things in great attitudes like affective polarization, right? And there's been a lot of stuff that's come out of the internet research thing that says, man, this new thing in the internet is having this huge impact on things and we should just change it and things would be fixed, right? Like polarization is obviously coming from a lot of different angles in society. And again, it would have been great if we could come to you here with a solution and there's still projects ongoing and maybe some of the projects ongoing will allow us to do this. But it may just be a reflection of the fact that like there are big societal factors that have gotten us to this point in this country and I can speculate on what those are from my political science hat that have led to these high levels of polarization that we see in the United States, that we see in Europe, that we see in India, that we see in lots of places there are big global phenomenon collecting this and simply changing one aspect of people's experience on one platform is not gonna be enough to sort of deal with these kind of problems. All right, what can we not conclude? And this I wanna be really, really, but really with clear about to end on this. This is the question that everyone wants answered is is it social media's fault that we're living in a polarized world? That's a different question from if we change aspects of social media, can we make things better? We can't answer that question. Nothing in these studies should be interpreted to say if there hadn't been advances in computing power 15, 20 years ago that allowed these social media platforms to come about, if they hadn't, you know, if Mark Zuckerberg hadn't, you know, met somebody in the dorms over there and this that and the other thing, we still don't know the answer to those questions. Like if the world had done this differently, it's the wrong counterfactual. We can't rerun the last 15 years and any study we run now will be run against the background of a world that has had social media for 15 years. By the way, I think this is a very important point about AI as well. We start to think about the impact of AI in society and we start measuring these things. So consequently, we cannot conclude from these experience these experiments that social media has or has not had an impact on the growth and political polarization that we have seen globally and that we've seen in this country over the last 10 to 15 years and to claim on the basis of these experiments that that's the case is claiming too much from the basis of these experiments. And unfortunately though, the null results don't allow us to pinpoint a particular feature of social media as a potential candidate for exacerbating polarization independent of the larger information environment. So thank you so much for your time. We're super excited to answer any questions and discuss and I hope this has all been informative and thanks for all you guys do here at Berkeley Mines. So thanks so much. Awesome. I'm gonna selfishly offer the first question. One of the observations that the project rapporteur noted in his piece of science is that from his perspective the Meta's data architecture is so massive and complex and that as an outsider coming in it can be hard to know just the universe of observations and things you could get your hands on. So I guess it would be awesome if you could speak to that and have that shape to the questions you were trying to tackle the designs that you were coming up with. And then also taking a step back, looking at things like pad or other transparency proposals that are out there. Do you think those have enough teeth to ameliorate that sort of concern and coming out of the project what sort of provisions are really necessary to make sure that independent outside researchers are able to tackle the questions in an informed way? You wanna take the first question? Sure, have to. It absolutely is the case that being an outsider coming in to Meta, there is all sorts of things about the infrastructure of the data and what's stored and what isn't, what the retention policies are and how things are operationalized that you don't know. And so there most certainly was a learning curve and a lot of back and forth where we would say we're thinking about this and they would say, okay, well we have it in this way and okay, that's. So I think that that's an important thing to take into account is these are infrastructures that have built up over decade, right? And data infrastructures that have been developed and the way in which the code is stored and how the tables are populated and lots of bureaucracy in terms of what is, what is captured and what is not captured, what is retained and what isn't. So I think that that's a really valuable lesson actually for anyone that's thinking about doing this sort of platform research is how complex the data structures are. I mean, just imagine how many people are in there and how many things can be recorded in any given minute. And maybe I'll just say very briefly about the second part of your question and then pass to you, Josh, is I think that this makes clear that we need to have some infrastructure in place to make it possible for those people who are outside of companies to evaluate what's happening on them. But I think it also makes clear that you have to have some level of expertise on what's happening on the platform in order to do that because there are all sorts of parts to the infrastructure that you wouldn't know if you didn't have some of that background. Yeah, I just want to add on that second point. This is something that I feel passionate about and I've written about a lot. I think that, I mean, just pat it unless T, pat it hasn't been passed yet, right? I feel like, let's start with pat it, right? Like, you get that pass. But DSA is super exciting. The fact that these are kind of models, you know, we're really hopeful that maybe the platforms will respond to DSA by saying, well, we have to make this data available in Europe. We're going to build APIs that make it available more generally. It is absolutely clear that we live in a world right now where if you want to understand social media's impact on politics or on society or any of these things, that no matter where we go, we're kind of at the whim of the platform, right? Like, and Nate personally and I, in the conclusion of our book, Social Media and Democracy in the State of the Field, like, in our concluding chapter, we wrote like, look, you basically have four options as a researcher, right? Like, one is you work around the platforms and you don't collaborate with them and you try to get the data any way you can and that has pros and cons. And we go through, you know, well, I'm blessed that I haven't talked to lots of them. One is you collaborate with the platforms and you can. That has pros and definitely has cons, right? Like, one is you lobby, you know, not lobby, you speak to, you inform public policy about the importance of things like ADA and the importance of having data access, you know, for researchers. And the fourth is that you just give up and say we're not going to try to go with these questions. And if we think the questions are so important that we don't do that, we're, you know, we're left with the other three. I got to say when we wrote the book, the one that looked like the long shot was like getting governments to get involved in regulation and making data available. The, I've been around that has really improved in the last few years. And, you know, and I know thanks to work that people are doing here at Garmin Klein and HLS, but like that's gotten a lot better, I think. But, but to your teeth question, right? Like, I think one of the things that have come out of this project is that it's possible. Like you may just agree with particular ways of ways that we did things. There might be, we have ways that if we were doing this over from scratch that we think we could improve this. But the big picture is it is possible to do a thing like this where you are able to run causal studies on super important issues, right? That can only be done by changing things about the platform experience itself to gather observational data to be able to see at the scale of the platform what is happening and to get this data out there. And, you know, my thought is, is that I hope is, is that this project will have two impacts, right? One is the direct impact. Everything that's in the papers we now know, we already know way more about 2020 than we did know about 2016. And the fact that this data is available for replication when people get in there, people are going to dive in and get more stuff as people look at the appendix. We have more papers coming from this project. That's the direct impact. The indirect impact is the model now exists. And so your question doesn't have enough teeth. I've heard people who are saying, who are, you know, super enthusiastic about DSA in Europe, responding to these studies when they came out and saying, wow, even with DSA, there's nothing in DSA to compel the platforms to allow people to run these types of experiments. And I think the model to think about is stress testing banks. Like in this country, we have no, I mean, I'm in Harvard Law School, so I'm there, people who know this only time more than I do. But we have setups where people are, you know, where it's said, this is so important. The bank is so important for society that they have to submit themselves to these kinds of tests on a regular basis. So we know what the impact of X and Y is on the banking system. Why are we not saying that, you know, about the information environment system? And so my hope is that the secondary impact of this study is that it's now a model. People can look at this, and if you're sitting in Indonesia and you want to know what the impact of Facebook is on Indonesian elections, maybe why are you sitting there and saying, hey, we should get to do this too? Like, why is this only being done in the US? So many viewers and listeners online trickling in with questions. I'm gonna try to summarize the question of Renny and Jonathan, which is somewhat connected and has to do with this issue of pre-hardened politics arriving at polarization. So Renny is asking, are you saying that rather than alcohol manipulation has negligible effect of impact, it's possible for people arrived in the political season with pre-hardened positions and therefore are less likely to be affected by manipulation or variables and those harder positions may have been facilitated and exerted by the social. And then Jonathan's asking the polarization, specific questions about the polarization study. One is in finding limited impact on polarization, how did you control for the amount of prior exposure to polarized content which might have caused someone to harden their beliefs already? And two, in saying only 20% of people got 75% plus of content from like-minded sources, did you look at people's propensity to vote for actual voting? Because only a subset of people vote even a small percentage of echo chamber, people could have been this proportional electoral effect. So let me try to answer those really quickly. So the first question, yeah, that's the answer is yes. Absolutely, thanks for the questions. That's what I was trying to say at the end. What we have learned is that changing these features of the platform experience for three months around the 2020 elections did not have a noticeable impact on polarization, bull stock. Could that be because people arrived at this point with very hard and polarized attitudes? Yeah, everything about me is a political behavior person thinks, yeah, polarization is pretty, these are pretty hard and attitudes most of the time. And they're hard to change anytime. Could that be because of the last 15 years of being exposed to social media? Sure, but I don't know, it could also be about the Great Recession and the aftermath of the Great Recession. It could be about political elites in this country and the type of rhetoric they're using. Like there's a lot of things that could be causing it. I don't, from this study, I don't know the answer to that. On the, how do we explore prior exposure to content? So there's two answers to that. One is, that's the beauty of experiments. When you randomly assign people to treatment and control, as long as your randomization process works, you should have that in both the treatment and control groups. And so the effect that you're trying to measure is independent of it. We also, however, pre-register a large number of heterogeneous treatment effects, which is what this is a class of questions for. And in these, that's way too much for it because there's heterogeneous treatment effects across all the papers and other things. It's way too much for us to get into in this kind of talk. I encourage people to see the appendices, to go look at the papers, if there are specific heterogeneous treatment effects you're interested in. But, the TLDR on this one is, It really quickly, because we did look at people's prior exposure to like-minded content, for example, as well as heterogeneous treatment effects and TLDR, and TLDR. And TLDR is that we didn't really find any heterogeneous treatment effects. Like we looked at tons of heterogeneous treatment effects and we found almost nothing. And so you can go back and look in and see these particular ones. In this particular case, I think we definitely didn't find anything because it would have been, it would have been something that was highlighted in the paper. But all that data is there. And if there are heterogeneous treatment effects that you didn't test for, you can go and you can go get the data yourself and test for these other types of heterogeneous treatment effects you want to test for, in the case. And just adding that we did look at voting in some of these and there were all several effects. So that's included. Do you guys just want to bunch your questions together? Sure. So thank you for being here. I know you talked about the system designer, just the platform itself. But what do you think, if anything, this study or these series of studies show and for content creators, in regards to trying to make engaging content? It seems to me that there are two Facebook related causal studies that precede you guys, one of which is Black Hat with Cambridge Analytica. And one of which is 15 years of AB studies for stickiness in algorithm designs, internal Facebook slash methods. You guys have a white hat, you know, gay. But did you get access to and compare the data on those two things? Did Facebook actually give you access to their AB investigations as part of the study? My question is at the very beginning, you said you were trying to study four things. You spent the rest of the time on polarization, so I don't know what the other three were. Did you get any non-null results on any of the other three things that you were studying? Take those, okay. Don't we just start with all three and then you can add on, okay. So thank you for those great questions. The first in terms of information for content creators, if you look inside of the studies when we remove reshares, for example, or when we switch to the chronological algorithm, you can see what sort of content is being algorithmically prioritized. So if you compare what sort of content is in the feed with chronological to what sort of content is in the feed with the engagement based ranking, if you compare those to each other, that's essentially telling you what the algorithm is prioritizing or deprioritizing. So if you're a content creator and you say, okay, I want my things to be prioritized algorithmically, then look at the sort of things that are being prioritized algorithmically. So that would be one tidbit there. We did not have internal data on Cambridge Analytica or other AB studies that had been done in the past. Of course, you all know here that there's a trove of TINA here released, but it was a lower. So there's some information there, but we do not have access to that. And then the third question about other effects, in the reshares paper when we blocked reshared content, what we found is for those people in the sample, they were less, they had lower levels of news knowledge. So if you remember for reshares, there was less political content in people's feed when we blocked it. And for the sample itself, we see an impact in terms of news knowledge afterward. There's some indication there that it might be there for factual discernment as well, but once we adjust for the number of tests that we did, that effect is no longer significant. So there are hints of it there and both of those things point in the same direction. Then I think there's something going on there in terms of what people know. And the other two things you were saying, you got no results also. That is correct in the study. We didn't look at all outcome variables across all studies. So just with a caveat there, that there's more to be done there and it's not a perfect match, but across the board, mainly no results. Well, thank you very much. Congratulations for this study. My name is Dimitry Nasebova and I'm a professor in communication theory at the Complotencia University of Madrid. So first of all, I want to invite you, recommend you to repeat this study. As you said, maybe in multi-party countries or one-party countries. And also I would like to recommend you to do this in Spain because Spain is now a very interesting experimental field. We have a parliament with more than 10 parties. So my questions, how do you imagine would be the methodology in a multi-party, sorry, country, like the case of Spain, more than 10 parties in the parliament? I imagine that one of the difficulties has to do with ideology because it will be not the same. There will be cross-cutting topics between some of the parties. So the question is how do you imagine this study, thank you very much. Yeah, without going into a massive divergence in how we measure parties in multi-party systems which I would be happy to talk about for a long time. I would recommend that you take a look at the comparative study of electoral systems which is a cross-national collaborative election study project that piggybacks that puts common content and piggybacks on the backup of individual national election studies. I mean, I think things like affective polarization to the extent that we measure those with feeling parameters, you can measure that with lots of different parties. The question of how you measure ideological polarization in multi-party systems is much more complicated how you measure it at the systemic level. But you can look at things like whether people who are exposed to different types of feed treatments or something like that have more extreme positions on issues, that would be one way of doing it. It doesn't translate one-to-one to what that means to the overall system level of polarization, but it could get you an insight into that. One more listener viewer tuning online with the question, and this time it's Matt Mahon. There was one slide that you showed diversity of content on the left and the right, and the far side of the right soup was way more drastically than the left. Does this mean that the extreme right opinions or content are much further from zero than far left opinions and content? No. I was going to say, no, no, no. What that, just to be really clear about that, and I think the important thing to wrap your head around in the Gonzalez and Bion paper is that the unit of analysis is the article, the news article. So the unit of analysis is not individuals, individual people, and what we're looking at is the audience composition. So the maximum you can get is 100% audience composition. The composition is all liberals and no conservatives are all conservatives at no level. So what that figure showed, which is essentially a distribution of how that's distributed across the entire space going from 100% liberal and zero conservative to 100% conservative and zero liberal, it's spread out over the space. So there's plenty of content that is on both sides, but that peak at the right side, which is not mirrored on the left side means there is this bunch of content. And again, we're saying nothing about what's in that content. This is just the audience composition. So what it means is there are a bunch of URLs on Facebook that get seen only by conservatives. And then that's a, there's a chunk of things that kind of fall into that area. We don't see that a lot. That curve could look like this, right? It kind of looks instead, it kind of looks like this, right? And so you get across the distribution. So there's some content that's seen, there's some news content that's seen by both liberals and conservatives, but it could also look like this, right? If it was really that most news was being seen, who was seeing news with Lord Fogg on all the partisanship, then you would expect it to all be bunched in the middle. And that's clearly how that we found it. Just wanted to follow up on the content creator question. And for transparency, I used to be on the city of Cuba Facebook, so we work with some folks who know very well. One, I actually do think there are some opportunities on the production side to test some things out, especially on the reshares. So for example, what if there was an interstitial that popped up when you were about to reshare content that came up in the city classifier that sort of just gave you some words to the lies? Like, hey, make sure this content is x, y, z before posting, et cetera. Like, I'd be really curious about the consumption effects then of content that's been produced in a more thoughtful way in comparing those different AD groups. Yeah, love that comment. And I don't think anything we're suggesting precludes any of that and more innovation in terms of how these platforms are structured. So another hat I wear is working with new public. And they're trying to think through how is it that you would design social spaces in more public-friendly ways that allow people to work productively together, for example. So I don't think any of this precludes new innovation in terms of thinking through how platforms should be designed. But I think that it does say that the effects of some of these things that have been proposed as ways to reduce affective polarization, for example, they don't seem to have that effect at least under the confines of the study that we did here. And I would just add to this, one, if you look at the research on misinformation, there's been a ton of work on various interventions, including there's a whole subfield that's called nudges. And then there's also a friction subfield, which interestingly enough, if you look at Molly Roberts' work on censorship in China, like that's basically what the Chinese government does is try to use his friction for a ton of censorship. So I think there's lots of good examples to look at in that particular regard, people that focus on misinformation, but you could focus it on other things and you could focus it on higher quality content and those kind of things. And just one quick caveat on Talia's point, answer to you, like, if I was a content creator right now, like I think I'd be interested in looking at what was more likely to be reshared and what was prioritized by the algorithm for this study, but I would do it with a caveat that this is data from 2020 and these things are very live, living, breathing organisms, well, that's a bad, isn't it? But they're changing all the time. They're complex, like this was going back to the first question, there's complex interactions of lots of different features and I would not for any way shape assume that if you could tease out from something we found in 2020 that it would still be in place today. It might be, I just would go into it with like, you know, cautious eyes, which again, going back to the first question, I think points to the importance of hopefully this kind of study is not a one-shot deal that it becomes something that's repeated over time and repeated in different contexts in multi-party systems and we get, you know, we're able to design more studies that are able to be implemented in other ways. Awesome, we're a little past one, but if you guys are willing to take the last couple of questions in the room, awesome. Both of you have been mentioning really about left and right, liberal and conservative. How about so-called moderate, like, I myself consider to be a moderate and I just, at the time of election, I would depend upon what the candidate stands for. It's not the left and the right and even it happened in Massachusetts. You know, Massachusetts, we had Republican governors and even it voted for President Reagan in 1984. So, and the conservative, there can be quite religious people who are good-minded, respected, respectful to others. Then there may be people who are hating others. Still, still, they're all clubbed and medged as conservatives. And liberals can be with all good ethics and then with all ultra, so-called, I mean in my opinion, are unwanted ethical aspects. So, merging everyone as liberal or conservative. So, to me, it is somewhat not very informative. I mean, just for my understanding. And again, when you ask your users, how far they were very true to their conscience? I mean, displaying their privacy. They might have altered their behavior just for your study. Is there a possibility? Someone altering their behavior just for your study? Great questions. Happy to make you start with the first part of it and jump into the second part as well. So, the first part of the question is about what about moderates? And across these studies, we do look at moderates as well. So, in several of the studies, we look at those people that have views, you know, 0.65 and above, 0.35 and below. And consider those in the middle to be moderates. And in our life-longed study, we did do it so that we split people right at 0.5, 0.5. But we did do some robustness checks to ensure that the results would hold if we also allowed for some moderate in the middle. So, I think it's a really good point. I think there is a moderate component to this. And as we saw in some of the work where we're looking at what sort of sources do people see in their feed, it did change the percentage of sources that people were seeing from moderate or mixed ideological sources. So, I think you're definitely onto something. And also, just to acknowledge your broader point, I think, which is, are there more axes than the left-right axes even with the middle? Yes, absolutely there are. And we just focused on that one in this study. But there are more complex ways to look at this beyond thinking about that. And then I think the second part of your question is asking whether people would have altered their behavior as a consequence of being part of this study. You know, for those people that were informed about the possibility of the study, might they have decided to change things? You know, I think there's a possibility, I can't quantify it for you necessarily, but I will say this took place over a three-month time period. We can look at how people behaved in a control group versus a treatment group where we were altering things and we do see differences in their behavior. So, maybe they both altered in some way, but we still see differences between treatment and control. And we would expect that to be randomized across the two. And you know, these were subtle changes, right? I think it would be hard for a person to all of a sudden like, oh, they removed reshares. I just think that would be exceptionally difficult for someone to have really thought through and have that conclusion in a way that they would either hypothesis guess or act against the hypothesis. But you may want to add more. Is there one last question? Yes, there's time. Very brief. Very briefly, the, so the only non-mull that popped off the slides and then he alluded to in the Q and A is the decrease engagement with political news under one of the conditions and then decreases in news knowledge. Was there any sense that there was a concomitant decrease in the intensity or salience of sort of politicization among those people, even if there was no movement in beliefs or affective polarization? Yeah, I mean, that's the reshare study, right? And so that's the one that did have this interesting finding where it made, you know, what we discovered from the first part that Talia was showing you is that people were getting less news and then not surprisingly, they knew less about the news, right? And this is pretty consistent across, there's been, my lab has done a couple of different Facebook deactivation studies, one in Bosnia, one in Cyprus, right? Oh, thanks, thanks. And in those papers, and then in the all cotton gas scale that did in the US, it's pretty consistent across those studies. This has nothing to do with this study, you take people off Facebook, they're a little happier, but they know a little bit less about what's going on, right? Like those are, there's a three studies that all find those kind of findings. And so, it's not, you know, so what we learned was that, what we may have learned is a little bit of the mechanism, but like we've learned that reshares play a role, which kind of makes sense, right? If you think that like, lots, there are some people who are more, you know, we all kind of know these, and it's gonna be a little anticodal, but like we know people who are, who go online and like to share news a lot, right? Like that's kind of the model of Twitter. And so if you get people away from the things that people are resharing, maybe you're getting more original content. And so, I think we've learned something about that, that part of the news ecosystem comes from sharing links as opposed to everyone posting them organically. Did we check, I mean, what we were, you know, the place we were checking the intensity was on the affective polarization. That was like, would you have lower levels of affective polarization? I mean, that was the one, you know, that was the one thing that we went in there expecting to find from everything that I've ever heard about it. So the surprising of us is these results came in too. And just adding that there is a pioneer too with respective partisan news collects that kind of relates to all of this as well, that you can find in the papers, but yeah, we didn't have a specific measure of salience in some most important problem. All right, well, thank you, Josh. Thank you, Talia. I know there's more papers coming out, so hopefully we'll have you back in a few months for an update perhaps, but thanks everyone for joining online and in person today for the awesome questions and hope to see you next week for our last installment with Ethan Zuckerman. But thanks again to Josh and Talia for today. Great, thank you so much. Thank you.