 So, we'll get started. I was sort of hoping I would be there and up in which is why this is my wallpaper, but like, I guess I couldn't make it probably for the best with all the effort chaos that's going on. So, today we're going to talk about if the news media is actually as polarized as our perceptions are or are we like being conditioned to think that it is. It's a little bit about myself. I'm a student at UMass Amherst. Social technical analysis is like a hobby of mine. I've spoken at Picon ZD, which is in South Africa, Pio, Ohio, etc. before. So, this talk is specifically about a field experiment because I guess we have to talk about a field experiment and what it taught me about news bias and polarization. So, a quick survey of the room. I know not all of you have Zoom access, so I'll probably not be able to see this. But how many of you think in the room that the news media today is like heavily polarized? You can like put your hands up so that everyone else in the room knows that you're thinking that. So, they have like a fair sense of an idea of like just how many people think that it is fairly polarized. So, this is what like certain surveys say. As you can see from 1984 to 2017, clearly we have all been through something because we seem to think that the news media is not careful to separate fact from fiction or a fact from an opinion. And thus, it's like heavily polarized. It's the same in the UK newspapers and at this particular point for like this analysis, I'm only going to like English speaking news, newspapers just because of the natural language models available to me, which are mostly in English. So, as you can see over here, this is another survey that was done in the UK. And the first thing that you'd probably notice about the survey is that it's super weird because like every newspaper, there's like a bunch of people who think it leads either left or right, which will probably make you think that people don't know what they're talking about. But that's what the survey says. This is a chart that was created by all slides, and it was created off a survey, and they basically ranked news organizations in the Anglosphere, which is like the English speaking, prominently English speaking word of sorts into super left leaning, left leaning, centrist, right leaning and like super right. And I decided that I was going to try to see if a natural language model could identify why all slides are like why there's like this perception of a particular political bias associated with a media house of sorts. So, the next question that we sort of come to is why should we even study this phenomenon? So, I'm going to answer this in like three different parts. So, a question that I have always asked myself is will the usage of natural language processing be as common as statistics one day? So, if you've gone to like the history of how science has sort of evolved, we all know that statistics was not as commonly used as it is used today. It was popularized in the 15th century, but now it's like it's a super common tool. And it's used to describe phenomena you wouldn't have thought would be described in terms of statistics, for example, biology, astronomy, etc. So, we can use statistics because mathematics is the underlying language of science. But like, what is like the language of society? The language of society is what we actually call language. And we all know that all of our laws are judiciary, our social sciences, our culture, all of these are based in certain languages. I thought that natural language processing was a really good tool because it is like this bridge between natural language and mathematics. So, it was a good tool to quantify certain social phenomena. So, maybe like in the future, natural language processing could be like the next statistics. The second point is to quantify the biases in language. We all know that languages are biased because languages are basically an encapsulation of the concepts that are common to a particular society. So, we know that we've been using terminology that we've deemed inappropriate. For example, master slaves in electronics are the usage of the word blacklist. Why we know that this is probably like a phenomena that comes from back in the time when it was okay to use words like that. It could be triggering for people who have been at the receiving end. Another point is social responsibility. Now, this is a big one because we all know and I think I would say 90% of all of us would sort of agree that social media at this point is crazy. So, social media sites across the world, they have hate speeches, they have calls for genocide, they have sexual harassment. And there's like a lot of policy scientists who look up like various biases in our justice systems usage of language. And as most of the people in the world struggle with what is globally called media bias, I believe that as mathematics and statistics become commonplace measures, so will machine learning models. So, this work is an example of an intersection of a traditionally non-scientific field with computer science and mathematics, trying to quantify measure and identify non-mathematical phenomena in the language of mathematics. This is important because it could be the basis of scientific approaches that the next generation policy makers, voters, non-profit organizations, even governments could use to make life-changing decisions for their citizens. So, how polarized is the news media really? And how does one go about quantifying it? To go ahead with this experiment, like for every experiment, you sort of have to scope what that experiment actually means, what you're going to try to do, what is like the scope of that particular experiment, what data sets you're going to use, and this part of this talk talks about like the scope of this particular experiment. So, my definition for news media for like this particular experiment was established news organizations. So, I was not including late night shows or opinion shows because though those are like sources of news for a lot of people, but they're not news shows. They're technically classified as entertainment shows. Online content on their website, some articles would include opinion pieces, so I was not including those. I was also not including social media content just because everything that news media organizations put out on social media content may not be reporting and properly news. So, the aim of this particular experiment was to study the varying levels of polarization induced in a neural network by feeding it newspaper articles with manufactured sentiments. According to the All Slides Media bias chart, for the level of paid people on various islands of the political spectrum have. So, basically, my X would be a vector and the vector would be derived from the news article itself and why would be the credence or like the faith that people sort of put into that particular news media organization. On the basis of where the people lie on the political spectrum. Some of the state of the art or related work is mentioned here. So, people have actually worked on related topics, but not on this particular specific experiment. There was an online news media bias analysis using an LDA and LP approach. Now, this is like a super super old experiment. That is when the back force approach and LP was popular. So that's what they used. But we do not know how it would perform with the models, the language models and the neural networks that we have today. There was another that was a political ideology detection using RNNs. But what this was doing was it used to go through like speeches of certain politicians and it would categorize that particular speech as being on one side of the political aisle. There was one more called identification and analysis of media bias in news articles. Now this one, I would say, eventually led to like a good data set of sorts. It was done by NewsBud, which is an aggregator and it basically presents shared and different information on topics. Another one was quantifying perceived political biases of newspapers through a document classification technique. There was document level and though I tried starting my experiment with a document level analysis, the data required for a proper document level analysis, I was not able to scrape through in that given amount of time. There was another one called media bias, the social sciences and NLP automating frame analysis to identify bias by word choice and date link. This one depended only on word choice biases and it basically would work for like a very specific set of articles in the sense that if there were like an incident that sort of took place and there was like one side who was calling like the aggressors in the incident terrorist and there were like other people on the other side who were calling them freedom fighters. So it would be like very specific and that was not something that I was looking for. Another one is detecting media bias in news articles using Gaussian bias distributions. This required heavily annotated data, which I didn't have, which is why I eventually went forward with this approach. I use the variety of data sets just to make sure. So the data sets I use toward the COVID or the COVID alien data set. This was you can use media topic based 200k news headlines from the year 2012 to 2018 by Kaggle. NELA is 2017 news data sets. All the news by Kaggle, E-M-N-L-P based a data set. I also use a fake news data set, which was compiled using news headlines from this booth. An online journal that publishes fake news or satire or jokes. I'll tell you later why I use that. So the specifications of this particular experiment were whether was it going to use document encoding or sentencing coding. So as I mentioned before, the data required to do like a document level analysis was not something that I had on hand, which is why I decided to go for a sentence encoding. I went to a couple of data sets, went to a couple of encoding techniques. And there weren't like any patterns that were sort of emerging, which is why I decided to ask myself this question. It was, is a headline a good representation of bias? And the answer surprisingly was yes. And you'll actually be like super surprised to discover this the question time. Do these next sentences show you any level of biases? These are two headlines. They're taken from actually the same news media organization, but they've been published for like two different editions of that same news media organization. So as you can see, the first one sees Bernie Sanders scores victories for years while legislative side doors. And the second one says why are legislative side doors Bernie Sanders, one modest victories. So you would think if you were the person reading the second sentence, you would think, okay, it's probably not such a big deal. He scored like a modest victory. And the first one would make you think that there has been some sort of progress happening there. And they're from the same news organizations. In fact, on an anecdotal basis, I've seen headlines that like super misleading, misleading to the extent that when you go to the article and eventually read it. You're like, this doesn't correspond to the headline at all. So I would say that headlines are a good measure of bias. They are like very useful when the news media organization wants to elicit like a certain response from you without compromising on their credibility as the news media organization. The article is super, super accurate or accurate to like a fair degree, but the headline is a little off. So there were other problems to address. Some of the problems were is the sentiment evoked from a headline enough to indicate some bias. Yes, I would say yes. Can a headline be considered a review of the state of affairs? Maybe if you would think of it that way, can a neural network be polarized if it were to learn from the perspective of a biased news consumer? I would say yes. Does the category of news, a biased consumer or a neural network trust completely affect that perception when it comes to fake news? So the last point is why I included like a fake news data set because I wanted to see if there were people on, there were people consuming news from organization on like one side of the political aisle. Would that affect like how vulnerable they were to actual fake news? So the methodology that I basically used for this experiment, the first is sentiment classification and the second one to confirm the vulnerability was binary classification in the detection of fake news. So my pipeline for the sentiment analysis look a little like this. On the top, you can see all the different data sets that I used. I fed all of the headlines from all of these data sets and news organizations to pre-trained sentiment models. So there are pre-trained natural language models based on Twitter and Yelp. So the Yelp models are based on Yelp reviews. They have Yelp ratings to like sort of base off their learning from. And for Twitter, there's a robot or model that basically does sentiment analysis. So what happens when you feed in a sentence or two of these sentiment models, you get like a vector at the end, which is an encapsulation of what the model has learned through its dusk of sentiment analysis on Yelp or Twitter data. When I had the vectors and I had a score for that particular news organization. So as you can see on the right side of my screen, you can see there's an example of a headline and a score. So the headline would be the actual headline and the score would be a score given to the news media organization based on where it lied on the political aisle. And by creating like a vector using a pre-trained sentiment model, we were adding like pre-learned sentiment context to these embeddings. Then I fed it into a variety of neural networks. I had CNNs, I had LSTMs, GRUs, etc. And for the fake news addiction, again, we have the data sets on the top. We have these sediment embedders after that and I use like a lesser number of models for this one because this was binary classification. So I use the CNN and LSTM and GRU. So these are some of the results. I'm not sure if you can see them as well as I can. As you can see, there's like hardly any data set that actually polarized a neural network that actually led to like some sort of good results. And you can see how it sort of varies according to the data set. EMNLP was like, I would say, induced like the most, most polarization, like the neural network learned something. And if you look at like the axis on this particular chart, you can see that there are certain data sets that are appended with Trump, which is basically like it was a subset of the data set, like all use Trump is like the subset of the data set all news. But it was like, I kept only the headlines that mentioned Trump specifically just to see if there was any sort of topical bias that was sort of induced. So my conclusions from like this experiment were that the only data set that polarizes a neural network, at least in this context, in this particular design and in this scope was the EMNLP. The EMNLP is a data set that is not scored on the basis of the source of the news, but rather has been hand annotated by annotators to detect the stanzas left, right or center. Now you'll be surprised to know that when the annotators detected the stanzas of the headlines in EMNLP as left, right or center, there is no correlation between their stanzas left, right and center as hand annotated and between the all sites media bias chart, which would have given me like a different score based on where it sort of lay on the political spectrum. This means that while some level of polarization exists, there is no polarization on the basis of the perceived bias as calculated by all sites in their surveys. Some news articles do take ideologically polarizing stanzas in the articles, but this is not true with respect to most of the articles. Secondly hand annotated articles are the best data set to conduct news media by studies as the neural networks seem to learn best from the features learned by these annotated data sets. For the US COVID data set, it's more polarized than the UK COVID data set or both the NALA 2017 data set and the all the news data set, a further version of the data set that included the keyword Trump polarized the neural network more than the non-filtered version. This points to the possibility that topical bias may exist in these publications and needs to be examined. So for this fake news, as I said, I conducted two experiments. The other one was the fake news vulnerability analysis that gave some interesting results as well. All the categories for all the data sets are pretty much robust and not vulnerable to fake news to a great degree. The CNN and the bioGRU models gave around 100% accuracy for all the cases. Biolistium showed some variations. Here for the UK COVID news data set, when we used Yelp and Bedding, the vulnerability decreased for centrist news agencies and opposite trend was observed for US COVID news. The centrist news agency reporting had a lower accuracy when detecting fake news. This could basically be because of the way Yelp and Twitter robot or models learned features or it could be because of the fake news that was sourced from the website this book, which was like super US centric. This is an interesting trend where the NALA 2017 data set when filtered on the keyword Trump developed some level of vulnerability towards fake news, especially when it came to like the extreme left-leaning news publications. So some of the observations SVMs actually perform better clustering with respect to the categories than neural networks. However, the maximum does not cross 67%. The most significant conclusion from this work is that though there is a positive bias when it comes to news agencies, when looked at from a neural network standpoint, it is negligible. Mainstream news agencies are not able to polarize a neural network with inherent biases in their headlines. There may be topical biases that need to be examined by using an entity linking and bias calculation approach. Most mainstream news agencies do not make the consumer vulnerable to believing fake news. This study needs to be conducted with data from popular social media news groups or popular TV shows that masquerade of news, but may technically not be so. It's safe to conclude that perceived bias that stems from social media polarization is being extended to news media when their contribution to polarization may actually be minimal. So the significance of this work is to be able to transform like a social problem into like a technical one and using neural networks and machine learning techniques to try to gain some insights. Hopefully using these techniques to find deeper trends will become mainstream and help policy makers and general citizens approach news media bias in a better light. This is about the future world that I hope to do. And we'll probably give like a talk in person maybe like next year. Thanks so much. I'm now open for like questions. Thank you. Do we have any remote questions? No. Anyone in the audience here have a question? If you do, please come up to the mic. I guess we don't. Aroma thank you very much for the talk. Thanks so much.