 Welcome everybody, and good morning, good afternoon, and good evening, and so today we have another great seminar speaker, Anaya Sen from CMU to share with us a very exciting paper, the editor and the algorithm, value of data and the externalities in online news. So we will be using the first 40 minutes for the presentation. If you have thoughts, suggestions, please feel free to use the chat window to share this. And if you have some clarified questions, please feel free to raise your hand. And we will then use the last 20 minutes for Q&A. So without further ado, Anaya, why don't you go ahead. Thank you, Feng. Thank you. Thank you for having me here. This is a joint work with your closer who's at LMU Munich. I think he's joined us on Zoom here as well. Christian Boykert, who's at ETH Zurich and in Kethulika, and I'm going to talk about the editor and the algorithm. So let me start with the big picture. You know, we all know that news plays an important role in a vibrant democracy, provides information, especially during the pandemic, we're seeing how potential misinformation, even from mainstream outlets can have drastic consequences. So for credible news outlets to exist, you know, it is super important just as a pillar of democracy. We also know as a digitization crowd, as a platform crowd, that digital transformation over the past couple of decades has severely affected news industry's financial health. And this is, you know, not only in the US, it's across the board, barring a couple of countries. More recently, though, it's been hypothesized that new technologies such as artificial intelligence, the use of artificial intelligence, and machine learning in particular within the newsroom can potentially shore up for revenues. Now, the idea is that you have a large amount of data on people's preferences. You can have algorithms which can mine these sort of data sets and identify certain references which traditional human editors might not be able to do. And I, you know, the paper is about human editors and algorithms mainly because human editors are the status quo, especially in mainstream news outlets. So the broad question that we're looking at is whether human editorial decisions can be aided by algorithmic recommendation. And we're going to try and address this question in a simple news curation setting, but it's a large-scale field experiment that we run with one of the biggest German news outlets. This issue about human editors and algorithms is, you know, is a debate which has been happening in the news industry for a few years now. For example, Facebook, you know, Facebook as a news curator first had human editors, then it fired its team of human editors, then it had algorithms choosing trending news, then they abandoned trending news, and now they've hired a bunch of humans to curate news stories. Similarly, Apple, you know, this became big news in 2018. Apple hired about 40 mainstream journalists from, you know, from the news industry in the US, and they said that, you know, these humans are going to curate the news for all their users. These are news, you know, these are sort of news aggregators in some sense, but mainstream news outlets have been have been really reluctant to sort of take this approach. It became big news last year, like exactly a year ago with the New York Times saying that for the first time they are going to allow personalization on their website. So there'll be an algorithm, which, you know, which will take into account what you sort of read before and try and recommend new things. And, you know, again, they talk about how, you know, the human editorial judgment is the most important. So more generally, when we talk about human editorial decisions and algorithms within the news industry, you know, it's not completely clear where one might outperform the other because it's, you know, it's curation inherently subjective. There's a creative production process. You have to try and understand what is quote unquote important or, you know, or what might get clicked, what might drive up your revenues. And you have to be really quick because timing is critical as, you know, in this paper by Julia Kajie, who shows that, you know, most news stories online on some of the biggest news stories get copied within the first three to five minutes. News events are happening all the time, consumers preferences might be changing. And hence, it might be the case that this is a place where human judgment might have a reasonably big role to play. So we try and dig deeper into this question to try and understand how algorithmic performance might be affected relative to human judgment and how does different types of data affect this, affect this performance because any conversation about how algorithms do or how machine learning does in general cannot, you know, cannot be divorced from how much data goes into feeding the algorithm. So we're going to look at two dimensions of data, personal versus social. So individual level versus aggregate. So, you know, let's say new strengths versus what you have clicked on. And also the stock versus the flow of the data. So you need to continuously update your, update your model or the fact that a person has clicked on enough to accumulate a stock of, you know, a stock of data. How do they interact? I frame this in terms of the news industry, but, you know, this is more broadly related to the debate about, you know, what the value of data is, what should, what should policy look like. The economist saying, you know, in 2017, if data, the new oil, Andrew Yang, who was, who was a candidate in the democratic presidential primaries this year is suggesting data dividends where tech companies get back, get back, then, you know, you pay a bit of the revenues back to the users who create that data. There are a couple of studies on the value of data, but you know, they point to, they point to mixed results. In general, you know, I think there's a consensus growing that data might not, you know, or certain dimensions of data might not be as important as they seem, but, you know, the debate is still going on. One of the issues that has come up in these sort of papers is sort of summarized by the sentence in this Amazon paper by Pat Byry and co-authors is that the effect that they identify due to access to longer histories of data can also be driven by continuous improvements in forecasting technology. And what we will hopefully be able to do in this, in this field experiment is actually identify potentially the causal impact of data histories. And finally, you know, to sort of round up the whole topic, the flip side of this is that if an algorithm keeps recommending you stuff to click on, how does that impact the user's consumption preference? And this is, again, important because there's this issue about filter bubbles, algorithmic reinforcement, especially in, you know, in polarized conditions. And that's the, that's the final question that we ask. How does algorithmic curation affect consumption diversity in this context? So if I do run out of time, just in case, you know, as an overview of the results, on average, we find that personalization in particular, so not, not social recommendations, personalizations increase clicks by 3.7% relative to the human editor, but diminishing returns kicking quite quickly. Can human curation ever perform better? Is there a role for human judgment? We do find certain instances, especially when there is limited individual data. So when the recommendations are based on aggregate news trends, we find that the editor is able to identify the average preferences better. Moreover, a human can do better is when there's lack of continuous updating of the algorithm. This is mitigated a little by the existing stock of preferences. But we find, at least in this context, that you need to keep updating your algorithm. How does personalization affect consumption patterns? We find that treated users do reduce their consumption diversity. So they, they sort of do narrow their consumption. And this is, you know, primarily sort of driven because personalization is playing a big role in increased click rates. And we find that this effect gets reinforced over time. And, and I'll, and I'll try and show you a picture about how, how things progress. If you come on to the website 10 times 50 times 100 times. So that's the broad overview of the paper. Is that I don't know if there are any questions at this stage, or I should just dive into some of the details. Okay, so it seems that we're all good. So setting is that we partner up with one of the largest German news outlets. This, this new outlet gets about 20 million unique visitors a month, that's about 100 million clicks a month. So it's quite a bit. They care about clicks based advertising revenue. So, you know, it's not, you know, editorial judgment would potentially take into account, into account, you know, how interesting or how many clicks an article might get. The homepage of the outlet is curated completely by human editors, or content is produced by humans. And a user who arrived at the homepage sees the same content across the board. So it's not, it's not personalized based on location or anything else. So what the experiment does is that each time a user comes to the landing page, he's randomly assigned to a treatment or a control group. If the user is in the control group, then she's going to see all the articles on the homepage as curated by the human editor. So this is the status code. If the user is in the treatment condition, then it will see one slot on the homepage, which is customized based on her reading preferences. And the only thing that we could negotiate with this news outlet, along with the data science team, was slot number four on the page. But remember that, you know, this is a large news outlet with, again, over 100 million clicks a month. So slot number four also gives us, you know, a large amount of traffic to deal with to precisely estimate our parameters. The experiment ran for five months. So it's a reasonably long period of time, which actually relative to other papers allows us to trace out some of the dynamics because often these experiments last for a week, two weeks, three weeks at the most. Just to make sure that we're all on the same page in terms of the experimental setting. So again, as I said, it's a simple curation exercise. What the control condition would be is just the human editor choosing all the articles on the homepage. And they are on average 80 articles on the homepage at any particular point in time. What the algorithm is going to do is that if a user comes on to the website and let's say the user has some prior history, the algorithm identifies what potential article this particular user might want to click on and it might identify, let's say, article six. In the standard status quo setting, the algorithm might identify that article six is more likely to be clicked on than, say, article four. And so it's going to bump up article six and push everything down. So the ranking ordinarily remains the same, but it's just one switch that happens. So again, it's a simple setting, all articles written by humans. In some sense, the initial curation is done by humans as well, but then we're trying to see whether this can be augmented by algorithms and how that varies based on the data that they have. I've been talking a lot about the algorithm. Ananya, could I just ask a quick question there? Yes. Which is, did it ever happen that the algorithm could put onto the homepage an article which would not otherwise have been on the homepage at all? No. So the algorithm has quite a broad choice set at any point in time, because it was going from one to 80. And we can actually see what rank it was pulled up from and it never goes beyond 81 or 82, which is generally the maximum number of articles. I have a question, too, please. In your introduction, you made very broad statement about what personalization would do in general, but your experiment is on a very specific form of personalization. Will you try to explain to us where you think what's general and not generally the lessons you are getting? Right. So I think it's, with this type of a question, there are different dimensions of where it might be generalizable or not. I was getting to one of them, which is the algorithm used. So it's the algorithm and how the algorithm used interacts with the data. I mean, I don't want to walk back. I can have a general conversation about how I see the literature on the value of data and the study. But I think this is going to be one piece of the puzzle. We're not necessarily saying that this is how let's say Google or Facebook would necessarily value their data. But as of now, we don't have that many estimates. And so this is just trying to take a first step. And maybe towards the end, I'm going to talk about the contribution of the paper and how it fits in with different strands of the literature. And I also link it to the computer science literature. And maybe then we can have a broader conversation. But again, as I was saying that, I've been talking about the algorithm quite a bit. And this is one area where there are many different types of algorithms. The data science team, in this particular case, used a highly cited algorithm, which was developed by Google engineers. So that was developed in 2010. That was used as the baseline. And then they put some bells and whistles on it, which we are not completely aware of. We know that we know of the 2010 paper. It's been cited heavily. And the model is reasonably flexible in that it tries to predict an individual's preference for different content categories. But these content categories are extremely fine-grained. And on any given day, there are over 200 content categories that the model chooses from. As I said before, the algorithm works such that it uses a combination of social data, which is collaborative filtering and personal data, which is information filtering. This is computer science recommended system talk. Basically, when there is limited data on an individual, then the algorithm is going to overwhelmingly use social data, that is, other people's current reading behavior. As the algorithm gets more and more information on that particular individual, personal data starts dominating in terms of what recommendations are made by the algorithm. The aim of the machine learning model is to maximize clicks by the user who visits at any given point in time. Of course, we're comparing the human to the algorithm. Here we have a well-specified objective function for the algorithm. For the editors, it's of course less clear. It's hard to get at it. What we do know from the literature and from this particular setting is that the editors definitely do care about clicks and trying to get as many clicks as possible to get a sense of what the editors do. The line in blue is the share of all articles by topic that are produced overall. The red lines are the share of articles from that particular topic or from that particular category or that particular broad category, which make their way onto the homepage. You can see that there is a reasonable amount of variation in terms of what goes on to the homepage and the green shows the overall share of page views. What we can show in this setting is at least correlationally, there's high levels of correlation between clicks and what makes it onto the front page. Of course, that's not causal, but that's again half a step in trying to get a sense of whether editors care about clicks at all or not. Let me move on to the empirical strategy and please feel free to stop me and ask any sort of question that you might have. The baseline specification is we're trying to explain clicks of user i in session s. A session is defined as a 30-minute interval. It's defined as an interval when an individual logs on and if there's 30 minutes of inactivity, then that session ends. The unit of observation is the user session. We look at clicks of user i in session s and whether the individual was treated in session s. Due to this randomization, what we can do is we can use within a user variation with a user accounting for user fixed effects as well as a time dimension, a time domain to capture general trends in the new cycle. This is just a randomization check to say that on average, the randomization happened okay. This is based on individuals who we saw both during the experiment as well as before the experiment. These are pre-experiment characteristics and we find that in terms of the number of days that they were active, you know, clicks during the day, total clicks, clicks from Germany or outside, they seem to be balanced on average. So, you know, the first sort of sanity check. Okay, so in terms of the baseline, three columns here, in the first column accounting for time fixed effects and individual fixed effects, what we find is that clicks on slots for increase by 3.75 percent when the user is in the treatment group relative to when in the control, relative to the control group. But of course, in the way that I motivated the question, you know, the average effects are less interesting than the heterogeneity that it might capture based on the amount of data that it has. And so we try and see how the treatment effect varies based on the number of observations or the number of times that individual has logged on to the website. And what we can see is that if the individual had no prior browsing history, and hence the algorithm was recommending something based on, based on social data or news or current news trends, we find that the editor seems to be able to predict those preferences much better. But as the algorithm keeps on getting more and more information on the individual, the individuals start clicking more when in the treatment group. So the sort of case that we're building is for better, a better match in terms of taste, which is identified by the algorithm. And when we look at, you know, we see the same pattern when we look at the total number of clicks, you know, when there's limited data, then when you're, and you're in the treatment group, the treatment doesn't perform as well. But as I get more and more data on you as the algorithm, then the algorithm starts performing better relative to the human. Of course, this is also super parametric, you know, we can, what we do next is try and trace this out as much as possible. And what you can see here is that again, you know, in the first few visits, which would be, you know, let's say eight to 10, in the first eight to 10 clicks, the human seems to be beating the algorithm, then the algorithm, you know, is learning, and it's improving over time. But and statistically, these points are different just because sample size, but economically, it's not that much of a difference, or at least, you know, depending on your prior, it's not super high. But what one can say is that, you know, initially, the human, the human seems to be doing better, then the algorithm catches up, diminishing returns sort of set in, but even for someone who's been who's come onto the website about a hundred times. And this is, you know, at the 90th percentile, the algorithm outperforms the editor by about 15% here, which is, you know, which is, you know, depending on the estimates reasonably high. So this is, you know, trying to show that, yes, personalization works, social data doesn't work potentially as well relative to the human personalization works, but diminishing returns set in, you know, reasonably quickly. There are some gains, but, you know, maybe not as large as one would expect. But we want to dig into this sort of personalization story a bit more. Basically, our idea is that an individual who is coming back on provides more and more data to the algorithm. Sorry, Ananya, one question. Will you provide us data on whether or not people come back to the site more often or not when there has been personalization? Because another measure of how well the thing works is whether or not people find the site more useful and whether or not they've got more visits. So I mean, maybe one way of interpreting this would be to, like, total number of clicks is probably correlated with the number of return visits, but we haven't looked at that. But I don't think that it's highly correlated with the total number of clicks. Well, that's not obvious. I mean, you know, people say that, I mean, assume that a page shows your real junk, but, you know, you can't prevent yourself from checking, you know, the 10 fastest men in the world of 20 more ridiculous women dresses or whatever. You might, when you are on the page, you might click on this, but you might decide I'm not coming back. So something to do with this discussion or whether you've got addictive addiction, you know, to websites and so on. Right. So yeah, I think we can run that. If my memory serves well, we did run that at some point, but I don't quite remember. The only slight issue would be that the algorithm was explicitly trained to maximize clicks of the user. So, but we can see if there are spillovers onto that other correlated dimension for sure. And that would potentially be a more longer term outcome as well. Okay. So again, thinking about clicks and preferences, preference heterogeneity amongst the users, we create a base on users who we also see in the pre-treatment period. We use a measure of sort of how far the user is in reading behavior relative to the others using a pre-treatment cosine distance measure. And for this, if you look at columns one and two, what you find is that on average, the further away you are from the average guy, the worse the treatment effect is. But as I get more and more information on you as the algorithm, then I can cater to your preferences more. And hence, if you have sort of misfeeding behavior, I need more and more data to try and cater to your preferences. Similarly, we look at rank effects and match quality. Again, the idea behind the algorithm is that you potentially click on slot number four because it's picked by the algorithm. And this is an article that you might not have seen if the algorithm had not pulled it up from down below. Of course, we can only see what the algorithm does in the treatment group. So this, the variation that we see here is only based on when the users are in the treatment group. And what you can see is that higher the ranks, so further down the page the article is, the less likely the article is going to be clicked. But as I get more and more information on you, then I'm more likely to recommend an article which is a better fit with your taste, and hence as a higher probability of being clicked. So this is sort of one part of the people where we're trying to pin down how the algorithm works relative to the human editor in terms of social data versus personal data. The other dimension is that of the time dimension, the stock versus the flow. The news is of course, a fast paced sort of industry, you potentially need a lot more, a continuous flow of data to update the algorithm to keep up with what the users might want. And till now I haven't said too much about the time dimension here. So there's like, people come back, there are 100 visits, 200 visits, etc. To credit the time dimension, we use another source of variation, which is what we call the new year bump. What had happened was that when the algorithm was being launched in December of 2017, what the data science team unfortunately did was that they hard coded the year to 2017. So when it became first of January 2018, the model was not updated to fetch personalized data. And they didn't realize that there was this glitch till the 6th of January, because I think they were basically away on holiday and they didn't really check what was happening. So what we're going to use is this sort of data bug as a natural experiment within our experiment, keeping in mind that see on the first of January, when users are treated, the data is still sort of fresh from the 31st of December. But data gets more and more updated as you go along that particular week before the bug is detected. And here what we find is that, again, focusing on only January, only December, January, and February, is that on average, again, the treatment effect is positive. But during those particular days of the new year bug, the treatment didn't do as well. And it did increasingly worse as you went along. So the new year day trend or the bug day trend is one for January 1st, two for January 2nd, and so on. And you can see that it kept doing worse and worse. So this sort of gives you an idea that while you can cater to preferences, you need this continuous flow of data, even if you're this large sort of news out. But one question is, okay, fine, continuous flow of data seems to matter within this setting, but can the stock of existing data actually have like a mitigating effect on this sort of lack of continuous flow? That is, if as an algorithm, I've seen you quite a bit before, maybe I need to update my model less because I have a fair sense of your reading preferences, at least within that short period of time. So what we do is that we, again, use this new year bug period and interact it with the prior stock of visits. So this is on the 31st of December and prior to that. And then again, and this is solely the new year bug period. So this is those six or seven days. On average, the treatment effect was negative. But this is, you know, this is sort of mitigated by an existing stock of data that the algorithm has access to. Right. So again, you know, continuous updates matter, but an existing stock of preferences can actually mitigate that to some extent. So that's the second type of data that we've that we've looked at in this, in this setting, which is continuous flow of data and the stock existing stock of preferences. So this sort of rounds up the first part of the paper, which is trying to trying to pin down the value of data. And now in the minute few minutes, I'll just quickly go through what happens to consumption news, news consumption patterns. And what we, what we do is that we can we construct an HHI index for consumption shares of, of a particular user across different topics. And we try and see what happens. What happens to consumption diversity when you're in a treatment group relative to when you're under control. And what you find is that since most of the effect in our case is coming from personalization, when you're in the treatment group, your consumption diversity doesn't decline. So a higher user, a higher level of HHI means a reduction in consumption diversity. And this spills over a little bit to other slots, but not too much. I mean, you know, it's an order of magnitude lower. What happens, what happens when, you know, when I, when I, when I as the algorithm see you over and over again, if you can see that the consumption diversity keeps on reducing over time. And this is sort of in line with our, our returns to, returns to returns to data picture where the algorithm consistently beats, you know, beats the human at, let's say, you know, four to six percent, like, you know, after the initial burnout rate. And so if you keep clicking, you know, if you keep getting served by the algorithm, and, you know, you have a higher probability of clicking on the treatment, which is personalized to your taste, you do see these sort of dynamics taken. I have other results related to, you know, other user characteristics, which might explain why an individual interacts more with the treatment relative to control. And, you know, this is more descriptive. But again, I can talk about it. But I think I'm sort of running out of time. So I also want to sort of go back to how I started out, you know, a general sense of where we are in terms of the value of data. So as I mentioned before, you know, we contribute to this literature on scale effects in data. For example, this paper by Leslie Chu and Catherine Tucker, they use, they use a natural experiment in Yahoo search to find that actually, if you reduce user history or access to user history from like six months to three months, it doesn't change too much in terms of the search precision. By any at all, they find some diminishing returns sort of in line with, in line with our setting. Schaefer et al, they find some, they find some effects in terms of the value of data. And, you know, I think there's a new version out, which I'm not sure what exactly the mechanism that they highlight. But again, it's like, you know, here, the contribution is that we were able to sort of pin down and bring these together within a field experiment. In particular, if you think of the data curve, maybe it can reconcile some of these conflicting results. That is, when, you know, when you have limited data, then additional data for the algorithm or search engine might play a big role, but diminishing returns might kick in reasonably quickly. Of course, this is one context, but the idea might actually generalize. Moreover, this paper is related to the computer science literature, which has evaluated the effects of recommender systems. And this, the computer science literature has the same sort of issue, like with this scale effects of data literature, which is that they look at offline evaluation. So they have a data set, and then they have a training sample, then they have the holdout, and then they try and see how the algorithm does in the holdout sample. But the computer science literature is more and more like embracing these real-time randomized experiments. And what they do find is that these offline evaluations are often at odds with randomized experiments. That is, you know, the results often slip for some reason or the other. So they are also trending in that direction. And in the computer science literature, you know, they don't focus really on the economic value. You know, they're looking at different features and their measures are slightly different. We can actually try and transform these click-based measures into overall revenue measures for the news outlet. And finally, in terms of the result on how algorithmic recommendations reduce consumption diversity, what this is related to, you know, a bunch of papers by Dokyuli and Kartik Hosanagar. But what we do here is that we have 20 weeks instead of, you know, the standard week or two weeks. We can look at dynamics. And actually, our result is almost the opposite because they look at collaborative filtering, which is social data and find that individual consumption diversity actually increases. And what we show is that actually personalization is the key which leads to reduction in consumption by the end of the century. So those are the three broad strands of literature. And that's all I have, you know, trying to outline the value of different types of data within, you know, this field experiment. We find that it's personalization, which plays a big role, at least not in our case, social recommendations don't play as a big role. Continuous flow of data is required, but it's mitigated by an existing stock of preferences. And finally, algorithmic recommendations, due to personalization, are reducing consumption diversity, and this increases over time. And that's all I have. Thank you so much. Thank you so much, Ananya. So now it's the Q&A period. So feel free to unmute yourself and ask questions. Paul, go ahead. Yes, thank you, Ananya. I enjoyed that very much. Can you, I'd like to push you a bit on the meaning of your measures of consumption diversity, because I could imagine two very different stories. Think of consumption diversity in a different dimension, say, in the kind of food you eat. Now, you might look at somebody who eats Indian food tonight, and Mexican food tomorrow night, and Italian food on Thursday, and Turkish food on Friday, as somebody who really enjoys diversity. Or you could look at them as somebody who's desperately flailing around to try and find out what they like. And if it's the former, then recommendation systems ought to be able to figure out that they like diversity and adapt their recommendations to offer them a reasonable variety in the recommendations. If it's the latter, then the whole role of the recommendation system is to find out that they really like Turkish food, get them to that equilibrium, and keep them there. Now, it seems to me your measures are not really able to distinguish between those two stories. You don't know whether the reduction in diversity you see as a result of the recommendations is the fact that basically the algorithm is better at guessing the type of topics you want to look at. And maybe there are other dimensions in which the actual treatment of those topics is quite diverse. All whether what's going on is that the recommendations are somehow rather narrowing the choice available to the consumer, perhaps because of sort of behavioral reasons, they're lazy, they just do what they're recommended, in a way that in some sense betrays their genuine underlying preferences for a diversity in the forms of their consumption. And I wonder if you could think about how you might be able to use your data to distinguish between those two stories? From the outside again, this is one algorithm, the particular algorithm is sort of trained to identify your preferences and try and cater to that and personal data kicks in as soon as it sees enough number of visits. Again, I haven't bought too much about trying to distinguish these, but maybe I'm thinking whether we can use our measures of cosine distance in terms of heterogeneity in user preferences and try and see how that interacts, how the trends in consumption diversity might be different for people who are relatively mainstream going in relative to those who are reasonably diverse starting out. So people who we observe before the experiment as well as during the experiment, maybe that's one way of getting at it. I'm just hypothesizing right now, but maybe that's something that we could look at. Since we're looking at that in the value of data section anyway, we can definitely operationalize it here and that might be it. Any other thoughts, questions? Yeah, sorry, maybe that I missed something when you were speaking, but did you speak about whether or not the total number of clicks change and what we have a substitution with other clicks? So in general, we find the same sort of trend with total number of clicks that is when there's limited data, then overall clicks go down. But as you go on with better algorithmic recommendations, the total number of clicks within a particular session increases. So what we do find is a little bit of business stealing from other slots and the overall the impact of overall click on average is maybe slightly negative, but not less than half a percent. So it almost seems to be a pure business stealing fact. We don't really dwell on that too much because again, because of the specificity of the experiment, if someone is to adopt algorithms, you'll potentially want to maybe do it for the whole website. But yeah, we do see some business stealing based on our estimates. Did you try to use the fact that for some people, the switch was between the fifth and the fourth slot and for others, from the 20th to the fourth slot and whether or not this made a difference in the consequences. I mean, it seems that for some people, what your algorithm did was basically change the order very later. Some, you know, the new story of such in fifth place became in fourth place. Right. For others, it was the new story which was in the 60th place. Does it make any difference? Yeah, so we have, if you can still see my screen, right? Yes, yeah. So if you look at this one, so this is looking at the rank effect. And this is again, you know, this is sort of correlational because we see variation only within treatment. So on average, as you go lower down the page, you're less likely to click in general. But as I get more information on you, and then the algorithm can pull it up from further down, then you're more likely to click. And this is for, you know, including slots. This is starting at slot five onwards here. And this is starting at sort of slot 14. So, you know, like 10, you know, 10 positions down. And you can see that, you know, the effect becomes stronger. So in some sense, by looking at what happens at the beginning, you would have a measure of how different is the taste of individual from the taste of the editors. If you looked at the what happens at the beginning for the first visits, you would, in some sense, you can measure whether or not the individuals have a very different taste than the editor and what is the effect, you know, what does this make for the how people react to the ranking and so to the right, right. So maybe what you're leading to is sort of merging like this, our measure of cosine distance, which is basically using information on on people who we observe before the experiment and how far they were in terms of, you know, their diversity of consumption. And then maybe interacting that with the rank effects. Yeah, I'm not entirely sure. Because basically what you're trying to say is that the rank effects would also be affected by how diverse you were initially versus not, right. Yeah, and how different you were from the editors in terms of Yeah, from the editors. Right. So this is the the measure of cosine distance is trying to capture how far the user is relative to relative to other users. And you're saying that the benchmark should be the editors. We could do that actually, we could do that because we have we have data on what the control group looked like day after day, hour after hour. So we can construct that measure. Yeah, and then we can interact it with these direct effects. I have a question. I was curious whether the algorithm is, you know, said it's maximizing clicks. Is it doing that in a myopic way or is it is there some role for exploration that is, you know, maximizing clicks, not just today for this user, but in present discounted sense. Yeah, so it's from from what we know of the algorithm and based on the paper that they that that they use as the baseline algorithm. You try and maximize clicks for when the user logs on. So I think it's reasonably myopic in that sense. Because if it had a sort of a longer term, I had an exploration aspect to then, you know, some of the implications would be different in terms of, you know, people getting stuck in built a bubble and so on. True, true. So I agree. And, you know, until now, as far as I know, most algorithms try and just, you know, maximize clicks trying to identify your preferences. And but more and more in this literature, people are trying to see how algorithms algorithms or recommended systems can be devised such that there is room for exploration. There are some papers coming out with that. But yeah, in this case, it's maximizing clicks myopically. That's, yeah, right. And this is sort of also related to, I think, Jack's question about, you know, do people come back more often, which is a potentially longer term measure, but yeah. Yeah. And you should have mentioned like, personal data information filtering and then social data collaborative filtering as in two aspects, depending on whether, you know, how much history they use ahead. Is there any way to sort of understand how the two interact? Like, I would imagine there's sort of an interaction effect that, you know, the more the more social data that actually helps you learn faster in terms of the personal data. Yeah. So potentially, if we if you look at this sort of picture, you know, it's, it's generally, you know, generally diminishing returns kicking in. And, you know, this initially there's only social data. And then there's, you know, personal data being fed into the model. What we move from the algorithm is that as, as more and more personal data comes, comes within the algorithms reach, then it gets a higher weight in the recommendations. Can we completely separate the two? Not, not necessarily. And that's why I focus a bit more on, you know, when there is no, when there is no personal data, that's like the cleanest that, that's the cleanest situation for, you know, identifying how this, how social data is doing relative to the human editor. And, you know, maybe, let's say the 50th visit or the 100th visit is, you know, these are two extremes. Can I, you know, help us identify the role of personal versus social? There might be some interaction in this middle, in this middle, in the middle here, which we can't separate out. So that's why even in the paper, we try and, you know, we try and make this reasonably clear. Thank you. So speaking of a diminishing return, like you mentioned that the past studies also kind of discover the diminishing return in their data sets. Like, have we seen any papers? Like maybe this is also for the audience. Have we seen any papers with a non diminishing return data effect, like in the sense that I mean, it might be a constant return or increasingly return. I'm trying to figure out like, like, were we ever seen like a scenario with like, yeah, a different situation. Yeah. Yeah. So, and this is, you know, like, related to the broader conversation about the value of data. So, you know, Catherine Tucker and Leslie choose paper, they don't find any effect. They just use a natural experiment. So they don't, they find none effects. The binding paper finds some level of diminishing returns. And here we have, you know, reasonably clear diminishing returns. I think, I think part of the issue is the in the general conversation is that, you know, what if Facebook uses data from a variety of sources. So, you know, uses, you know, uses WhatsApp data uses, you know, maybe it has, you know, some fitness app that it might promote at some point, you know, uses running behavior, uses, you know, data from Facebook wallet, and puts all of that together for an individual or, you know, a bunch of similar individuals. Then there is some sense that increasing returns might take in at some point. But I have not, you know, it's hard, you know, like, it's hard to get access to that sort of data to actually understand. But I think that's the biggest why when we think about, you know, regulating, regulating, but I haven't seen too many papers which try and which are able to highlight that. Thank you very much. So, any more questions? Not a question, but I would just comment on your question, which is, you know, there are some examples where you, some applications where you will get increased, you would expect to get increased returns, like autonomous vehicles, right? Like, you're going to get almost nothing in terms of value from the data until you get to a very high level. And then once you reach some threshold where now it's actually useful, then you get a massive increase in return. So by the nature of the problem that they're trying to solve, you know, you think about the returns to data, it's like, you need an enormous amount of data to get anywhere. But at some point, it'll, you know, be a huge, it'll be a huge increase. And then maybe after that, it's diminishing again. So I would think, I would think an S shape curve could be quite natural for quite a lot of different applications. But do you know of any, any paper which is even, because I like, in my, in how I see the literature, I don't see any papers. We don't have the data on autonomous yet, right? Because we haven't reached that point, but this is more like from the, from the theory of it, or at least from the people working on it, rather than an empirical paper. Why don't we stop right now, because it's been an hour and I would like to stop the recording. Just to say thank you very much for everybody who's participated and because we are French, who are giving you August off. So, but we are looking forward to restarting the seminars on in September. Julian and Andre have made a very great program. Thank you to both of them, actually, for the great project. And thank you, Ananya. And you know, we're keeping the session open. So if you want to insult Ananya when it's not recorded, you can do it as soon as I close the recording. Bye-bye. Thank you so much. Thank you so much for having me.