 Great. Thank you, Tiffany, for moderating. Thanks, Hanes, for taking a look at the at the paper. And thank you to the organizers, Jacques, Julian, Andre, for having me, having me here. This is, you know, this is my, this is my second time in fact. And this second talk is, you know, is also going to be in line with the, with the theme of, of the first one, which I checked my calendar was around three years ago. And the broad, you know, the broad issue that we want to try and understand is, you know, what is, you know, the value of data from, from a platform's perspective. And, you know, the, the sort of initial motivating theme is everyone sort of knows and has seen this cover of the economist in like, I think this is from 2017, where they said that the world's most valuable resource is data, right? And, you know, whenever we've had these conversations around the value of data, the focus has mostly been the value of sort of quote unquote, internal data, that is, you know, people come onto your platform, you, they leave digital traces, and you are able to leverage those to, you know, maybe provide better recommendations, maybe iterate on your product. And, you know, maybe that, that makes it better for the platform. And in fact, you know, trying to quantify that, the value of that sort of data was, was the theme of the paper that I presented three years ago, where, you know, we looked at the editor versus the algorithm in joint work with Chris Boykard and, and York Lawson, where what we basically did, oops, sorry, yeah, where we basically did what, what we basically did was that, you know, it was a large scale field experiment where we tried to analyze human curation decisions versus algorithmic recommendations. And in particular, the focus of that paper was how does the value of, you know, internal data matter to make your algorithmic recommendations better, right? So how does the value of personal information matter in a general sense, the traces that they leave online. So in particular, the volume of personal information, the stock versus flow of internal data, the variability in external events, you know, that creates variability in the amount of data related to news stories. So, you know, this is sort of a throwback to, to this seminar series, but also the broad sort of picture that we've generally been looking at the focus on internal data. Now, this paper is motivated, you know, by, you know, within this broad umbrella by the observation that, that practitioners more and more highlight the need to use external data. For example, you know, over 90% of analytics professionals have said that firms need to increase the use of their external data. You know, many consulting companies have put forward the ideas that, that firms may gain an edge by incorporating external data as part of their broader data ecosystems. And how, you know, how does this external data, you know, how do you actually leverage this external data often through these data sharing agreements, right? You know, APIs or application programming interfaces, they're becoming increasingly common. For example, you know, Google search, Google search and Google search suggestions as an API for publishers and developers. So, in some sense, that's the, that's the focus of this paper. In particular, what we mean by external data is, you know, digital representations of acts, facts or information that's provided by external providers, often through data sharing agreements. And the context of this paper will focus on, you know, one particular dimension or one particular type of data input within a search market setting. Now, why, why we think, you know, this is an important area or broadly an important question is despite, you know, the talk around it, despite its potential economic relevance, it's been challenging to pin down its causal impact. Now, Seth Benzel, John Hirsch, and Marshall Van O'Steen, they have a nice paper which looks at, you know, what's the, you know, what's the benefit to firms when they open up their data or, you know, when they open up their platform through API access, not necessarily only data. And they try and quantify that. In some sense, our paper is going to be a complimentary paper, which we'll look at the other side, which is that if I'm a smallish sort of company or a new sort of product, what's the benefit that I get if I tap into a larger player's, you know, data or data set or larger players broader data ecosystem. Now, you know, this, this topic is not only important from, you know, let's say, a managerial perspective that, okay, you know, platforms can maybe or companies can leverage external data, but this also has a policy flip side to it. For example, you know, with the Digital Services Act and the, and the Digital Markets Act passed recently in, you know, in Europe, it sort of classifies companies into gatekeepers. And, you know, if companies are of a certain size, above a certain threshold, there are these larger companies that are term gatekeepers or market leaders. And at least, you know, in, in the recent drafts or, you know, the regulations that have been proposed, especially in the context of the search market, imposes certain obligations on these larger gatekeeper companies. In particular, they say that, you know, you have to provide to any 30 third-party providers of online search engines with access to ranking data, query data, click data, view data, you know, assuming, you know, it's in a de-identified manner. So, you know, that these regulations are sort of imposing these data sharing agreements, right? And this is not only at, you know, this is not only an EU thing in China. Recently, there was a 20-point agenda around the data economy with the idea that data sharing should enable the growth of small and medium-sized companies. I haven't seen similar regulations being proposed in the US, but, you know, US in terms of regulation is always half a step behind. And in the US, I always look at California. I feel that California is the Europe of the US. So I'm sure, you know, we're going to see something come up in the near future. So with this sort of, you know, this big picture setting, the questions that we ask in this paper is, what is the causal impact of access to the market leader's data on the focal company's product performance? Our context will be of the search market, where, and in particular, we're going to look at search suggestions, think of autocomplete. And the market leader in our context is, you know, is basically the biggest player, or is the biggest player in the field. And the, you know, when we're talking about market leader's data, we're going to be looking at one particular dimension of the market leader's data as an input into the broader product of that's being developed by the focal company. That's the baseline question. And then we try and see how, you know, if there is any sort of impact on getting access to the market leader's data, how does it vary across types of users and types of content? You know, heavy users versus new users can leveraging external data, maybe mitigate the cold start problem. How is this sort of data input into the search process? How does it vary in its efficacy in terms of like popular or mainstream content relative to niche content? You know, we have to keep in mind that, you know, one of the fundamental benefits of digitization is that, you know, we can identify people's preferences and cater to even long tail preferences, right? An interesting dimension of this, of our setup will be that the queries through our, through the API will be de-identified. So in some sense, there won't be that much personal data at play. But, you know, we still want to see whether such, you know, data sharing agreements can actually lead to, you know, benefits in terms of both popular and niche content. Moreover, the final sort of question that we'll be looking at is the impact in the short term versus the longer term. You know, of course, these are relative terms. Our, we'll have a field experiment. And so our field experiment goes on for about 16 months, I think. And so, you know, we can look at the effect in the first two months relative to how it traces out over time. Just to give you a sense of where we are headed with these results, our experiment will be about removing access to the market leader's data. So access through an API. So think of this as, you know, think of this as external data leveraged through APIs. What we find is that removing access to the market leader's data leads to a 4.6 percent decline in engagement with the product. You know, it's, we, you know, so we can have a conversation about whether we think that this, you know, this magnitude is big or small. I'd love to get your input. Then we talk about, you know, how it varies across types of users and content. What we find is that heavier users are affected more. But despite the de-identified nature of these API calls and how the API data is used in the process, we find that both popular and niche content are affected. So this gave us a sense that, you know, maybe at least this particular type of data sharing, which I'm going to go into the details, can actually, you know, in a de-identified manner can help both popular and niche content even though the API calls were de-identified. Finally, what we find is that the average effect is much smaller than the decline in performance overall. So initially, the decline in performance of the product is much larger. And then it becomes less negative over time. And we have a reason, you know, or at least we posit one, you know, one mechanism which might explain that, which is that potentially when you're leveraging external data, your own internal systems are not being able to develop as much based on internal data. So when you remove, you know, access to that crutch, you know, it helps your systems develop better. And in fact, like at least anecdotally, you know, in conversations with our partners, we do, you know, one of the reasons it seemed as if they were willing to run this experiment was to see whether their product would stand on its own. Okay. So that's the broad overview. And now I'm going to go into some of the details. Tiffany, we're good on questions. Right now, we haven't received any questions, but I will let you know if there's sounds good. Yeah. So, you know, with that broad overview, let me give you more details of the context. So we partnered up with a leading Chinese technology company. It has, you know, millions of active users. Now it's, you know, the product, it's an app, like a super app as in, as is the case in, in many of these Chinese applications, which has a number of functions. There's the news feed, there's videos, there's, you know, you can stream ebooks, there's the search engine, file management, et cetera. Our focus is going to be on search suggestions, which was a product that was being, that started being developed by the company in 2020. We treat, you know, we partner with the team that was developing this product, which is part of the, of the larger company, but it's a startup like setup, right? It's, it's not a large number of members. They, they have, to some extent, limited access to computing facilities, et cetera. The new product, which is the search suggestions or autocomplete is embedded within this super app, as I was, as I was saying. So in particular, you know, the, the, the context is that of search suggestions or autocomplete. So, you know, keep in mind that as you start typing something in any search box, there are these drop-down suggestions, right? And the main aim of this team was to try and make you click on those suggestions, basically because it'll bridge the gap between the user's intent and content consumption. Companies seem to invest a lot. In this, there's, you know, more and more evidence coming out that this, this has economic relevance. In some sense, you know, we've been talking about, you know, we've been talking about generative AI or, you know, chat GPT, et cetera, over the past few months. This is one of the, the early applications of, of generative AI models. The main aim of the team that we, that we partnered up with was, as I said, to, to have people click on their suggestions, right? That's, that was their main metric of trying to, you know, of trying to understand whether their search suggestions are actually relevant or not. There, you know, this was the initial goal. And it's part of a broader media, medium to longer term goal, which is to, to actually monetize these suggestions, right? So it would be such that if, you know, if you click on these suggestions, then there would be a link embedded, which would take you to the website straight away rather than taking you to the eventual search page. And if you do actually, if you're able to monetize it, then you actually get revenue from, you know, from the, the website that was actually embedded in those suggestions. So for example, if, you know, if I'm sitting somewhere and I start, you know, typing Carnegie, right? For Carnegie Mellon University, I type C-A-R-N-E, there'll be a drop down. Maybe there's Carnegie Mellon University. I click on it, then the, then the focal company gets some revenue. And what happens is that the link takes you straight to the CMU website. So, you know, that's the, that's the broader goal of this team. Hence, our outcome of interest would be the click-through rates here, which is, you know, a way standard measure of trying to understand, you know, a standard measure of success within, within such search settings. But we also look at alternative dependent variables. So now let's get to the experimental design. What we have is the following. So think of, you know, think of the way that these search suggestions or any sort of recommendations are generated, think of it as a funnel. You know, there's millions, you know, there, there are billions of items, which in the initial stages, there's, there's a lot of pruning, you know, reducing dimensionality, you know, duplicates, et cetera. Our, our experiment is happening at the ranking stage. The ranking stage is when, you know, you have maybe hundreds of, of candidates after removing duplicates, et cetera, where you have, you know, where you're finally providing the, the top 10 as suggestions for the end user. Our control condition will have had about a million users. And in the control, they, there was access to the market leader's API, which provided certain candidate data or candidate inputs into the ranking stage. So across both conditions, the ranking algorithm remains the same. This is where, you know, there are super sophisticated algorithms where you have to merge, you know, data with, with, with personal information, et cetera. But keep in mind that queries through the market leaders API are de-identified. So there is no personal information involved. That was the status code. So when, when we came onto the stage, the experiment was removing access to the market leader's data. And so in this case, what happens is that the entire funnel is simply based on, on the focal company's data, right? All candidates that are, you know, after pruning, et cetera, that serve as an input into the ranking stage are provided within the, the same system and there's no access. So keep in mind that the, our treatment is the removal to the access, removal of the access, the market leaders API. I can always come back to this figure, but I want to provide a couple of other details. So, you know, what's in it for the market leader to actually share its data. And, you know, as we, as I, the, the paper that I mentioned early on by, by Benzel Hirsch and all, Van Allstein, when the market leaders opens, opens up its data ecosystem, for each query that's made, it gets, you know, some, some amount of money, right? So the market leader is getting some, you know, some fee, while the focal firm is getting, you know, sort of data input into the critical ranking stage. One of the, one of the reasons why the focal firm, you know, really wanted to tap into this was because, you know, they were worried that they just didn't have enough candidates, right? It's, when you're starting up a product, remember that the product was started in, in 2020, in the initial days, they just, if you don't have enough data, you know, what do you serve up? What do you serve up potential users of that product? So the focal firm thought that, you know, there might be quantity and quality, depth and breadth of data, and that's why they entered into this relationship. If a few more details just so that you get a sense of the context, and then I'm going to, I'm going to dive into the results. This API is available for other companies and individuals within the same market. So it's not as if, you know, this is some super special arrangement. These are, this is a commercial API that's available. As I've mentioned before, queries to the API do not provide personal information. The ranking algorithm generates a final ranking and the top 10 candidates are displayed and they're basically using state of the art, you know, machinery here, the experimental architecture is, you know, is in line with the industry standard to the best of our knowledge. Of course, you know, one caveat here is that this is one data sharing context which is out there in the market, right? So, and it's happening at the, you know, at the stage where there are candidate inputs into the ranking, into the ranking stage. We feel that this captures one dimension of what the regulations have talked about, if, you know, to sort of recount, you know, the obligations of the gatekeeper within the DMA setting, you know, is potentially making data available on ranking, query, views, click. So this is, you know, again, one type within that broad setting. Though we've, you know, even though it's one setting, we feel that it's important, not only from a policy perspective, but it's, you know, this type of data sharing agreements are out there in the market. Google, you know, Google with its search solutions also does something very similar. So it's just also good to understand what value they bring to the platform within such a setting. So, you know, with that sort of state set, let me go a little bit more into the details of the data and then the results. So about two and a half million users split equally between treatment and control over 100 days. Now, this is a between subject assignment. So when an individual is assigned to the treatment group, that individual stays in that treatment group for the entirety of the experiment. Remember, the treatment group is the removal of access to the market leaders API at the ranking stage. Our unit of analysis will be at the level of the user exactly because that's where the randomization has happened. Our main outcome variable of interest is our click-through rates, but, you know, you can, you can substitute it with total number of clicks, probability of click, etc. You know, it's all very, very stable. Our NDA, you know, prevents us from disclosing the baseline clicks, so absolute levels. So we're going to be talking about the lift in click-through rates. You know, it's, it's basically the incremental CTR amongst the treated users as a percentage of CTR amongst control users. But, you know, this is a, is a standard way of, of measuring, of measuring these outcomes. Okay. So I think that's basically it. And now let me, let me sort of go into the data. Tiffany, we're good on questions. Just checking. Yeah. Right now, there's no question. But if any audience has any question, just free free to raise your hand or post the question in the chat. Oh, so Hanna has, oh, Hanna actually posed a question. Does the firm strictly forward users to external pages? Or does the firm forward user to the core product's website? Oh, right. So do I think, I think maybe a bit more clarity in the context is the following. That since we were partnering up with the team that was focused on the search suggestions, you know, the, all the data that we have is what was, you know, in the sort of dropdown menu. So we don't, we aren't actually able to track what happened, you know, or what pages, you know, they, they were led to post, you know, after they click. Right. So it can be, you know, it can be a combination. It can be like internal, you know, internal pages, external links, as I said, you know, for example, like I was talking, you know, I was talking about university links. But if there is, you know, if there's a query related to COVID, it can be, it can be user generated content. It can be, you know, a different health website, a doctor's website. It can be maybe some information that the, that the focal company has on its own. So it, as far as I know, it seems to be, it seems to be a mixture. The, the sort of, you know, this is, it was Hannes, is it? Yeah. So, so Hannes, let me know if that's, if, you know, if that's okay. But, you know, one of the, one of the maybe limitations of, you know, partnering up with companies is that, you know, you partner up with the team. And that's, you know, that's basically the data that you get, you know, because you are buying from that particular team. So we are not able to, we are able to sort of see whether they engaged with the, with the drop down suggestions, but not that much more afterwards. Okay. Okay. So, so now just a few quick randomization checks, you know, just to make, you know, just so that we're all sure that, you know, the randomization took place properly. It's a valid experiment. You know, when we look at user characteristics, cross-gender location, operating systems, you know, their activity in the past, query data, etc. You know, they're, they're pretty well balanced across control and treatment. On the right-hand side, panel is the assignment of new users to the treatment relative to the control group over the course of the experiment. And that also, you know, seems pretty balanced. So we were happy that, you know, what, what we are measuring would, would in effect be the impact of the treatment rather than any selection issues. Okay. So baseline results are the following. Overall, removal to the access of the market leaders API reduces engagement with the product or click through rates on those search suggestions by about 4.6%. Right. So we, you know, I, I like, you know, to get a sense of these point estimates just because, you know, it's 4.6%. To start off with, it's not 46%. Right. It's, it seems to provide some benefit. And so to sort of not be in my sort of subjective bubble, we tried to benchmark these magnitudes, you know, across different settings, you know, within the literature. For example, you know, the paper by Berman and Israeli, they look at the provision of data analytics information to online retailers and find that, you know, revenues increased by 4-6% when you provide that data analytics information. Bryn Yolson, Eric Bryn Yolson, Wang Jin, and Christina McElherrin, they find that, you know, when there's data-driven decision making, you know, it increases, it increases sales and profits by about 3%. Again, throw back to the same sort of, to the, to the same seminar series, the average effect that, that I find in my paper with Chris Boykert and Yolson, you know, of algorithmic recommendations relative to human decisions is about 3%. So, you know, we're, on average, we seem to be in the same sort of ballpark. And I just, you know, I feel that these exercises are important so that we get a sense of, you know, what's in it for platforms and what's in it, you know, when we think of regulators thinking about how big a benefit it might provide to data sharing agreements, it's, you know, it's useful to pin these down. Okay, so that's the baseline effect. Then we talk about some heterogeneity. And what we find is that heavy users in red measured by, you know, different, you know, using different measures seem to be affected, seem to be affected more relative to new users. So if you, you know, if you remove access to this third party API data, the, the decline in click through rates is much larger for heavier users. This suggests that, you know, while it does impact new users, data sharing, you know, through these agreements has, you know, doesn't have that big of an impact, maybe enough to solve the out and out cold start problem. I'm going to come back to this result when I talk about the dynamics to understand what, what might be happening here. Next, I want to talk about the impact on the different types of content. In particular, I want to, you know, remind us that queries through the, to the API are depersonalized. So depersonalizing the sense that if I'm, if I'm typing COVID-19 symptoms, when there's an API call, the API, the, the external party's API or database doesn't get information on my location, on my browser, on, you know, on the type of device that I'm using, et cetera. And despite this depersonalization, what we find is that the click through rates on popular and niche content declines by similar, similar amounts. And, and, you know, we can dig deeper. So this, you know, columns one and two are looking at popular nation. You can define them in different ways. What we also find is that, you know, the number of unique categories of the topics that are clicked on, which we can classify through, through our teams algorithm, the number of unique categories clicked on also decreases when you remove access to the third party's API. This suggests that, you know, as the number of unique categories clicked through rates declines, this suggests that the gatekeepers data might be broader. So despite depersonalization, because of maybe the breadth of, of the gatekeepers data, you know, there might be a benefit for a startup like, a startup like company or product to actually leverage access. Now, the, maybe the final, yes, so I have about five, five or six minutes. I'm going to spend that talking about the long-term effects. And then I'm going to very quickly wrap up. Given that, you know, we were fortunate enough to run an experiment or leverage an experiment that ran for almost 16 weeks, which is, you know, much longer than the standard one or two-week experiment that you find, you know, when you're, when you collaborate with industry partners, we can actually trace out the click through, the click through rates of the treatment effects over time. And what we find is the following, that the treatment effect initially became very negative, you know, around 8.5% in the first two or three weeks. But over time, it became less negative or, you know, stabilizing around 3.5%, you know, between like weeks 11 and 16. So it's sort of pretty stable. So the magnitude of the effect is, you know, half as large in the longer, in the longer term relative to the first few weeks. And we feel that just, you know, as an exercise, when we're trying to pin down the value of data or the benefits that that a platform might get, or from a policy perspective, you know, it's good to look at, you know, how, how things evolve over time. Because, you know, if we had stopped our experiment in the first two weeks, the first three weeks, we would have really overestimated the value, the value of this particular type of data in particular. Now, to try and understand what might be happening over time here, we posit the role of, you know, algorithmic learning, maybe algorithmic learning is, is, is slightly too strong. But just the development of internal algorithmic systems due to internal data. I can go into the details in the discussion more, but there are, you know, behavioral reasons for why we might be seeing this, this upward trend. And one, you know, one thought that we had, which we didn't necessarily believe, but it's, you know, one behavioral explanation could be that look, you know, if I'm a user who's been using this product for a while, and I see a drop in quality, initially, my engagement with the product might decline. But over time, I might just use, get used to this not so great quality, right? And so I'm, you know, either, either I change the type of prompts I provide, or I just, you know, that's, that's the quality. And that's what I get used to. Now, to rule out that behavioral mechanism and the fact that, you know, this might actually be, you know, the internal data systems actually becoming better because they are not dependent on external data anymore. But for, you know, they have the full access to the internal data through the different stages. We look at the heavy users and new users again. The, the idea is the following. If I'm a, if I'm an old or a heavy user, you know, from prior to the experiment, I know what the product was like before. And hence, you know, if you remove access to this data and the product quality declines, you might see this upward sort of trend. I might get used to it, et cetera. If I'm a new user using the search suggestions for the first time, then I didn't know how it was earlier, right? So then you should, you know, if it was just me getting used to the quality of the product, you shouldn't see this upward trend for new users. If you do, then maybe there is something else going on, which we posit to be this sort of, you know, algorithmic learning behavior. I'm going to come back to this, but what we do find is that, you know, this sort of upward trend or the treatment effect becoming less negative over time is true for old users as well as new users. And we define new users here as users who log on and use the product for the first time every week, right? So if this is the second week you're using the product, you're not a new user anymore. And we see this upward trend of like, you know, about 6% going down to about, you know, 1.5% or 1% negative 1. So we see that, you know, this sort of negative trend is becoming less negative over time. If this is happening for new users, then maybe there is a fundamental improvement in product quality. And that's where we are talking about, you know, the sort of training of the internal systems using the focal platforms data. And we believe that this sort of makes sense because often, you know, these data-sharing agreements, for example, you know, we got the usage agreements of, you know, using chat GPT from OpenAI. And often these data-sharing agreements prevent you from, you know, prevent you from using the API data as training, you know, as training data to improve your product. For example, this, you know, this, this passage that I've highlighted in the terms of use says that, you know, you cannot reverse assemble, reverse compile, translate to discover the source code or underlying components of models, algorithms use output from the services to develop models that compete with OpenAI. Right. So our, our sort of thought process is that, you know, we had these two competing explanations, at least we can rule out one with potential suggestive evidence. Jacques, I'm, I'm just, I'm just getting, I think I'm almost out of time. So I'll wrap up and we can, we can chat after. Okay, sure. Yeah. There was also another question from Hannah, but the last way after the Q&A. Yeah. Okay. So finally, you know, I had talked about, you know, sort of thinking about external validity, the fact that, you know, we don't want, you know, we, we don't want to try and over claim. We are looking at one context. There's, you know, there's, there is some external validity to our context. For example, Google had restricted access to its autocomplete API in 2015 and it became quite a big deal. First, we were trying to, you know, use this as a natural experiment where we just didn't get the data to try and understand, you know, what its implications might be. We weren't quite sure, you know, what the right context would be here. And so just that, you know, it's, it's happening elsewhere to restricting access. We have, you know, we have a smaller field experiment where we tried to make sure that we carry out the right manipulation checks. I'm happy to go into it if, you know, if there are questions that come up. Okay. So maybe I'll, maybe I'll conclude with this slide. You know, again, talking, you know, talking to this crowd is always, you know, is always wonderful. I'm looking forward to Hannah's comments and, you know, and other questions. But, you know, the way I'm thinking about, you know, quantifying the value of data, there are these different sources, right? And there are all these papers that are coming out sort of now or in the past couple of years, which are really trying to quantify it. And so, you know, the way I think about it is three sort of different pillars, you know, the focal firms, internal data, that's, you know, that's what I was talking about right at the beginning of the talk. There's, you know, external public data, which can, you know, create opportunities for new entrants can lead to more innovation. So these sort of, for each of these implications for companies or platforms, there's always a policy flip side. And finally, you know, when we, what we focused on in this, in this paper is external data or third party data or data, you know, access to these sharing agreements or APIs. And again, you know, I think that this is a really exciting space with lots of, you know, new papers, new work coming up all the time. So that's all I have. And I'm more than happy to take any questions and listen to what Hannah says has to say. Sure. So I'm grateful to the seminar organizers for inviting me to discuss the paper. And I would really like to compliment the office, especially Ananya for this really nice piece and excellent presentation. So first, want to share a few observations of why I love this study. So first, I think the context is highly relevant, as firms increasingly share access to their algorithms and data with APIs, being really the primary means of doing so. APIs are so like easily adopted, particularly when they result in like performance improvements that you immediately in terms of time gains or monetary gains. However, an overlooked fact is that adopting APIs really makes firms dependent on providers. And this paper really highlights these unintended consequences of relying on such external data, inhibiting the development of one's own algorithm. It really has huge implications for data sharing policies such as those outlined in the DMA. Second, I think the presentation and paper provide really nice insight into how firms actually implement and structure their algorithms using APIs. The study can serve as a kind of blueprint for other firms looking to conduct similar experiments and for researchers interested in evaluating the impact. After thoroughly reviewing the manuscript on SSRN, the most recent one published in May, and listening to the presentation, I do have several suggestions that I hope help Ananya and his team to further enhance the work. So a first recommendation relates to motivating the outcome metric used in the study. The click-through rate on search results. The authors demonstrate that removing access to the API decreases the CTR and alternative metrics such as the absolute number of clicks. And the authors justify the use of CTR as an important metric due to the potential revenue generation mechanisms on the site. However, considering post-search metrics as an alternative outcome measure could provide valuable insight. For instance, firms often value time spent on a website. Although there may be fewer clicks on those search results, as Ananya has shown, consumers might actually spend more time browsing individual content on the site of the focal firm without really feeling the need to search for new content. Consumers may also care about the quality of the content they find through their search. For instance, it's conceivable that a company's own algorithm is more likely to direct users to existing content of the focal firm while the API may guide users to categories with maybe limited content to discover on that focal firm. Thus, a higher CTR with the API could imply that consumers need to search longer before finding what they truly desire. To address these concerns, the authors could compare a session duration after users have clicked. And if such data, an obsession duration also decreases after the removal of the API, it could provide further evidence for the claim. If this team doesn't have access to such data, which I think they don't after listening to the presentation, exploring insights on null results of the search function could be an option. If such null results, like the search functional leading users know where, for example, because of experiencing high bounce rates, are comparable across these experimental conditions, it may alleviate concerns that the API produces better search results, but ultimately fails to guide users to relevant content on the site. As a side remark, and I didn't think of this before, but maybe the authors may also want to motivate that search functionality tighter in the paper. Like from the presentation, it was clear that the product really was searched whereas while reading the paper, I thought more of a digital platform and then there's this little search functionality. So I think the presentation was a little clearer than what I saw in the paper. And then second, I want to highlight a concern relating to the firm's own search algorithm, which supposedly underwent drastic improvement during the experiment. So again, after listening to the presentation, I'm not really sure this happened, but that was kind of the feeling I got when reading the paper. I believe it's highly important to explain why the firm actually worked on this algorithm and improved it. In a way, it feels like they have been messing with their own control groups. Were they interested in cleanly measuring the effect of the experiment? Did they take a look at the initial results while the experiment was running and then accordingly started tweaking the internal algorithm? If so, what do we really know about these improvements they have been making? My biggest worry here relates to how the answer to all of these questions affects the way we can interpret the long-term effects of the removal of the API. Without a firm making these undocumented changes to their internal algorithm, should we maybe rather expect the short-term effects to apply in the long term? One way forward is to write more with details about these improvements that have been done. So we know about the conditions that other firms may have to create to expect a similar attenuation effect over time. If no further details on this algorithm can be obtained, then maybe they all know how the firm can curb the negative effect over time through improving their own algorithm. But right now I'm really struggling to see whether we should rather expect this effect to be 4.6 with this algorithm improving over time, which may be done by the firm itself, by actually intervening or whether I should rather believe this 8.9 or something, this short-term effect without expecting the firm to make any improvements. And then I have a few minor suggestions. The moderation story is really interesting, but only gets very shortly tutored in the manuscript. Similarly, the authors could more strongly motivate the additional field experiment at the end of the paper. Probably it's value, but it kind of gets lost a little bit at the end. Finally, I would be interested to see and hear the authors' perspectives on comparing external algorithms without personal data, such as this API, to a firm's internal algorithms that do use personal data. With such a data advantage, can we expect firms to beat external algorithms faster? If so, this should have implications on how a smaller firm store and use personal data. And it may be worth discussing this at the end of the manuscript. So all in all, let me again compliment Ananya and his team on a very strong manuscript, and I hope you find these comments useful for further improving your work. So all the best for this study, and let's conclude my discussion.