 Thank you very much. I see so many familiar names. I wish we were together and in Toulouse. That would be fun I'd like to see that new building finally. Anyways, happy to talk about this paper and I see a couple of my co-authors are on So when we get tough questions, I'm happy to share let them share in the opportunity to clarify All right So this paper is about this general data protection regulation called GDPR and you I'm probably you know what it is But if you go to websites and you see them asking you whether you accept cookies, then you know You're living in in a GDPR world. It's because of GDPR that websites need to ask you to give permission For the use of your data that's sort of one of the aspects one of the visible aspects of GDPR to delay people and even you know sophisticated people That's why you see the accept button a lot. I guess I'm sort of a deeper sense GDPR does a few things it makes restrictions on the use of personal information Which if you think about it the way the web tends to work that's going to make ad revenue lower Because sites won't know as much about people it also has compliance regulations Which are going to raise the costs of being an app or a website or whatever And if you believe those possibilities and they seem like plausible possibilities of the outset Then you might think that it could you know raise exit and perhaps reduce entry and if that's true It could have big potential welfare consequences, especially now Here's an ax that I personally have to grind especially if product quality or value is unpredictable Entry you know this is one of a guess a bunch of papers I've written where I think about the welfare benefits or costs of either getting a lot of new products or in this case Maybe losing some products and unpredictability is a big part of that story So if you think about Unpredictability and the welfare gains from digitization so many of you know the last 10 years or so a lot of my talks A bit about a related but different phenomenon But well what's happened under digitization the enormous growth in the number of new products and and how beneficial Is that to consumers now the reason unpredictability is important in that context and frankly here as well if product quality were perfectly predictable and Costs fell relative to revenue. We'd get a bunch of new products But because of predictability these products would all be worse than the cost threshold the old cost threshold So it wouldn't that be that big a deal on the other hand if product quality is Unpredictable entry and the products I often talk about are things like music and books and movies And we all know that it's completely unpredictable almost completely unpredictable in those Context whether products will turn out to be valuable or not Well in that case things that cause an increase in the number of products entering Can have a really big effect on welfare because the new products with unpredictability are as good on average as the existing products Well with that kind of backdrop again about my axe to grind in this context is that it raises the question whether GDPR by potentially reducing Entry especially I really want to focus on the distinction between entry and exit But by potentially reducing new entry is it like digitization in reverse? You know causing us to take a lot fewer draws and therefore to discover a lot fewer valuable products Okay, so the context and the questions The context we're going to study is the Google Play Store. There are four point one million apps we're going to look at this period between July 2016 and October 2019 and this is apps for for Android devices and we're going to start with four descriptive question First of all, what happened to entry and exit and the number of apps available. I guess and that's some sense That's already three, but we're going to call that one question. What happened to the privacy invasiveness of apps What happened to app usefulness as inferred from from usage that the post GDPR birth cohort apps get up There's a lot packed into that sentence and I'll talk a lot about that over the next 25 minutes or so What happened to app development costs as one might infer from the usage that they get You know, if you think about free entry and needing to get revenue to cover costs And then we're going to turn to a sort of a light structural exercise And we try when which we try to draw some inferences about the long run welfare impacts Of the of the what we'll document as a reduction in entry. We had a context with a great deal of unpredictability By the way, what are the ground rules here? Do we get questions going through or do we do we what do we watch the rules? I mean, I probably should keep my mouth shut, but I want to be fair I guess it's up to you. Okay. I mean, I'm happy especially clarifying questions at any time and then, you know, we can talk about deep stuff later, but All right, fine So a little bit about the app market. There are two big platforms. There's apple and there's android again We're going to be studying android just a little bit about the economics of how this works The revenue direct from users is on the order of a third And so these are kind of big numbers 43 billion in 16 up to 83 billion in 19 Ad revenues a bigger deal 80 billion up to 200 billion over this time period now android overall is something like About from 40 up to about 100 billion over this period again because apple is a big separate chunk of the market So just just I guess this is a slide that says this is kind of a big market and it's growing on the market size The number of devices is about a billion in june of 2014 up to about 2.5 billion in may of 2019 Which is just a useful background fact It's also going to play the role of our market size and some of the estimates down the road Okay, so a little bit about GDPR and I'm not a super expert on this But I think I know enough to tell you something so what under GDPR what developers have to do is to protect the user data It's really about giving users control over their own data And so they have to protect it by quote design and default Now it binds on on essentially the whole planet in the following sense all EU based sites as well as sites visited by EU users That's why even non, you know, you go to a non EU site and you see this do I accept cookies business because if they want to To stay available to EU users who are a big part of the world economy They have to comply So there's this is actually a challenge for us in the next few slides There is essentially no really untreated part of the world at least none that we could identify In any event though There are pretty, you know, toothy fines for violations and these fines have been imposed Now, you know, if fewer users are going to allow cookies are going to press that permit button then ads are going to be less valuable And as I already said on the intro slide, the bottom line possibility is that revenue would fall and costs would rise Now it passed in 2016 and became effective in 2018 What we have on the right side is just some pictures of sort of time series reflecting awareness of GDPR So this is like google searches on GDPR GDPR comments at reddit and stack overflow and editing of the of the wikipedia GDPR article This is all just to say yeah, this isn't just something we're thinking about I think that the world was aware of this, especially as it as it came into into effect including developers You know, moreover, uh, we and my co-authors did a survey of some german app developers just to ask them, you know About kind of how do they understand GDPR to to be affecting them in prospect and 85 of them mentioned administrative burdens 48 mentioned additional costs One in seven of them mentioned removing an app exiting and one in 11 reported not launching an app in prospect Under the prospect of gpr. So all of this is just to say this isn't just something we're imagining might be important I think the world is aware of it including the players or participants and in the market. We're studying So there's a lot of literature by now and I I hope I probably missed a tons, but I mean But I want to divide it into chunks So one chunk is just about the welfare benefit of new products I'm of course very famous papers about that are up many many vans and apples and been cheerio cheerios But you know, those are kind of contacts to our product enders and we sort of know what it is and how valuable is it Work i've done with lewis agar has been about this question Well, when success is unpredictable watch the welfare benefit of new products And now it's a little more complicated because you have to think about which products water would not exist In the absence of some kind of you know, some change like digitization There's also a kind of a long tradition in thinking about entrepreneurship as experimentation, which is not unrelated You know the ideas new products are unpredictable. And so, you know entrepreneurship is like experimentation Then there's a sort of a series of sub literatures just on the effect of gdpr on various kinds of outcomes You know for you know, there's a bunch of interesting careful papers Simply even asking does gdpr have an effect on the extent to which Websites collect data and stuff like that I don't think there's I don't think there's much over on the on what we're doing exactly on innovation Although that may have changed Then you know, it's related also just to generally this literature on on privacy and because of course gdpr at you know At heart is a is a privacy regulation And I guess at some deep sense what we're talking about in this paper is the the some maybe unintended cost of pursuing Pursuing the privacy regulation then of course, there's also a bunch of papers just about the app market per se And so this is all all relevant useful stuff All right, so our theoretical framework In the 40 minute version of this talk is that you know gdpr reduces revenue raises costs Now the way we're going to think about it It's a free entry world And so, you know think about free entry if revenues Fall and cost rise you expect a smaller equilibrium number of apps and maybe it takes a while for all that to happen But I want to emphasize a couple things one is really different effects on and of entry and and exit And moreover the the the entry side, which is really going to be where the important action is There's been some confusion about this among our critics, but the the effects of Changed entry are really going to depend a lot on the predictability of app quality So exit let's just sort of break this down on the exit side Apps that are already in existence have known realized quality that is their developers know how many people have downloaded how many people are using it That's not there's no uncertainty left there and so If revenue falls and the cost of bringing the firm or the app into compliance rise We would expect pretty immediate Exit but the exit's going to be low value apps the apps that aren't worth updating Okay, so exit, you know, we may see a giant extinction event But it's not going to be a big deal because it's going to be the really low value apps that exit Entry a little different here, you know with with GDP are coming App developers expect Less revenue and it's going to be more costly to be compliant So we expect entry to fall even in advance of the of the you know as it's being discussed or understood to be coming into existence Now again, if you think about the welfare implications of these exit and entry You know outcomes the exit of low value apps again while dramatic looking is not going to be a big deal If a bunch of apps nobody's using leave the market not a big deal from the welfare standpoint Decreased entry though. This is sort of the dog that doesn't bark. So obviously this is potentially a much bigger deal, especially over time Okay So what we're going to you know, sort of the the title of the paper refers to this lost generation of the apps that don't get Launched that would have been valuable to consumers All right, now I want to also talk a little bit about an extreme Case that's not quite true, but it's close enough to being true to be and quite easy to understand that it's useful to talk about What if it's literally the case that nobody knows anything here? I'm quoting William Goldman the screenwriter who wrote butch Cassidy and the Sundance kid and also this wonderful book about Hollywood called Adventures in the screen trade Screen trade excuse me and hit almost as mantra is nobody knows anything Which is his description of the total inability of people in Hollywood to predict which movies will be successful Well in our context, nobody knows anything means that Apps that you know if we have an x percent reduction in entry You know what what usage share would they have counted for well again if Success were perfectly predictable only the worst apps would enter and it would be much less than x percent But if we're if app success is completely unpredictable and x percent reduction in entry is going to have a long run Expert hit reduction in the amount of usage that those apps account for okay now I'm not saying so just to be clear We're going to worry later on about how apps can be substitutable for one another and so that's not going to mean an x percent reduction In in consumer surplus, but still that that's a bit of intuition That's useful for us as we think about if there's an x percent reduction in entry What's the reduction in the share of usage accounted for by the new smaller Entry cohorts and if it's all the way to x then that's No, predictability a complete unpredictability Or nobody knows anything to borrow the phrase Okay, so a little bit about the data We have a quarter. So the data are aggregated to quarter or we put them together as quarterly data It's a quarterly panel over this period 15 to 19 We see all apps available each period a little with effort that i'll explain in a minute The way we're going to measure usage so we have two bytes at the usage apple One thing we observe in the data is the number of installations a categoric measure of the number of installations not a continuous measure Over a million or whatever We also observe the the change in the number of times an app has been rated. That's a nice continuous measure And I just say now we're going to focus on that Mainly use that measure but over on the right hand side We have a little picture showing the relationship between these two measures the categorical one and the continuous one And they're they're pretty monotonically related So we feel more or less comfortable using this change in the number of ratings But I mean to be clear we wish we had direct quantity data. We don't It's a problem studying this market So there are some details though So the biggest detail in data collection is that the way we we collect the list of operating apps Is we look at a big long list of apps Which is not all apps and we query the app store and ask it Hey, and by the way, when you query the app store for an app it says here a bunch of related apps So we use this process of querying and and adding all the suggested related apps to the list of apps We think are in existence and we do this actually for I think there was like a year before the data collection period Started to sort of get it to converge So we had all the apps that you know in existence and any new app Showing up there was literally a new app Right. So anyways, we start with a long list and add the related apps But there's still a problem Even though let's say we have all the apps as of the beginning of the sample period It's not the case that we are going to see every app right and when it enters There's what we call a delayed observation problem. We'll show you a little bit later You know in the we observed some fraction of apps in the first quarter in which they exist and some other fraction By the second quarter in which they exist and so forth But that's really important for us because if we're going to make some claim about entry falling toward the end of the sample period We don't want to mistake the fact that we haven't seen something that does exist For actual reduction in entry. So that's why this is sort of a boring issue But it's really important for us to not claim that the mere fact we don't see something yet means that entry has fallen I mean just in short what we're going to have to do is say things like well How many apps do we see that we're born? You know that are observed within one quarter of their birth or within two quarters of their birth to create an apples to apples time series Uh covering both the pre and post GDPR period There's also another problem with sometimes we don't observe an app like we observe it in quarter one and in quarter three but not in quarter two, but that one we just sort of punt and linearly interpolate the The cumulative number of rating measure between the two periods Okay, so those are the those are the skeletons in the data collection closet So we have a 4.1 million apps 31.4 million app quarter observations We do observe some app characteristics that we use we use we observe the category in which the app is You know, it's a is a game or whatever The download price for those that have a price or whether it has a download price We have some privacy variables that we use to characterize how intrusive apps are before and after And so so let's just look sort of at a first glimpse the number of apps available Just the number of apps available and it's pretty clear that the number of apps available has been rising Then falls it falls even a bit before the onset of GDPR There's a lot of app reduction. Okay, a lot of a lot of apps exited around GDPR I mean the thing about this picture, although it's a little bit dramatic It doesn't really distinguish any of the different effects that we think are interesting But still something maybe is is going on there's some kind of big extinction event. I like this dinosaur slide But this isn't the answer to the question because this could be a bunch of low-value apps exiting So this is not the answer. Okay. This is not the answer, but it's a fun picture Um, yeah, it's not the answer for two reasons again One is low value apps might exit and besides even if apps exit and other remaining apps are close substitutes Again, this wouldn't be the answer for welfare Okay, so let's uh, I don't know a couple of slides here that are that are like sort of boring But I think kind of important. There's sort of the way in which we deal with this delayed observation problem so, um We want to say what happens to entry over time And it would be a simple question if we just had clean data where we observed every app entering in each quarter But it's not quite it's not quite simple because of the delayed observation problem If we have an observed end of sample entrance, it looks like entry is falling even though maybe it's not So the idea is to have fair comparisons like apps entering this quarter That are also first observed in this quarter So n is like the number of apps and t is calendar time and v is vintage t minus v equals zero means born this quarter Right, so we could use this idea for for any any k any age of apps and just What are you know the the apps observed this quarter uh born first observe this quarter born k quarters ago And that gets around this sort of delayed observation kind of problem Anyways an example of that is apps born this quarter, right? So there's some noise here But and it's falling and it sort of continues to fall uh post gdpr Not quite the answer yet. This is just an illustration of the idea We can use a similar idea for our usage measures And why are we going to study usage? Well usage is our glimpse into welfare And I think uh, I want to be careful. I'm going to use the word usage I think some people give us a hard time about that. This is I mean, of course It's it's literally it's not literally like hours spent using the app this period So that you're right to give me a hard time in that sense It is simply our continuous measure of the extent to which the app is as well It's not quite installations, right? I showed you it was correlated or strongly related to installation So think of it as a proxy for the usual measure of the purchase of a durable good Which is you know installations, but it's yeah, it's a few steps removed from what you wish you had But you do what you can when you decide the question is interesting So qjt is quote usage again It's delta number of people rating the apps between this quarter and last So again, I'll call it usage, but I've been trying to be clear what it really is Um usage of app j and quarter t We'll define sjt as this usage measure divided by market size because that's going to be useful for Down the road for the logic stuff And you know s will also define an stv which aggregates together the jays that are born in vintage v Okay, collective usage during quarter t of apps born in vintage v But again, I've already kind of apologized extensively for the word usage But I don't want to explain it in every sentence. So I'm going to keep using the word in the talk Now if you look at this usage measure No question. I think the police has a question. Oh great So, uh, Joe excuse me that one important element in this world is discovery and you're using the number of apps that were discovered and Correct me if I'm wrong. Are you assuming that that's a process is constant with respect to the number of new apps And so I'm not I'm not sure what you mean by discovery though. Um, or are I mean, so I mean, I mean a consumer's Finding out about the existing of new apps So we are measured the number of apps in Existence is not consumer discovery of it although our usage measure So this may be what you're talking about the thing we're calling usage is based on changes in ratings of it Are changes in the number of times apps have been rated. So if that's what you mean, then yes, we're You can only you're only used to the extent that you're discovered Okay, maybe I should wait then. Okay. Yeah, no, I don't want please do come back. I want to make make sure we talk about it so these over on the right hand side this picture is just this usage measure for one two and three quarter old apps over time and again It's against it's against the birth quarter on the x-axis. Sorry. Is there a question? Yeah, I think Chuck has a comment Chuck. Do you want to chime in? Yeah, so You say that you know the value of apps is unpredictable And you know, I'm willing to to accept this for games and things like this But there are lots of apps which are just extensions of other services for instance your bank your airline and so on and I would think this would be a totally different category in terms of predictability your bank pretty well knows if it launches another How many people are going to use it and so on? Yeah, no, I so we're not we're going to be a little light on heterogeneity But what I will show you in a few slides I will show you evidence consistent with the idea of unpredictability in the following kind of sense If it's really true that app success is unpredictable Then not all that then reductions in entry should not only have reductions in aggregate usage But for example the share of apps born By quarter that eventually they reach some absolute threshold of high usage again up by our measure That should also fall and I'll show you that so I Entirely concede the point that you know any app from google That's already installed on your phone is probably predictably valuable And we're we're we're not going to be thinking about that kind of heterogeneity and the way we model it but I will show you evidence about the relationship between birth cohort timing and the propensity to achieve an absolutely high level of usage which And I will hint now that that's going to be very consistent with very little predictability Okay I think time for the next slide maybe so so let's let's now try to address the four descriptive questions Again, the first one being watch the effective gdpr on entry exit and number of apps available What happened to the privacy intrusiveness of the apps? What happened to app usefulness that is how much quote usage to post gdpr? At birth cohorts get relative to pre and what happened to app development costs again as inferred from average usage per app So first of all exit is is really dramatic. It's a large immediate effect Which by the way, we kind of predicted or I guess it's intuitive in the sense that there's no reason to exit until you have to comply And at at the time you decide to exit, you know, which apps are worthless and so it's that's the time to do it Um, so there's a huge a huge spike in an exit But again, this is also not not not going to be driving our interesting effects Now, of course, uh, I should also say that we very much, you know Wish as as with everyone who studies gdpr for an untreated part of the world So we could do diffs and diffs and we really can't find we tried pretty hard to find an untreated part of the world And couldn't find one so go ahead and apologize in advance for that We but we do look at a number of features of the of the exit and the change in entry that make us think it's gdpr Not something else So anyways, uh, but as I want just want to make sure there's no Although this picture is dramatic This is not our headline picture because this is a bunch of low value apps exiting doesn't really have a lot of effect on welfare Entry so this this is sort of more important Now we already had a hint about this just in what I described how we sort of apples to apples compare Usage measures over time But what we're now going to do is aggregate the data by Calendar time that is calendar quarter and vintage of birth I'm going to aggregate it together and estimate these a simple model that just a simple descriptive model that says entry Um, uh, sorry the number of apps that we see operating in calendar time t that were born at period v Is equal to some mu for age. So that's going to uh And then and then an eta I guess for for vintage And so these age effects are going to arise from depreciation How many apps are first observed t minus v quarters after birth? And the the aides are the vintage effects of interest after counting for the delayed observation is entry lower for those Those those time periods those v's that are after gdpr or I in some sense I should say after gdpr is known to be on the horizon and especially after gdpr So these this is a picture of those those aidas Where they're set to zero at the gdpr onset quarter And so you see it's pretty constant pretty constant at zero And then it falls for the post gdpr Vintage is again on the x-axis is the vintages of birth not calendar time And on average now we could you could quibble but let me just say the average for the post period is 47 percent the Below the pre period because it's you know, it's exponentiation of the law and coefficient and whatever blah blah blah You could say hey, it really seems to fall a lot at the end. So maybe it's not quite 47 I don't know what it is, but it's kind of a big negative number is the claim that we're making in the paper Now let's talk about some some just some some other evidence that is important to take into account What happened at other platforms? So here we have some app monster data on the apple platform Remember this paper is about the google platform So we have a google entry picture and an apple entry picture or time series in the same picture And what you do see is that there's there's there's some reduction happening at both platforms The apple one seems to go into effect or occur even earlier than at the but but The point of this picture is to say Whatever we think is happening doesn't seem to be happening only at google because some people and we actually talk about in the paper How google did various things to clean out their app store and so forth But it's since it's some things are happening at apple as well It doesn't seem as though it's all going to be driven or maybe any of it driven by google Google policing of its own its own environment Okay, full stop This is uh, this is an entry picture, but it's entry achieving high thresholds of success So think about this as being relevant to usage as well So what happens to the entry of ultimately successful apps? So we define a sort of arbitrary time to define these thresholds Of 10 000 eventual installations or 100 000 installations as measures of success And then we ask, you know, what happens to the entry that achieves this level of success And you can see that these pictures look a fair bit like those raw usage pictures Remember we said that entry fell by uh, 47 or sorry the entry picture entry fell by 47% on average in the post gdpr period And we're saying that ultimately successful entry fell by 40 and 44% Respectively that's not quite 47, but it's pretty close So it's consistent with it's nearly consistent with complete unpredictability The only I don't mean to push that too hard. I just it's very much easier Expositorily to talk about complete unpredictability, but I promise you I will talk about partial predictability before we're done And so I won't I won't be dishonest about that But it's it's really consistent with very little predictability. Oh, I see a hand Aaron Hey, um, so I'm trying to understand um with some of these apps, uh, if Uh developer might have um planned to have a new version of an app and maybe this developer releases a new app by name To signal that it's a much higher quality Now having to comply with this regulation that's going to change the timing of when I'm going to move on to the next version I was wondering if you can track developers and see if They're just changing the timing of when they're moving on to the next app Hmm. That's a good question. So the expert world expert on updating apps has been lied and I don't know if he's here with us today I don't I don't know. Uh, yeah, so just if I wanted to explore this the question would be which Updates show up as new entry as opposed to just showing up as updates because I think for us updates are just updates and not entry Um, but you're thinking gdpr could affect the timing of Yeah, I mean it's like You know, I again kind of depends on the type of app and you can imagine Some apps show up as a different name all together when it's an updated version of it for other apps You care about but having the same app Yeah, so I may maybe maybe my co-authors have the insight into this, but that's a good question. Thanks Hannah Nice to see you guys by the way this similar I guess similar question, but um, do you see the new apps but that are Created by developers that are serial developers because then it's not an update, but it's a new app But it's a you know the fifth Up that is doing a similar thing as previous apps or they have enough Enough experience and then they are delaying the issuing of a new app because of the gdpr Yeah, so I don't know we do know who the developers are and we do some stuff That's not in the talk that is in the in other versions of the slides and perhaps the paper about about developers as opposed to products entering and exit exiting but Yeah, this question. I think related to erin's. I'm not sure. I know the answer up the top of my head Thank you Okay, so I'm gonna kind of push forward because I know there'll be time for more discussion as we go So remember I told you entry fell I told you entry of ultimately successful apps fell nearly as much as entry Now if we want to think about kind of usefulness of apps, we already had a hint about that from the Reduction in the number of ultimately successful apps kind of the more direct way is just to ask what that usage measure Perhaps and so this is again this usage of one and three and two quarter old apps But if we all and this all falls after for vintage is born after gdpr Arguably falling a bit before But if we want to just pool the data together and do the similar exercise to what we did For entry we can do that And so what we're interested in are these rows the vintage effects of interest and again think about this as the Some measure of the usefulness of the birth cohorts born After gdpr after accounting for age is collective usage lower for both post gdpr Vintage is and so if we make these these pictures You know, indeed we have we have these these pictures falling Afterwards that's falling after the gdpr period and actually it's about 45 percent On average so very close to the reduction in entry. It's it's an almost but not a quite nobody knows anything environment There's some predictability, but not much and again predictability Which we'll talk about toward the end makes makes this a lot more work The shows that we went to grad school and stuff, but it doesn't add terribly much Well, actually that's not true. It's not that it's something important, but but don't feel bad that we won't get to it till the end Two additional descriptive facts that I think are interesting one is about average usage per app Excuse me average usage per app. So if you believe our story, you'd think that you need more usage In equilibrium after the after the after the rule remember because revenue has fallen per user Probably and the cost of compliance has risen. So we would expect apps to need more users per app And so that's really kind of what this what this measure is It's it's a measure of the average usage per app by birth cohort and it rises for the post gdpr period But you need more usage to be viable in a gdpr world than you did in a pre gdpr world One other thing I want to say is about uh privacy intrusion So we have these sort of measures of of how how intrusive the apps are And this is very important for us to acknowledge because you know at what is this paper about this paper is about a potential cost of gdpr Well gpr also has a potential benefit Right, especially its proponents and any sane person would say well Hey, the goal of the thing was to protect privacy So it seems quite, you know sensible just to take a look at this and just ask is there's something going on here And yeah, the privacy intrusiveness measures are falling. You could say they were already falling Maybe they continue to fall But it's at least possible that there's some privacy benefit arising from uh from gdpr, which we want to be uh forthright about I mean one thing it doesn't seem to do though You know, you could imagine a world in which the privacy benefits of the post gdpr apps Are so great that people flock to them and use them and notice we didn't see that in the usage measures But I mean you could also say all apps have to now be uh have to not now have those privacy benefits of gdpr So maybe you wouldn't expect flocking to the host gdpr cohorts But anyways the point of this slide is to say yeah, we know that the act is about protecting privacy And there's at least some evidence that it does So all this table does is to to reduce to a scalar each of those things that I said was either below or above before So I think I already told you the answer here. So I'm not maybe going to dwell on it But these are just the average post gdpr relative to pretty gpr effect effects rather in the pictures that we just saw in the last five slides or so Okay, so now I want to turn to welfare. I mean so I mean just maybe I should take stock for a second I mean I think what we have so far Is evidence that Again, the exit evidence is fun, but irrelevant. We have evidence that entry fell Let's say a lot, but the number we were banding about is 47 percent We see that the entry of ultimately successful apps fell by on the order of 40 to 44 percent We see that the average usage of the birth cohorts born afterwards fall about 45 percent So that's where we are so far kind of so it looks like maybe maybe something happened here Um But now we want to try to translate this into some kind of welfare measure So we'll focus on consumer surplus and the game plan here is uh is Well is as follows we're going to compare a pretty gdpr period in which there's whatever 1.1 million available apps To a hypothetical post gdpr equilibrium with 47 percent fewer apps So this is long run analysis. All right. This is not like the immediate effective gdpr. This is in the hypothetical long run Now it's easy ish if we have nobody knows anything because then what would you do to do this counterfactual? You would remove 47 percent of apps at random, right? So that's why I like that as a rhetorical device Let's let's try that first Now it's harder since that's not quite right Uh, and we're also, you know, we're going to need an entry model To think about the order in which apps entry if we're going to take seriously this this predictability stuff But again for a little while we can just think about removing 47 percent of apps at random Now we also need a way to translate somehow data into consumer surplus measures So we're going to estimate a logit model of entry, which we can then use to calculate Calculate consumer surplus. So in particular my favorite workhorse here would be the nested logit model that allows for some Substitutability across apps. We don't want to say that apps are all distinctly valuable Instead this this model has sort of two kinds of ways in which apps well first of all apps differ Simply because of their mean utility differing But they also have substitutability operating through the nested logit parameter Which we estimate the substitution parameter sigma which we estimate to be like 0.36 And by the way, we're also going to do a bunch of sensitivity analysis And a little later in the talk in the paper where we say, you know You tell me what you want that substitutability parameter to be and we'll translate the result of interest into Into some other some other number Okay, I should also mention I should also mention We don't we do have a price that is some apps are priced most apps are free and that's the purchase price not the in-app usage Price we're not actually going to use the price parameter We're not going to dollarize the or euro eyes the consumer surplus estimate We're just going to think about proportionate changes in the consumer surplus estimate So think about the consumer surplus formula, which I should have put on the slide But didn't has a one over alpha in it, but we'll be dividing one over alpha in the counterfactual world Over one over alpha in the you know in the in the status quo world. So so that's that's going to go away So we're making our lives sort of as easy as we can All right, so let's start with that intuitive remove 47 of apps at random So when you do that, you get a pretty big 34 reduction in consumer surplus Okay, so this picture on the right hand side does a two little bits of sort of robustness if you will And one is to vary sigma So we have sigma varying between point two five and point seven five The the baseline the dark line is the point three six one that we estimate Then we also have what fraction of apps being removed on the x axis. So this is kind of a Pardon the expression. We report you decide A slide if you think that you know, you want more substitutability like point seven five Then the reduction in consumer surplus would be on the order of like 15 percent with a 47 percent reduction in the number of Apps if you think 47 over states that you can take it over and and say 80 percent of the existing apps are available Anyway, so so there are a lot of numbers here, but I think that are a lot of potential numbers a continuum of them but I think that for a pretty wide range of entry reductions And substitution parameters. There's a a large ish effect of losing losing a bunch of apps Okay So, uh, the bottom of this slide though then again says but remember there is some predictability There is some predictability. For example, the 47 Entry reduction reduces those apps detaining the 10 000 threshold by 43 which is less than 47 And reduces apps attaining the 100 000 threshold by 40 So maybe we should worry a little bit about some predictability. So let's let's do that Here's a dense kind of a dense slide so The the idea is that uh, let's say that the entry occurs if you expect your quality to be high enough So that the marginal entrant would cover costs So how are we going to think about this? We have realized quality realized quality is the usage measure or it's basically the delta in the Loja associated with the usage measure for the for app j So we know how how good it turned out to be if you will We're going to pretend that we're going to come up with a forecast of how good it's going to be By taking its true value and adding noise to it And then pretending that the entrant takes this true value plus noise and that's what they know They know true value plus noise and then they decide whether to enter Now we parameterize this this this this unpredictability with a parameter kappa times a normal zero one variable I forget is that eta whatever that thing is Okay, so if kappa is zero, it's perfect predictability if kappa goes get it's bigger than zero. There's some unpredictability Okay, so kappa is a scaling parameter now the game plan here is to try to figure out what What kappa parameter what degree of unpredictability? would cause the simulation to deliver the Following fact the following facts I should say like what fraction if you had a 47 reduction in entry How much predictability would you need so that would deliver whatever we just said a 43 reduction in apps attaining The threshold of 10 10,000 or the whatever the threshold was for 100,000 So that's how we're going to identify or calibrate that parameter We know that there's some predictability because the the the You know the reduction in successful apps wasn't quite as big as the reduction in apps So, you know kind of conceptually try a kappa drop the four the bottom 40 per seven 40, excuse me Let me let me try to say this again in English So to have a kappa draw simulated You know use the the true deltas then draw some delta j is given the kappa you assumed now drop the 40 the bottom 47 of them to simulate a 47 reduction In entry and then check and see what's happened to the reduction in the apps attaining a certain quality threshold Choose the kappa that delivers the reduction in post gdpr Usage that we actually observe. Okay, that's that's what we do And so, you know good apps are more likely to enter we're going to continue using the the logic estimates we had before I think this slide is a little repetitive. I see a typo, but we end up we find We find a kappa That and maybe a more intuitive way to talk about kappa is in terms of the correlation between the expected quality and the Realized quality and a correlation on the order of point one two Is what rationalizes the data? Okay Point one two the correlation between expected quality of an app and the realized quality of an app So there's some predictability. It's not zero some some predictability Joe two minutes What's that? Oh, I have two minutes perfect. Okay. Oh, that's pretty good. All right, so What I want to say so actually predictability does matter, right? So instead of the 34 percent reduction the degree of predictability We have makes the reduction in cs about half as big something like 17 percent So it's still kind of a big ish number, right? So it does matter It was worth it to do that hard work last two slides. It does cut it, but it's still kind of a a big ish number Okay, two minutes. So I'm really coming in on time here GDPR Has had a big effect on the app market massive exit that's fun to look at but doesn't matter Dinosaurs and all but way more important a seemingly big slowdown in entry in new entry And again, my sort of axe to grind here innovation with unpredictability is a big deal That is, you know, if quality is unpredictable Then the welfare effects of either adding a bunch of products or subtracting a bunch of products again in entry Can be substantial and here with GDPR. We have substantial effects on consumers. I shouldn't have the word producers there That's for a different version of the slides apologies But big caveats big caveats. I mean, I've already sort of talked about, you know The skeletons in the data closet and all that but I think the bigger caveat Is the issue about what does this say about the advisability of GDPR? And I think it says it's it's it's at most half the story, right? There are benefits to GDPR that we don't really make any attempt to quantify here What this it's about is a potential cost to imposing a privacy regulation that has an effect on on entry And so this is at most part of the puzzle. We don't have an answer to the question is GDPR a terrible idea We just we just don't people are even assumed that we are saying that we're not saying that we're just saying on the on the on the innovation side it seems to have an effect on the on the Value of the choice set that would ensue in the long run under this policy versus not under this policy and I think with that I am I am done Thank you so much Joe. Our discussion is a question picker And I should stop sharing right? Yes. Yes So thanks a lot for having me. It's been a great pleasure to read this paper and To use five minutes to talk about it. So I'll start with things that I like I like that you guys are looking at the supply side effects, which I think is very important and interesting I like that you're being able to estimate welfare effects. I think that's great And it's especially done in a very clever way here. It's also an impressive data set Not only the the data that Joel talked most about which is the secondary data But also I like the survey that gives you some, you know, qualitative Insights on what the developers are actually doing and then the last thing I like is of course the authors These are one of the sort of the dream team That I that I can think of So I have a couple of, you know comments slash questions First with respect to the model when I say model I I mean the theoretical framework It's not entirely clear to me that revenues must fall So it could be that GDPR drives a selection of high types into premium version of apps A bit like in some of the GDPR papers or privacy Papers where there's some kind of trade-off between Giving up your privacy and and paying a price for an app And it could could be that revenues Can be driven by in-app purchases Or even by changing business models because of GDPR And I think you have some evidence That that apps are actually changing their The privacy intrusiveness and there might also be, you know, a different business model behind this And The other comment that I had was I was thinking whether You know, most likely it's not but maybe it's a problem that the price coefficient is only identified With premium apps that have a positive price And there's not a lot of them about six percent And for most most of what you do, I don't think it'll matter but I think where it might matter is because it might be consistent with defining that Which you didn't talk about Joel That you get a positive coefficient when you try to include the number of privacy sensitive permissions So so I think there's sort of some some kind of You know slice to be made in the in the data between Free and and paid for apps. I think that's what I'm trying to say And that might help you to address the elephant in the room, which is of course, you know Are you really capturing the full consumer surplus? And I know you Explicitly want to talk about sort of the cost of GDPR not so much about the benefits But I think could really help if you could expand a little bit on the on the short analysis that you have in a paper Or you didn't talk about here Where you try to incorporate a bit these Changes in in privacy sensitive permissions that That the apps have because I think this this gives you a lot of you know, could could give you a lot of traction in I'm not getting these types of comments that that I that I give you And I have a couple more minor things, but I'd rather leave the floor to other people have questions