 So I'm going to be talking today about web traffic. I've been writing about web traffic for about six years now. Web traffic plays an important role in my new book. It's played an important role in other things and other research that I've done. But I have to be honest, for the past six years, every time I've sat down to write something about web traffic, I've had this nagging sense that there was something big that we had been missing. Particularly, I felt that we have not understood very well the dynamics of web traffic, how websites grow and change over time, how they add audience, how they lose audience. And so a couple of months ago, I received a present from Bill Tanser, who is the Vice President of Research at Hitwise, which is a large internet tracking firm. I know that some of you are familiar with. And Bill sent me daily data for the top 300 sites on the internet and the top 300 sites in a couple other categories. So what I've been doing for the past couple months with the assistance of Bruce Rogers, who's a colleague, an applied mathematician in Arizona State, is sorting through these data, trying to analyze what they show us about the growth and decline of websites, and ultimately trying to model a lot of the behavior that we see. And as I'll be getting to, I think a lot of this behavior is quite unusual. But so that's what I'm going to be talking about today. But before I get too deep into the modeling, before I talk too much about this data, I want to talk a little bit about why this matters. As I talk about in the book, an awful lot of our scholarship about the internet has been predicated to a greater or lesser degree on what I call the Robin Hood assumption. This assumption that what the internet is really doing is it's robbing from the audience rich, and it's giving to the audience poor. There's all kinds of permutations in this argument. Talk about narrow casting or point casting, talking about the fragmentation of the media environment that we hear from media elites, that we hear from presidential candidates, all fits into this mold. And in the academy, we've seen, and in the popular press, we've seen recently some more sophisticated, more subtle, frankly, more persuasive versions of this recently through folks like Cass Sunstein, Yochai Bankler, the popular press, many of you are familiar with claims about the long tail, which is sort of a revised version of the Robin Hood argument. So I do a lot of things in the book. But one of the things that I do is gather up all kinds of different sources of data and try to test whether this is actually what's going on. Try to figure out actually where people are going online and what sort of sources they're using. And I use a lot of different data sources. But probably the most important, and the one I'm going to be referring to here today, again, is HitWise. HitWise data. I want to take a moment just to talk a little bit more about where this data comes from and a little bit about what its strengths and weaknesses are. So HitWise, again, is a large internet tracking company originally out of Australia and now in the UK, Singapore, the United States, a variety of other countries. And HitWise provides anonymous aggregate ISP level data. They install software on the servers of internet service providers that monitor traffic, clean it, aggregate it, and ship it back to HitWise. And the general HitWise sample is about 10 million US households out of about 110 million nationwide. About 7 and 1 half million of these are in the ISPs. About 2 and 1 half million of these are in opt-in mega panels, which they use largely to gather things like demographics. And I'm going to be talking a lot about traffic here today. And what I mean by traffic, unless I stay otherwise, is really visits. And visits are defined as a request for a page or series of pages from a website with no more than 30 minutes of an activity. And I can talk more about this later if there are questions. But I'm quite convinced that for a lot of things that social scientists are interested in, visits are the best metric to use. It's hard to use page views, for example, without installing software on users' computer. And if you install software on users' computers, what you see is very different behavior, and particularly in a couple of areas. As you might expect, visits to adult content drop precipitously if you install software on their computer. Measures of unique users or impressions are often not very consistent. Time spent on site, again, very tough to get without installing software on computers. And I think that visits is certainly a much better metric than things that you sometimes see like monthly global audience reach, which really doesn't tell us a whole lot about how important a site is in the public's media diet. So one of the things that we struggle with, I think, in understanding traffic, is just the scale of the phenomenon that we're trying to get at here. And this is actually one attempt of mine to get at the scale. This is actually a map of US web traffic for January 2008. And this map is to scale. Each of these sites, the area of each of these sites is proportional to the number of visits each of them receive. And what we see when we look at this data is that the top five sites in the center, which is Google, Yahoo Mail, MySpace, Yahoo, and Windows Live Mail, all together, these five sites get about 21% of all US web visits. Taken together, the top 500 sites get about 51% to 52% of all traffic. So clearly, it's not just a Robin Hood store. So one of the things that we, of course, want to know, a lot of our claims about the internet are comparative claims. We want to know how this compares to the patterns that we're used to in traditional media. And I go through a number of different comparisons along these lines in the book. But one of the simplest comparisons is just to look at some category of online traffic versus what we are used to in newspapers, the form of traditional media that has been most impacted by the advent of the internet. So this is actually the audience distribution curves for US newspaper circulation and for US news and media traffic. Now, it may look like these lines are close together. But this is actually on a log scale. So they're really not. What we can see here is that the US news and media traffic forms an almost perfectly straight line on a log-log scale. It's a power law or Pareto distribution, or pretty close, as many of you are familiar with. And US newspaper circulation looks much more curvilinear. And again, what we've done here is we've just ranked these outlets by rank, from biggest to smallest. And when we actually add up the numbers, what we find is that the top dozen US news and media outlets online have about 30% of the total online audience. Whereas the top dozen newspapers only have about 20% of US newspaper circulation. If we look at this middle part of the curve, which is all ranked 12 down to about rank 500, collectively, this group of newspapers is the large majority of US print newspaper circulation, 73%. Whereas it's only about half of news and media traffic. And below about 600, we only have about 6% of newspaper circulation. You're pretty much out of newspapers by that point. Yet there are still thousands of online news outlets that collectively get a sizable audience, about 21% of the total traffic. So what's really going on is not just a Robin Hood story. What we see is that online audiences are getting both more and less concentrated. So there's more eyeballs here and more eyeballs here. And it's really the sites in the middle that are being squeezed, that are showing relative decline in their audience. So that's the general context. So I want to spend the rest of my time talking about what I think has been missing from our understanding of web traffic. And I think the biggest single thing that has been missing is that we've had almost no sense of system dynamics. We haven't understood at all the process by which sites gain and lose market share. So for example, we want to know how sites are likely to do over time. We want to know the odds that Google will still be the number one site a year from now. Many of you, certainly myself, want to be good social scientists. We want to quantify our uncertainty. And we also want to know not just how the big sites are going to do, but how the smaller sites are going to do. We want to know what are the odds that this shiny new site at rank 100 will jump to rank 50 at the end of a year's time. What are the odds that it'll go to rank 10? So partly our claims, our hopes, are really about individual site performance. We also really want to understand, again, how this audience distribution curve is likely to evolve over time. Is online concentration increasing? Or is it decreasing? Are we becoming more or less fragmented over time? Or are online audiences converging to some stable distribution? And even more important, as I think we're going to see, do different sized outlets behave differently? Do small sites online behave differently than big sites? So I talked a little bit more about the hit-wise data. Let me explain this new hit-wise data source. So what we're using here, and what I'll be using for the rest of the talk, is daily market share data. So the portion of the total visits that are accounted for by individual websites. This is three years worth of data from July of 2005 through the end of June, 2008. And this data includes the top 300 sites, the daily market share, for the top 300 sites in three categories. First of all, all the sites, the hit-wise tracks, roughly 800,000 on the internet on a typical day. All sites in hit-wise's news and media category, which the top 300 sites on a given day, account for about 80% of that categories traffic. And then the top 300 sites in hit-wise's politics category, which is sort of a grab bag of everything from blogs to campaign websites to online forums that deal with politics. I'm going to be talking mostly about traffic over the entire internet and traffic within this news and media category here today. So just to give you a broad sense of what this traffic looks like, this is graphed over these three years the daily market share of the top five sites. We can see a lot of things in this data. So you may see that we have MySpace, which has an incredible run up until the middle of the summer of 2007, at which time it apparently stops being quite as cool and starts losing market share. It drops down to third. We can also see the incredibly steady growth of Google during this time. Google just keeps growing and growing. Just keeps eating up more and more market share. So how should we think about this kind of data? How should we begin to analyze it? Well, I want to make a suggestion. And it's going to first probably seem like a diversion, but I promise you it's not. I want to talk about stock prices. I want to talk about our intuitions about stock markets and how our markets, how stock markets, work. So most of us know that stock prices are a continuous process. Stock prices change on a daily, hourly, even minute by minute basis. Any individual on a public market can buy or sell any stock. We know that at any given point, stock prices can go up or they can go down. Yet at the same time, as we're very familiar with at the moment, there are long-term trends for individual stocks and for the market as a whole. We talk about bull markets and bear markets. We know that stock markets have very strong size effects, that they tend to be quite top heavy. A lot of the capital is invested just in the largest stocks. And we know that large cap, mid cap, and penny stocks behave differently. The smaller a stock is, the more it tends to vary on a daily, monthly, weekly basis. This is why we don't recommend to our friends, for example, that they invest all their life savings in penny stocks. We know that some stocks move together or in opposite directions. We know that certain sectors of the market move together. We know that some stocks are negatively correlated. Oftentimes when you see a big crash, and we saw a couple of times during this fall, it was a big market crash, and the only stock that advanced was Campbell's canned soup, canned goods, a classic head. So we know that the movements of stocks are correlate. We know that if we look at the FTSE 100, if we look at the S&P 500, we know that the structure of the market tends to be quite stable over time. If we look at the portion of the total market in the top 50 stocks, for example, percentage of capital in those top 50 stocks tends to be pretty consistent over the long run. And we also know from the econometrics and from the financial mathematics literature that price changes are log normally distributed. That is if we take a whole bunch of stocks and take the log of their daily changes in price and plot them over time, what you get at any point is a nice, normal bell curve that just expands the farther out we go in time. This is a very important property that allows an awful lot of financial mathematics to work, and we're going to be relying on it here today. But take another look at this list. What I want to suggest to you is that every single item on this list is a property that we either know or expect or at least should expect to see in the movement of online audiences. We should expect to see in web traffic. We know that web traffic happens all the time. It's a continuous process. Any individual can visit almost any site on the internet. We know that at any given point a traffic share of a site can go up or down. We know that certain sites, as we've seen, show long term trends. We know that the general movement of internet traffic, even though it slowed in recent years, a general trend, is for more and more visits over time. We know that the movements of certain sites are correlate. Certain sites tend to get bigger or smaller together. And we're going to be looking to see whether or not levels of concentration are stable over time, and whether these price changes really are, whether these audience share changes really are, log normally distributed. So let's look, first, at the level of concentration. So one of the things, I've done this analysis with a number of different measures of concentration in this. I have used the Gini coefficient, probably the single most popular measure of concentration in the social sciences. And if we look at concentration over every single day of this three year period, what we see is that it seems relatively stable. We get a somewhat different picture if we look at over the entire internet, and it seems somewhat more highly concentrated than we see within the politics category. The news and media category. But again, we don't see a clear trend over time. And in fact, the series seems to be more or less mean reverting. So we really do see pretty stable levels of concentration over these three years. So one of the other things that we're curious about in this data is whether or not this really is how hard this structure is. Because we would expect in the stock market that the more volatile sites in the stock market, we would expect that the largest stocks would be the least volatile. Do we see the same kind of behavior here? In fact, we certainly do. So this chart is actually on the y-axis. We have the days where the site at rank x changes. And on the x-axis, we just look at all of these different ranks. So at the start, we'll see that if a site begins the day at rank 1, it has about a 10% chance of being at rank 1 the following day. The top dozen sites over the internet in this hit-wise data trade places only about 10% to 20% of the time. And they tend to trade places only with each other. Notice the difference between the site that's ranked 12, which is only changing less than 10% of the time, and the site that's ranked 15, which is changing almost 80% of the time. The site at that rank changes about 80% of the time on a typical day. So we see these strong discontinuities in the level of volatility. We see similar kinds of patterns when we look just at subsections of the web. So here we've put up news and media sites. So the volatility, again, seems to be pretty strongly associated with rank. The top news site only changes about 12% of the time. And yet as we go further and further up the rankings, we see less and less stability, to the point where if you're the news site at rank 300, you're changing pretty much every day. One of the things that we're also concerned with when we have this list of top 300 sites is leakage. And again, this is something you see in financial indices, for example. So the question is, we know that on any given day sites are going to change in terms of level of traffic they receive. They could go up. They could go down. And if they go down, it's possible that they could go down far enough that they drop off our index. And what we see, again, is a quite regular structure of volatility. There's very little leakage up till you get about to rank 150, sites that start off a day ranked 150 almost never leave the index. And yet it increases. So by the time you get to about rank 300, the site that's at rank 300, more than 50% of the time will disappear from the index the next day. And new sites will come on. About 93% of the sites that appear on the index having not been on it the day before are sites that we've seen before. So there's a bunch of sites that are just right near this 300 threshold that just are going back and forth just on the volatility of randomness on any given day. So here's where I think it gets really interesting. So please bear with me. This slide shows the daily variance in relative market share. So the portion of traffic that they gain or lose, depending on their size, are lined by the log of the rank. And what we see here is that sites that are at the very top of this hierarchy, at the very top of this period, are very stable. And as we go further and further down the rankings, we see that the amount that they jump from one day to the next, given their starting rank, gets bigger and bigger and bigger. So if we look at right here at sites that start the day at rank 150, they have enormously more variance in terms of how much they're going to be jumping around from one day to the next than sites that start at rank 10. So this is the behavior of individual sites. And this is the behavior of individual sites based on where they start on a given day. But here's the paradox in my title. This is similar data. But here we're not looking at the behavior of sites. Here we're looking at the values at every rank. So whereas before we wanted to know what happened to the site that started the day at rank 100, here we just want to know whatever the value it is of rank 100 at any given day. So what we can see here, first of all, is that the average change for the value of every rank is almost zero. You can see there's a tiny little blip at the start and at around 50. But generally, this line is almost ruler flat. So the structure of the online system is incredibly stable. And we can see, if we look at this curve, that there are three different areas. And if you actually do the math, you find three different phenomena going on. These lines on the outside are the solid line is one standard deviation calculated, just calculating it. And the dotted line is the inner 68%. So it's basically what we see here is that the tails are a little bit heavy. The top rank site and every day is yellow. And if it changes with the purple site, the purple site becomes yellow again. But notice the bandwidth here. Notice how consistent the bandwidth is over this period of time. So what we've done here is we've tried to model, tried to replicate as much of this movement as we can just using these simple Brownian motion models, just day after day after day of draws from our normal curve. And what we get when we do this is something that looks like this. Now, this data is smooth in a slightly different way, which is part of why it doesn't have the spikiness of the previous graph. But generally speaking, an awful lot of the movement that we see in traffic is captured using these techniques. And particularly the bandwidth is pretty well captured just by using these simple Brownian motion models. So what are the implications of this? Part of what we're trying to do here is to show what the implications of these daily growth rates are over the long term. So what we did, Bruce and I, is we actually went out and we did 10,000 simulations of 365 days worth of data. And we tried to figure out where sites ended up given where they had started. And so what we find when we do this for the news and media site data, for example, is that if a site starts out at rank one, it has about a 60% chance over the course of a year of still being at rank one. And if it does drop below rank one, it doesn't generally go far. If a site starts out the year at rank five, according to our model, it's got about a 40% chance of being at rank five a year later. And it's a lot easier for a site that starts out at rank five to go down rather than up. And generally speaking, the further we go down in the rankings, the more these curves spread out, the bigger the estimated probability distribution gets. And we can find similar kinds of results if we look at data over the entire web. Remember that discontinuity in our volatility graph? That's replicated right here in site 10. Site 10, over the entire internet, according to our model, has a big gulf between it and the entire rest of the web. It has to lose an awful lot proportionally of traffic in order to get there. And it's very rare in our simulations for it to do that. So about 90% of the time, if you start off the year at rank 10, that's where you're going to be a year later. Similarly, the top site is very unlikely to lose its position. And it's also very tough for sites that start out, far down in the rankings, to break into the top 10. Sites that start off at rank 20, again, they have a much easier time going down than they do going up. And similarly, by the time you get down to rank 50, you can end up anywhere below this top 10 sites over the course of the year. So how does this simulation actually map to real data? So one way we can test this is we have three years worth of data. So for every rank, what we did is we said, OK, we're going to look at the difference between day one and day 365 for rank one. And then we're going to look at the difference between day 365 and the value a year on. And so we do this three times for every rank. And that's what's plotted in those blue dots for every single rank. And what we see when we actually do this is that the model works strikingly well for sites that are ranked 50 or above. About 75% of the sites actually are captured within two standard deviations. Obviously, we'd like 95%. But for a model that has almost nothing in it, that's actually pretty remarkable. So there's an awful lot of the movement of these top sites that we're capturing just by varying how much they move on a daily basis by that ranking. Below 50, what we see is that our model is slightly biased upwards and that the variance is compressed a little bit. This is actually the first run. And this is a result of not adequately taking into account the fact that sites that leak off the index are biasing our results a little bit. Because we know that they drop down, but we don't really know how far they drop down. The latest models that I was working on, just as I ran out the door, correct for this. And they actually get about 85% of the data below 50 in this two standard deviations. So what is all of this mean? I've showed you some pretty pictures. I know that many of you don't really care that much about econometric models of the stock market. But the reason why I've gone through this, partly, of course, as I think it's important to get the numbers right, to get the data right, but also because I think that understanding these phenomena has enormous impacts for understanding of the web. So let me talk about these more generally, and let me give you a couple of specific examples where I think that this matters a great deal. I think the first thing that we can conclude from going through this exercise is that an awful lot of the system level behavior can be explained as a function of these stochastic daily changes. We've known for a long time that traffic on the internet was roughly power law distributed. What we have not had is any credible explanation for why that is. And I think that what we're doing here is we're at least kicking the can down the road to force us to look at micro-level explanations that can replicate what we see in the overall structure of the web. And really, I think it's important to understand this paradox that from the perspective of any individual site, there is high and extremely heteroscedastic variance. The smaller you are, the smaller your levels of traffic, the more that traffic is going to fluctuate on a daily, monthly, yearly basis. Whereas the big sites tend to be quite stable over time. And I think this helps us reconcile the fact that we see that Google remains the top site on the internet, even though we know that so many other areas of the web are in constant flux. But the fact that their web is in constant flux shouldn't distract us from the fact that the audience distribution is remarkably stable. And part of what these models are meant to do is they're meant to be a first step. They're meant to be a framework for future analysis. Any linear combination of variables, anything you can put in a regression analysis, we can put in these models. And we have a lot of things that I'll be talking about in a second that might help us predict stability over time. Because an awful lot of what we want to know about the web, for example, regulators, want to know how likely it is that Google is going to lose its position in order to craft appropriate policies. But let's think a little bit more about Google. So Google has about 70% of the US search market, is similarly dominant in other Western European countries. When you actually look at the data, what you find is that Google seems to be both the driver and the beneficiary of this churn. The greater the portion of traffic that a site gets from Google, even controlling for size, the more volatility we see in their numbers over time. What this means is that sites that get a greater portion of traffic from Google are going to vary an awful lot more. And that probably has important implications for policy as well. I think this behavior also helps us explain part of why Google has been so dominant in the online ad space. Because one thing about Google ads is they scale as you're growing and as you're declining. Whereas with traditional media outlets, it's very hard to develop an advertising base. And that generally takes a great deal of time. And that's time that you're likely not going to have in an online audience. I also think that there are profound implications here for the fate of local media. And particularly for newspapers. We've seen an awful lot of inks spilled on both sides of the Atlantic worrying about the fate of local media, local newspaper. And there's been a lot of talk about hopes that the web could save newspapers, could save these flagging institutions. And some of the problems with the web as a savior for newspapers are well known. The fact that you need about 30 online readers to replace a single lost print subscriber. And the fact that newspapers in general have lost an awful lot of their display advertising, their classified advertising. For a lot of newspapers, Craigslist is a very, very bad thing, because roughly for a typical midsize US paper, about 40% of their revenue came from classified advertising. But what you see in these numbers when you actually drill down is that in both the US and the UK, national papers have gained online at the expense of local and regional papers. And partly what's going on here is the fact that online shelf space is limited. Because the structure of online audience is so stable, what you really have is big media outlets playing musical chairs. And how much audience they get depends on which chair they happen to be sitting in. And the fact that this shelf space is so limited means that traditional staffing levels are going to be impossible to maintain unless these media institutions manage to break into the upper echelons of the web. That's going to be very hard for them to do. But the even bigger problem, from the perspective of newspapers, is online volatility. Newspapers are used to a stable subscription base. And often, the smaller newspaper was, the more stable its subscription base was. The internet reverses these traditional patterns. The smaller an online outlet is, the more in general it's going to vary over time. Newspapers know how to go small. That may be tough for them. That may be painful. What they don't know how to do is how to survive in an environment where their revenue could change on a yearly basis anywhere from 50% growth to 50% decline. That's very difficult to how do you manage, how do you staff an organization facing that level of volatility in revenue? So with that, I will conclude. Thank you very much. Very much, Matthew. I think that was a really, really exciting paper. A Spanish journalist noted recently that we all have to be economists now, that we all have to retrain as economists just to read the news in a financial climate. And I think Matthew has shown very well that now we have to be economists to understand the internet and the internet behavior as well. So we can all be killing two birds with one stone. I think actually that any of us who are involved in sort of researching and trying to understand life on the internet can see a lot of value in drawing parallels with stock markets and internet traffic. Just because of the kind of capacity in the internet to inject competition, where there wasn't competition before. I mean, I mostly look at government, governmental organizations, political organizations, and the extent to which these organizations have had to become used to the idea that it's not automatic that people are going to go to government, because I want to find something out to get information, to get advice, anything like that. Government organizations, they used to the idea that people would sort of look at the phone number of the dictionary and come along. And the idea that they can get information from whole ranges of different sources and do is a challenging idea for government organizations. It's also obviously a fantastic, challenging notion for newspapers as you pointed out. I mean, the fact that they are no longer institutionalized in the sense that they're being tried to pay because it looks smart or they're paid because you've always thought it would have parroted that idea is eroded. And also perhaps the way that people kind of spend their public attention in this market for public attention, the way people spend it has also changed in much sort of smaller chunks. I mean, if you buy a newspaper, then you're spending a big chunk of your attention all at once. You're appearing on the grass with circulation figures. It's all gone at once. In fact, there are so many diverse ways that people spend their attention in much smaller bits. Maybe looking at the same news panel that they always look at online, but then sort of skating off in the New York Times, perhaps you could look at what the headlines are or any other area of interest. So I can see a lot of value in that. And I thought it was very interesting, your conclusion, we kind of expect all that to lead to volatility, and yes, you were finding stability, or at least in some parts of the spectrum. Another thing that I thought was really good about this is the fact that Matthew's got user data, and he's not afraid to use it. I mean, I think we all tend to think, or at least I was thinking before I started looking at specializing in life on the internet, that this was such a fantastic opportunity to vast amounts of transactional data about the way people behave with what we've never had before. And it is, of course. It does generate huge amounts of transactional data. And Google have got huge amounts of transactional data, and quite a lot of other companies have as well. But as academic researchers, quite often haven't got any more data than we had before. And in some ways, in some ways, perhaps we've even got less data than we had before. I mean, if I think particularly of overtime data, or see, we've got overtime data here, that's another thing that we don't tend to have, because we tend to, by the time we thought that it would be a good idea to have a look to that five years ago. It's too late to look at that five years ago. Whereas with newspaper, we thought once the newspaper became electronic, it actually was possible to look in quite beta ways that track media attention over time, for example. Here, why is data isn't perfect? And I'm sure perhaps it would be very good to be explaining why it isn't perfect. But the point is, we're looking to generate trends and patterns and models here, and it doesn't actually have to be perfect. It's data, and we should be using it. So I think Matthew's really to be congratulated for that. I suppose I just kind of think about two key questions. I mean, one, you said we would be looking to predict this market, and so you just say, social scientists, we have a space to say that. Certainly, political scientists, we never ever say anything like that. But can we really predict that this market any better than we can predict stock markets? I mean, your assertion that we could predict stock markets is perhaps particularly a fantastic idea. But I mean, I suppose it is a really interesting question. There was to be a big crash. Can we visualize a big crash? Could there be a big crash around the corner? Just as the word stock market, I mean, and what would that look like, what would you survive, and what would it be? And I suppose the other point, also in terms of prediction, what about new players? I mean, that's something that we tend to think about the internet, that these massive new players are plugging in from nothing to billions of users. And I wasn't quite seeing that reflected in the passage here, and I thought it would be interesting to think about that. I mean, I suppose in a way, I suppose it must be difficult for the hit-wise data to capture this, to talk about market share, but sometimes it's difficult to know what market new players are in. I mean, YouTube, for example, Clipper, Second Life, these sites, which show us that, in fact, people had much more time than we thought they had, and we had kind of gone to Second Life, just had huge quantities of time on gaming sites, on Facebook, huge, silly trashes, and people's days, but we didn't know if they were available, which in a way might expect us to make it far, far less predictable. And I didn't see, for example, YouTube appearing in the shorts in your data, and we know what it's doing, but I suppose that would be another question we have. We're at the very near fair, but I don't know what we predict about them, because they've been with everything we've done to get to the market share. Any questions? Thank you.