 Live from Palo Alto, California, it's The Cube at Pier 2.0. Brought to you by the Pier 2.0 Foundations. Learn, connect and grow. Now here are your hosts, John Furrier and Jeff Frick. Welcome back. Jeff Frick here at the Pier 2.0 Foundation Event in Palo Alto, California. We're with The Cube. We go out to the events. We extract the signal from the noise. We get the smartest people in the room. We can find them. We invite them on The Cube to ask them the questions that you'd like to ask them. And I'm joined for our next segment here with Bill Norton, Chief Strategy Officer for IEX. Bill, welcome to The Cube. Hi. So Bill shows up with a book. Let's jump right out. We love when people bring books on The Cube. We've launched companies. We've launched books. We've launched all kinds of stuff. So we got there the Internet Peering Playbook written by the one and only Bill Norton. Yeah. Well, this started out actually as a whole series of white papers. When I first started working at Equinix back in the day, one of the things I wanted to do was find out what is this peering stuff? What are their exchange points? How does all that work? And not just from the high level perspective, but from an actual Internet service provider and network operator perspective. So I went to Internet conferences all over the world and I asked basic questions like that. So what is the definition of peering that you guys use? When you decide to peer with somebody, why would you say yes? And when you don't want to peer with somebody, why would you say no? And I would document what they said in the form of a white paper. And then I would walk the next people through the white paper that the previous people had contributed to. And they would say, well, you know, we think of peering a little bit differently. We tend to focus more on the quantifiable business case for peering or what have you. And I would make modifications to the white paper. And after about a hundred walk-throughs of the white paper at an Internet operations event, I would have in my hand a white paper that represented the community mindset on a previously undocumented Internet operations activity. So let's talk a little bit about the folks that aren't necessarily completely up to speed on networking, right? Because we get kind of a broad audience out there. What is kind of peering 101 for the person just coming off the street saying, tell me what is this and why is it important? Absolutely. So to answer that question, I guess I would start by saying that most people buy access to the Internet using a service called Internet Transit. Internet Transit is a port on the wall that says Internet this way. You simply send your traffic to your Internet service provider. Maybe it's Comcast or AT&T or somebody else. And you send all of your traffic out that way. And they take care of absolutely everything. I call that connecting to the edge of the Internet. And there's nothing wrong with that. 99.99% of all Internet attachments are at the edge. But if you look at the largest, most successful content companies or networks in the world, they connect to the core of the Internet, which are these Internet exchange points. The place where all of these network operators congregate and exchange traffic with one another. And in that way, the traffic that is going ultimately between my network and your network is going directly between our two networks and not up through your upstream ISP for which you pay. So that's the PACS, right? The Palo Alto Exchange, I think, is one of the... The PACS is a great example of an Internet exchange point. Absolutely. Okay, so peering is the direct connection from exchange point to exchange point. No, a direct connection between network operators. The network operators are the ones who exchange traffic. The Internet exchange point is the facilitator of that exchange. Okay, so it seems like a pretty simple concept. And yet you went out and asked a whole bunch of people to get their definition of what peering was and why it was important. Talk a little bit about kind of different points of views that you uncovered in your search. And what are some of the real important items that people really need to understand about peering? Well, peering used to be a process that was primarily used by the largest ISPs in the world. So AT&T customers need to get direct access to sprint customers who might need to access quest customers and so forth. So there's a mesh of these large ISPs in the world that exchange traffic with one another. And that used to be primarily the domain for Internet peering. But then the tier two ISPs, the smaller guys, found that they too could exchange traffic with one another and by doing so they bypass the tier one ISPs, at least for that traffic that can be directly exchanged. So I started studying the Internet as an ecosystem because the tier one ISPs acted fundamentally different from the tier two ISPs. The tier one ISPs are defined as ISPs that have access to the entire country solely through their free and reciprocal peering arrangements. So they don't have to pay transit fees to anyone to reach any destination within the country. The tier two ISPs are everybody else. Those are the guys who have to pay transit fees to somebody to reach some destination within the country. And therefore they're motivated to peer their traffic directly with each other to bypass the tier one ISPs, at least for that subset of traffic between their two customer bases. So I started finding that there are different species inside the ecosystems, tier one ISPs, tier two ISPs and content providers and that each act differently. But what's interesting is around the world, Jeff, every internet peering ecosystem I've studied has the exact same type of structure. The names might be different, you know, in Singapore, it might be SingTel and Pacific Internet and, you know, another ISP. We're in the U.S., it's tier one ISPs or Quest and level three and actually a century link now but AT&T and so forth. So different names of the specific companies but the context and every ecosystem around the world is very, very similar. From a technological point of view, yes, but we just had our last segment talking about how different kind of the opportunity for say the consumer internet options in the U.K. are so vastly different than they are here. So talk a little bit about how things evolved over time in terms of consolidation and expansion and choices for entry points into this ecosystem. Well, I think if you look at the ecosystem from, you know, a mile away, you see that the interdependencies between the different players is really, really quite strong. The tier one ISPs pair with each other, tier two ISPs pair with each other and the content companies buy transit from the tier two ISPs and the tier one ISPs. What you tend to see in the evolution of the ecosystem is that you do see a lot more choices coming into the ecosystem. The cable companies once were underneath the auspices of at-home. At-home would handle all the internet activities for all the cable companies, but they went bankrupt in 2001 and all the cable companies had to instantly become ISPs. Initially, all the cable companies across the United States would peer with each other for free and get access directly to each other's eyeballs and in that way, those guys had really quite good performance and they reduced their costs as well. And then the large-scale network-savvy content providers like Yahoo and Google started evolving into these peering beams. A new species entered the ecosystem and they started peering with the cable companies. So now all of a sudden, the quality of the experience that end customers had got fundamentally better because Yahoo was literally directly connected to the cable companies' networks and the amount of capacity and performance, the lower latency, the lower pack loss, all meant that end users got a much better experience. And where's the investment coming in when you've got kind of this interconnected play of mutually beneficial relationships, but still someone's got to invest in infrastructure, they've got to invest in pipes and switches and data centers, et cetera. Is there kind of a, you know, let's all do this together because it's greater or it'll be a one plus one plus three? Or, you know, how has the kind of forces of who's got the influence morphed back and forth as it's evolved from, you know, mom-on-pop cable operators to more powerful cable operators to content providers and then, of course, now it's the whole Netflix Comcast flash, if you will, that's really causing a lot of conversation around this topic. One of the things I find fascinating, Jeff, is if you look at these different species of players, they have really different motivations for peering and interconnecting. So if you look at the content companies, we talked to LinkedIn, we had LinkedIn give a great talk today about their motivation for peering and some of the automation they've done for peering. Every single content provider in the world that I've worked with and I've studied this, they will peer to improve the end user experience. Number one, not to save money, they might actually save a lot of money by doing it, but it's to improve the end user experience. And they're willing to invest an enormous amount of money in building out a global, large capacity network backbone solely so they have a better end user experience for the portal. They don't want someone to go to their competitor site. They know that if the glass, the visible part of your phone, if it doesn't fill up within a second or two, people will go to an alternative site. And this is where the content provider is talking about, the application provider. Absolutely. Okay. Universally, content providers want to peer to improve the end user experience. Speed, speed, speed, right? I mean, they got to get the stuff there quickly, as you said. Get that screen up there, get those ads. If they don't get those ads up there, people can't click on them, they lose money. Right, right. But the ISPs, they participate in peering for a different reason. As I said, the tier two ISPs, they peer with each other to reduce the amount of traffic, they have to pay AT&T to deliver. They can optimize their traffic directly. For them, it's not about the end user experience, it's about the cost savings of not having to have that transit meter spin really quickly. Right, right. So it's interesting, there's different motivations for each of the players in the ecosystem. The tier one ISPs, they already have access to the entire country's routing table for free. They peer with each other and get access to the entire routing table. So they're peering because they need that connectivity amongst themselves to get access to that ecosystem. It's not to save money, it's not to improve the end user performance. It's because they need the global connectivity, in particular the connectivity within that region. So then you throw in the explosion of content creation with all the apps and music and movies and everything else now coming down, which is probably much bigger, not to mention enterprise usage or internet of things and big data. And then you have an explosion of consumption devices on the other end. So huge bump up in demand. How are the carriers dealing with that in terms of spinning their little dial faster, as you said? Yeah, one of the things that was most interesting is I started working with YouTube way back when they first started getting into this stuff and the amount of traffic they were pushing was starting to grow exponentially. And what was fascinating to witness at that time, if you remember back then, if you would see video on the internet, you'd see a little postage stamp, a very jittery type of screen of video and it just was not a very good high quality experience. There's a graph in the book called the video suckiness curve. And way back then, the video suckiness curve showed the little postage stamp up there and it just didn't work very well. But as you started seeing Doxus being deployed in the last mile, that opened up the pipes very big in the last mile. When you started looking on the content creation side, it wasn't that long ago that digital cameras came out where you could take a large amount of video and then connect to your Mac and have it brought into iMovie or something like that and spew that out right to YouTube. So the flow from the end systems that created the content all the way up to YouTube and the massive distribution system that YouTube used to get that out there all the way to the broadband, that video internet ecosystem, as I call it, from end to end was finally activated. And yes, at that point you started seeing the transit meters spinning faster. You started seeing enormous amounts of content being distributed using CDNs and transit and peering and it was incredibly exciting and incredibly powerful for the peering ecosystem. Those pipes got very, very fat, very fast. And how's the finance? Well, YouTube's now financed by ads. Back then they were just running a lot of data, right? They didn't have a lot of revenue. There wasn't a lot of ads. They didn't have the big Google cash machine supporting them. So who was financing this massive increase in demand on the infrastructure? The one data point I remember hearing was that if Google didn't acquire YouTube within six months YouTube would be out of business. They would have simply run out of money. But instead they had that $1.65 billion buyout because of the user base and because of the growth and the value of that brand. What an Amiga deal steal. That was looking back in hindsight for $1.6 billion. It seemed pretty inexpensive compared to some of the transactions that we're getting today. Back then that was a big amount of traffic. For virtually no revenue, right? They're really buying the future. In my position, I was the co-founder and chief technical liaison for Equinix. My job was to travel to different conferences around the world to study the ecosystems, to study how things were working, to hear what people were saying about Equinix, to hear what people were saying about other companies, our competitors, and to find out what opportunities existed in the marketplace and how things were dynamically shifting. During the time we just spoke about during the video internet evolution, it was a phenomenally fun time to be in the ecosystem because the more traffic that goes across the interfering ecosystem, the more traffic that people can peer away for free, the more desirable the exchange points are because that's where you can offload your traffic for free with one another and that just sends everything up just higher and higher and higher. And of course the flip side of that is that also meant that people could charge less and less and less and we saw the price of internet transit go from $1,200 per megabit per second in 1998. Five years later it was $120 per megabit per second. Five years later it was $12 per megabit per second and today it's $1.20 per megabit per second. If you follow that trajectory in just a few more years it will be $0.12 per megabit per second. Essentially free, right? Essentially asymptotically approaching zero and Moore's law wins again. Exactly, right. So here we are at Pier 2.0 First time for this event. What's Pier 2.0 all about in the context of the story in which you just told and what does it really set the path for for the next several years? Yeah, well you know there are lots of forums around the internet operations space now. Every region around the world has their own internet operations forum. In North America it's NANAGA, a group that I used to chair called the North American Network Operators Group. In Europe it's a French acronym pronounced RIPE and that's where they have their coordination. In Asia there's a once a year conference called APRICOT and these are places where people come to talk about internet operations and the topics in general. Those do a really fine job. There's also a variety of peering forums like the global peering forum or the European peering forum Asia peering forum and these are places for peering coordinators to meet one another, schedule meetings to discuss whether it makes sense for you and I to connect our networks together and peer, if there's about equal value the answer is yes, if there isn't the answer is probably no. But Pier 2.0 is not that. We're not trying to be an internet operations forum we're not trying to be a peering forum this is an educational forum. This is a place where experienced peering people can share their experience and share their expertise with the folks who are just starting to get into the peering ecosystem and if we do this right we can grow the peering ecosystem well beyond the well-established Tier 2 ISPs, well beyond the well-established content companies and include enterprises like Hertz and Avis and Visa and Mastercard, all these guys ought to be peering directly amongst themselves and directly with their key trading partners and if we do this right we can grow this ecosystem in order to magnitude larger and everyone's going to benefit from that. So are there technological hurdles to that happening or is it more just kind of a point of view and just kind of the evolution as to is that kind of the natural state of the evolution of where it would go based on the steps that we've already taken. It's an education thing I mean if you're a content company you're most likely buying transit like I said it's a simple service it says internet this way you just send your traffic to AT&T or Sprint or whoever and your traffic is out there you may not even have a network engineering team to know enough to know that you could also connect to the core of the internet and be directly connected to Box or Dropbox or Microsoft or whoever so you need to understand that that opportunity exists and it doesn't mean you need to build out a large networking staff to do this anymore it can be done very quickly and very easily and that's one of the things that I'm sure you learned that IAX is involved in doing. Do you think that that path will go down kind of a trading partner set or more likely an industry set what do you think kind of the natural bundles of activity and connectivity will be as it evolves is it hurts plus Sheridan plus Alaska Airlines or is it Alaska plus United plus Southwest I mean how's this going to evolve do you think? So here's where I see things going today folks are buying transit and they're blending in some amount of peering so they can still buy transit from AT&T to reach the entire internet but you know what a lot of my traffic that I care about is coming from Box we're using Box for example our shared data store for our companies mission critical that we have direct access to Box so we're going to peer with Box even though it's probably shadow IT and nobody knows it's there but that's a different conversation for another show we're going to peer with Box we've certified so the point is that there's going to be a blend so you're going to buy transit but you're also going to peer some with key trading partners Andrea Sturm from the DKX once told me important traffic is peered and I love that phrase important traffic is peered that means if you like you can be directly connected to those key partners where you exchange a lot of traffic most likely if you're a content company you want to be directly connected to Google for Google ads if you're an enterprise you want to be directly connected to Microsoft and Dropbox and other key trading partners and it's going to be that kind of blend you're going to blend in transit, peering remote peering which IAX offers, distributed peering that IAX is doing CDN services and then the question is what is the optimal blend for the network application at hand and there are a variety of different views on that so the point is you've got a much broader toolkit now available than you did four or five years ago you did before so Bill we're getting the hook on time I want to give you the last word your book was a collection of notes from a tour before what's the next book going to be on what's kind of the next kind of big thing that you're looking at from kind of a macro point of view I'm not really looking at doing another book in theory there are a lot of interesting topics so coming forward for example this edition of the book has a focus on international peering something that hasn't been documented before so when I started working a lot across Africa, places like South Africa and Kenya and Ghana Nigeria and such I learned that there's actually a whole lot of gamesmanship about how you decide what country to go into and when you go into that country do you buy transit do you appear, do you sell transit what are the different models for doing that and it kind of fits into that blend future of projection as well so I might expand the international footprint focus and evolve this book to the 2015 edition well you got to keep adding to it so Bill thanks for coming on theCUBE Bill Norton Chief Strategy Officer for IAX I'm Jeff Frick with theCUBE we're at the peer 2.0 foundation event in Palo Alto California we'll be right back with our next guest after this short break thanks for watching