 Today we have Jim Cowey from Renassess for years actually one of my problems, my personal problems in my life is that I constantly got asked by people in the press and otherwise is the internet down in Egypt, Russia, China, wherever and I would say and all of us would say I don't know, we can ask the six people that we know whether it's down and maybe they'll tell us where they won't and maybe it's their DSL problem and this was sort of a fundamental problem that seemed simple but it actually can be very hard to answer and this has become increasingly important in the past four years where large-scale outages of the internet have been used as modes of internet control particularly in the Arab Spring but in other places as well. So now thanks to Jim Cowey I usually have an answer for that and the answer is go look at Renassess's blog because they probably on the same day you asked the question answered it with actual data from traffic and from routing what parts of internet in any given country are up or down in Egypt. So Jim kindly agreed to come talk to us today to some degree about those specific issues of tracking internet outages but also more largely about the structure of autonomous systems and networks and provisioning of networks on internet in a whole and how that impacts questions of technical but also political, social, and economic control. So with that. Very good. Thanks Hal. So it's true we do attempt to tell people when the internet is substantially down in particular places we hope to get it on the same day. We've discovered a whole new sort of data source for this which is that when the internet goes out in a particular country reporters will start to call and so we say wow we got calls from the Associated Press and Bloomberg and they're all asking about Afghanistan then we know oh you know time to check the logs but so today I'm going to be talking with your permission about stuff that is down in the basement in the buried in the streets of the internet as opposed to where a lot of you are studying as I understand it which is control issues and political issues and application issues up top. Now it's not to say there aren't control and political issues down below but where we function mostly is under street level. We're studying how the internet is put together at the most fundamental level which is agreements among providers and the fibers and the physical resources that the internet runs on. We've been at this I'm a co-founder we've been at this for about 10 years now which seems like a long time. Studying the engineering obviously the performance but also the economics and the security when we can of the internet. The way we've always done this historically is we've connected to hundreds of internet service providers around the world and gotten them to share routing data with us that is they're literally giving us copies in real time of all of their routing traffic so that we understand how they reach everything on earth all the time and we mind that in recent years and but if anybody has questions just jump in and ask because there's no reason for this to be unidirectional. We then we augment that by taking a look at the structure of the relationships that are exposed in the pads for example if I need to get to a particular site in China I know all of the providers that serve that site I know all of the people that they buy from I know most of the people that they peer with I can try to trace that back to any place on earth and figure out what the path should have been if I tried to get from a server in Dayton Ohio to a server in Beijing. We're inferring continuously the business relationships the economics of connectivity from that data trying to figure out who is peering with each other whether it's settlement free or somebody is paying and when it's simply a transit relationship somebody is paying to carry traffic away and as you can imagine that evolves daily we we try to track the primary providers in every country we try to track everybody's customer list and we keep all of this data around forever so we can go back and do cool retrospectives and show how the internet diversity for example in an internet ecosystem like like Saudi Arabia or or Canada has changed over time we find that this this is the sort of thing that a lot of people need to know about network service providers obviously for business financial services industry wants to know the the fastest way to get trades executed around the planet governments are always interested any global enterprise these days that does business on multiple continents gets irked when the internet stops working and they have to dive down into the infrastructure to figure out why that is so I brought you a topic on geopolitics just because domestic politics is interesting but international politics to me is is is perhaps you know more so and then I had to stop you know geopolitics is supposed to have been killed by globalization right and globalization is the internet is geo is globalization's primary symbol so the internet actually told me that the internet had killed geopolitics so magically the internet is said to do all of these great things and you know it creates civil society where there was none it has completely you know torn down our understanding of how states should function you know we're all just individuals on facebook now obviously this is you know it's not even a complete list of all the great things that the internet magically does so obviously this is you know kind of bunk the infrastructure is no more flat than the rest of the world it's it's really ugly it was built out of lawyers and people trying to make money connecting to each other trying to squeeze money out of people in order to use their resources so you have tens of thousands of contracts written between consenting parties big and small all over the place sometimes they're in different places they speak different languages they have different culture and they're all trying to maximize profit they're all trying to squeak around under the gaze of the regulators the internet by definition crosses borders and so everything eventually gets messy and so everything that you see in in geopolitics everything you read about in the paper about spheres of influence and national interests and so forth has a counterpart in on the internet and how internet structure plays out and in terms of how disputes happen and are and are resolved and on the internet most often disputes are resolved by going around the dispute so what i'll do today just as a conversation starter and it is very quickly so we have lots of time for conversation is take you on a very brief internet tour of kind of the the ancient world from tangier to bishkek this is the part of the internet that we call the interesting part of the internet not that that north american internet isn't interesting not that european internet isn't interesting but they frankly are kind of boring everything kind of costs roughly the same everything's really cheap you can get connectivity anywhere you want you have six different providers to choose from unless you're a home broadband user in new hampshire say but that aside if you're a commercial user generally you have pretty good pretty good selection in this world you don't you almost always don't it's dominated by the old incumbents the old telephone companies they're often quite friendly with the governments that regulate them and in some cases own them and there's just a lot going on here you're going to see as we go through the the through the tour you'll see things that are are familiar to you from the newspaper grudges political relations spheres of influence that are developing a lot of references to things like energy pipelines and energy transport and highways because that's where you put fiber and that's what the internet at its core is actually built on so i always have to stop before i take my tour and and acknowledge my my patron saint of large networks in the eastern hemisphere so if you haven't looked up the travels of ibn batuta he wrote a book the 14th century about his incredible somewhat fictitious but probably mostly real travels he started as a lawyer in ten year he went all the way to china and he made it all the way back home again right in the middle of the black death and what he describes is familiar today as you look at modern internet he's talking about how information flowed along the 14th century networks of scholarship and religious education and commerce along the silk road and communication so this has all been done before so we'll just blow through these very quickly so you can see how different places experience different things as they try to get connected to the internet here we are in in the far west in the makhreb you've got marocco algeria tunisia and libya all served by these relatively short submarine cable runs mostly directly from europe these things are coming from from places like marseille or sicily and these are this is a best case this is this should be as good as it gets on the other end of these cables you get access to everybody who's serving europe you get very low latencies very low delays to to almost all the content you could ever want in europe the only thing that that perhaps throws a fly in the ointment here is that these are mostly run as i said by national incumbents tunisia has more diversity marocco and algeria and certainly libya relatively less diversity but still it's made up for to some extent by the fact that they have so much choice available at the cable landing and so their their their prices are relatively low the amount of freedom they have to choose how they're going to transport internet traffic is fairly high only europe very very possibly yes although they are i mean these are successive stages in some sense on the same cables and so it's possible that that on the same cable you can get connectivity from one place to the other but at a logical level it's more likely that yes the service provider you buy from is probably going to route your traffic back some place in europe it's also true as we'll see later that just because you have two different people in one of these countries doesn't mean they can actually send each other traffic if one of them is a customer of the big incumbent somebody else is a customer of his small competitor they may not actually be connected in the country and their packets may actually go back to frankfort or someplace or paris and and return to the country and you can imagine what kind of competitive pressure that places not very much if you if we move down the coast a little bit egypt of course was in the news this year for internet structure and not in a good way egypt has everything it takes to be an internet transit hub for this entire region it really has huge amounts going for it it's the natural traversal point for almost all of the fiber optics that are going from the Mediterranean through to the indian ocean and beyond almost all of this is coming ashore at alexandria maybe doing an overland traversal plunking into the sues and off it goes and so this is the physical pinch point for until now the majority of transit to the entire middle east and certainly now large parts of eastern africa the problem here has been although there was an effort by the regulators to liberalize this telecom egypt retained a very central role in terms of maintaining the physical infrastructure and it was actually possible during the troubles to to turn off all the internet service providers bar one or two just by cutting power at a single facility so you have logical diversity in the sense that there are a lot of service providers serving egypt but in terms of the of how regulation and physical infrastructure channels that traffic it turned out that there was a lot less diversity than we thought i was shocked when i saw that that most of egypt was taken off but there it is a different solution to the problem if we slide around into the eastern med very quickly israel has its own cables that come in that are not the same cables used by everybody else in the region so sometimes you'll hear you know strange complaints like we had a physical cable fault on simi we three or four and lots of countries around the gulf were thrown out for weeks and why didn't israel go down well it's because israel has their own cables they have landings on cables that are physically diverse from those that serve other countries um the palestinian territories are locked in and so by law i think they have to buy through israeli physical infrastructure but logically we now see them buying a lot more in the last year from european carriers like level three or deutche telecom um jordan is well well set up because they have submarine cable landings of their own at akaba so they have at least two uh connections to the big flag cable that goes around the world um they also have terrestrial connectivity to saudi and and and and to turkey to the north through syria okay so this is again not too bad there's there's there's reasonable connectivity here for those who want it and so prices for bandwidth are are higher than they are back in the maghrib but they're still modest we work our way up the coast it gets to be worse okay and you can see here the effects of geography and politics um reducing the choices that consumers have in country reducing the choices that enterprises have if they want to get to the internet lebanon was in a terrible position because uh they didn't have access to the big cables that landed just down the coast in egypt all they had was uh some old-fashioned uh relatively small connectivity coming from cyprus and then you could see that that was the case because a lot of the autonomous systems in the lebanese internet are just are buying connectivity from satellite providers so they're buying from satgate they're buying from providers where the next hop if you trace into lebanon uh is in germany somewhere or or in czechoslovakia czech republic um this all changed in 2010 about a year ago they finally got the big imii cable and suddenly they had a terabit allotted to them of capacity it's an incredible uh flood of potential internet landing at their doorstep uh but there were ongoing political problems between the people who had the cable landing and the government questions about how it was going to be tariffed i mean if you allow this flood of internet into your country the price structure is going to collapse which it is going to uh it took almost a full year just to get hypothetical tariffs published for how much it would cost to get one of these big pipe connections that they had never had access to before um and i think even just now we've seen providers in lebanon begin to take advantage of this wonderful new cable landing and as sure as the sun rises uh we've started to see their satellite connections turn off because nobody wants to keep an intel set satellite internet earth station working for one day longer than you have to because it's horrendously expensive um this is exactly the same sort of shift that we've seen in east africa with the arrival of the c con cable um syria is if any if it's possible and even in a worse situation they had the same sort of uh small pipe connectivity to cyprus and lebanon uh and i'll show in a minute how syria is turning to terrestrial routes to make up for the fact that they can't get adequate submarine connectivity um you can't talk about these guys without going a little farther north and talking about turkey so turkey as you know is positioned to become a really influential country in the region um politically and and sort of as a as a trend setter if you will and internet it's extremely true uh turkey has the bridge to europe the physical bridge um they are on all of the major fiber pads to europe which follow the old roman roads through bulgaria uh they have connectivity through greece um who gets it from italy all of this these uh this high bandwidth fiber from europe sort of flows down the old roads and into istanbul and an and turkey has realized that they are actually a uh potentially a major exporter of internet this is this could be a new industry for them and so not only turk telecom the incumbent but also their competitors uh have started building parallel projects to build fiber to all of their neighbors so imagine what happens in a world where um before there was no connectivity and now uh uh perhaps along the lines of some of the old railroads that were built by the germans you know before world war one um there's now turkish fiber going down into the hejaz to serve the saudi peninsula or headed across into iraq or even headed to georgia so turkey begins to establish a sphere of influence where they export internet service to a lot of neighbors that they would like to be good friends with in many cases they are because the engineering's already done this is really rugged territory in eastern turkey if you once you've got a pipeline cited um usually they have fiber buried alongside them anyway because they want to have uh control and communications between the various stations along the route so in some sense you almost get internet for free as well as the pipelines go through which is this this connection that recurs about the uh energy transport being very much linked to information transport in this region um ote is a huge greek company that owns two the two big fiber networks coming down redundantly one up italy one through hungry and uh in that sense yes they are because they offer turk telecom enormous amounts of bandwidth um commercial relationships are the best friendship right um so where does all that bandwidth go ultimately well the gulf states have been looking for alternatives for a long time the gulf states all around the saudi the arabian peninsula have been unhappy with uh the availability of submarine links they always get cut people drag anchors across them you lose your internet for weeks it's just untenable um so these guys have started looking for alternatives and egypt really brought it into focus for them virtually overnight they were talking about how do we get around the egyptian pinch point these people could deny us internet with a single accidental or deliberate break so um suddenly overnight consortium spring up to build overland paths that can be redundant with these submarine paths um perhaps a topic for for questions or another day we've worked closely with the telecoms regulator in Bahrain uh which is a place where um things were extremely expensive they continue to be very expensive but um this is sort of evidence that pushing for open competition and inviting creative alternatives for second and third uh providers uh can really lower prices for for consumers and businesses and uh i think what everybody here has on their mind perhaps Bahrain most of all is um when the energy economy turns here there has to be something to follow what follows is the information economy perhaps this is where your college graduates can go they have to get it jobs you need the internet for that it has to be fast it has to be reliable it has to be cheap and so this has become a major strategic question for all of these countries how do we get the internet cheaper faster more secure without going through people we don't like on that note these are some of the the ways people have not only hypothesized about getting internet into the gulf but actually started to build these um in red there you see the jadi link which is a uh consortium based path it's actually lit now it's working in passing traffic um Saudi telecom uh Jordanian telecom uh Syrian telecom and Turk telecom got together and built this route from Jeddah to Amman to Damascus to Istanbul it's still up through all of the troubles in Syria um if anything the Syrian government has been very um helpful and and willing to demonstrate that this is business as usual these pads are maintained and the service is lit uh you have the competitors in each of these countries the competitors to the incumbents building a parallel route the RCN route which goes from the submarine cable landing at Fujairah I should have a laser pointer to Riyadh to Amman to Tartus to Istanbul slightly different routes different companies in in in each country except for in Amman and in Syria no it's a no that would be cool wouldn't it uh now that's an acronym collision um one of the really interesting ones here is EPEG that's the the brown line up top is EPEG that's the European Persian Express gateway which has not been built but is really cool it starts in Amman where Amantel is talking about building uh an entire internet exchange in Tallahouse it jumps across the Strait of Hormuz just before the Strait to Jask in Iran it runs on Iran's terrestrial infrastructure all the way to the north crosses the border into Azerbaijan gets carried into Russia crosses the border to Ukraine at Rostov-on-Don and then cable and wireless takes it to Frankfurt that sounds like a crazy route but actually if you look it's a great circle route and could be very low latency and again it goes through different regions what you're looking for here if you're if you're at FIMA company in the Gulf and I want reliable connectivity I want to buy from consortiums or or from companies that can offer me diversity and these pads give me diversity we'll talk in a minute just briefly about Iraq which also is is emerging from from war and talking about becoming an internet provider Iraq is very central right if you look at the geography of Iraq you couldn't pick a better place to build a regional backbone in the south they have submarine cable landings for major cables at Basra if you look west they have overland through the desert to Jordan and Saudi Arabia and Syria if you go east you can hit Iran if you go north you can hit Turkey it's if there were a government there that wanted to make this a priority and the geography is actually pretty sweet at the moment they have as you expect sort of fundamental problems with keeping electrical power grid operations and so forth but you know over time these things will work out if you look at Iran of course we always since 2009 we've had our eye on Iran and their control of the internet where does Iran's internet actually come from you never think about you know where does internet come from some of it does come from the Gulf over the submarine cables that everybody else uses but a lot of it these days actually comes from the north it comes from Russia Russia through Azerbaijan through the Baku internet exchange is exporting internet transit to Iran which gives them again I wrote a blog about this they have to balance they have to maximize the diversity they don't want to get cut off by a cable cut in the Gulf they want to have a backup route through a friendly party to the north so it's just common sense Iran has written I guess as recently as May about the possibility of going it alone they you know they'd like to disconnect from the internet and run their own internet Iran does have a very mature internal domestic internet with a lot of richness and a huge number of autonomous systems so I guess we'll leave that as an exercise to see whether they can ever accomplish that but it's interesting because they actually use the internet regionally geopolitically to get influence over neighbors so Iran is a net exporter of internet transit because they sell to to to Iraqi Kurdistan for example if you're if you are in Sulemenia you can buy Iranian internet transit if you're in Herat in Afghanistan because the big ring of fiber that surrounds Afghanistan has has been somewhat dysfunctional for a long time in Herat one of your only choices for internet service may in fact be going across the border into Iran and Iran is happy to provide that service this is one of the the in addition to energy exports and financial support for that region this is one of the ways that they can be a good neighbor and and have influence over what develops there they maintain communications my next slide perfect I did not I did not schedule that so yeah go north so now you're we started where it was easy now it's really not easy okay now we are in a very sticky region politically in terms of energy distribution in terms of the politics of your neighbor to the north there is a cable system in in Georgia called the the Caucasus cable system which runs east to west you can you pick up service in Baku there on the right go across Azerbaijan you cross the border with the the the pipeline that goes past Tbilisi you head over to the Turkish border and you're off to Turkey if you go north to the coast you can reach I think it's putti where there are submarine cables ready to take your traffic across the black sea to Romania or to to Varna and Bulgaria big pipes they are alongside energy pipelines which are perhaps risky places to run transit in a in a dangerous place but that's what provides the east west transit that people want to use if they don't want to go south to Iran or go north to Russia so Azeri providers have been carrying a lot of the weight here they are providing everybody who's on the other side of the Caspian Sea eventually there will be pipelines across the Caspian Sea once they kind of get the international ownership of the sea sorted out and there will be fiber routes and those fiber routes are going to need backhaul to Europe and the Caucasus will provide it you may have read last year about a big internet outage in Armenia where a grandmother cut a cable in Georgia and this is her she has her own Facebook page she uh yeah a large part of this the Armenian internet just went out went dark and it was because up here just north just it was just west of Tbilisi she as the story goes was foraging for fire wood and happened to come across a piece of fiber which theoretically was buried six feet deep beside the Georgian national railway and she sliced it and she turned Armenia off so she you know it's interesting because she actually says she didn't do it she says she doesn't know what they're talking about she didn't wasn't anywhere near there but yeah she this is you know she she's the most powerful woman no I think somebody did this on her behalf but it makes you think incidents like this make you think if these information routes become incredibly important for people who are trying to circumvent let's say transit routes through Russia accidents happen if you cross into the transcaspian region and go to central Asia you're way up in the in the beyond these are classically places where they're getting a lot of their transit still from the Russian providers but there are also paths opening to China Uzbekistan was actually one of the people who stepped in when Afghanistan lost their links to Pakistan the the Afghan government for a time was buying the majority of their transit from Uzbekistan through that little pass there and it was all going to Russia Turkmenistan again there's huge amounts of energy exploration around the Caspian Sea Turkmenistan is in the in the core of this they've built new they are building new pipelines and I think railway links to Iran every time you do that somebody's gonna lay fiber because it doesn't cost very much extra to do that and so new paths will open up and we will see them emerge Russia really does have a huge amount of regional influence we actually took a measurement here from our routing table studies and said what percentage of each country's network providers are on net meaning somewhere upstream there's a Russian provider so as you'd expect is 90 percent all through Kazakhstan, Kyrgyzstan, Azerbaijan, Belarus, Uzbekistan Tajikistan Turkmenistan and Russia itself and then there's a tale Armenia has less why because Armenia buys a lot of transit from the Turks Latvia has less why because they're facing Europe and they can buy some of their own stuff directly Iran Georgia Moldova Ukraine all buying but on the border so they have other choices and and then through the Iranians of course Iraq and Afghanistan actually we're getting a significant portion of their internet traffic through the Russians so this is a this is a very real sphere that they have worked pretty hard to build it's it's economically quite important it may I'm not sure if that's an accidental omission or if it's just because they they simply went west if you if you can go west you should almost all of the transit that you see represented here is backhauled to just a few European cities like London or Frankfurt so if you can go there yourself over your own territory you shouldn't bother so real quick and then we can go with with open session questions as you've seen there are really obvious ways that energy security ties in with information transport so most people terrestrial at least don't set out to build fiber systems where it's expensive and they're breaking new ground usually they will follow existing pathways where whether that's power lines pipelines inside highways unless you have absolutely no choice submarine fibers the way that almost everybody on earth gets their long haul connectivity is famously safe why well it's you know geopolitically it's interesting it goes through international waters where it's hard to find let alone break but then if somebody does manage to break it well it takes can take up to weeks to get a ship out to the middle of the ocean and fix it these terrestrial fibers that we've described are crossing national borders and that automatically gives you risk political risk over who's going to control each station along the fiber would you buy into a cable that was running through four countries none of which were really your friends and in each country it was the incumbent the least responsive provider that was providing service it still may be worth it because they may give you a good price it may be the lower latency price the terrestrial routes out of the gulf are certainly going to be much faster than the submarine routes that have to go all the way around the arabian peninsula to get back to europe everybody's making these trade-offs all the time at what extent is the political risk a denial of service or a slow roll of service as opposed to the local security folks snooping in so local security guys dipping in is generally a risk for the local people right so access networks here we're mostly looking at i just want to cross your country i want to get my youtube videos and i want to stream them through your country as rapidly as possible so the really the as you say the risk is denial of service thing could break can simply be turned off we could say you know we're renegotiating the deal i think we'll try this again that has not happened and i think it's interesting to look at the the game theory of a multi-nation cable system because if you turn it off everybody loses revenue you can see here that that several of these countries are are hard at work continuing work that they've done in other arenas in the past so turkey emerges as a vastly important connector russia continues to be a vastly important connector iran and to some extent saudi arabia are both going to be regional hubs and possibly others in that space like the emirates or oman and of course way out in the end we didn't talk because i mercifully stopped the tour but china is sitting at the other end and also has significant interests in the internet paths so in all these ways there's a lot of a lot of economics you didn't mention india in this context i did not and even better i didn't mention pakistan it was skillful of me i thought but there is of course that there is at the karakoram highway that links pakistan with china so it's supposed to be four tanks wide there's a new chinese base at at qadar on the on the coast near the street of hormuz all of these things will contribute to the building of of international relations probably internet connectivity between pakistan and china over time which would be great because otherwise to get from pakistan to china takes a long time you have to go typically all the way around on fiber through the streets of malacca and all the way up the coast and back in so from the standpoint of you know the internet should get better the internet should get cheaper the infrastructure should get more reliable all the links should get shorter that's that's great the more connectivity is seldom a bad thing still didn't talk about india um india is well served by submarine cables very richly so they've been one of the original stops on both coasts on many major cables and so india doesn't have to worry a great deal about its international connectivity they don't have a lot of people that they seriously need to talk to to their north although shorter routes to china would be better um india has a as you know a great it industry and actually one of the the largest tier one providers in the world tata is an indian company projecting internet transit east and west from from from from india so it's interesting india is in some sense well connected they are more like in pricing and and in connectivity they are more like a european or a or a north american scenario in these other places i've described it's sometimes hard to understand how big the gap is it can be a hundred times difference in the price you can pay one euro to transit one megabit of service for a month if you're in frankfurt and it can cost a hundred or hundreds of euros if you are in the gulf or if you're in sub-saharan africa you have these massive differences in price and there are companies that have vested interests in in making money from that spread so every time you get more connectivity that spread will drop and the living standards for people on the internet and hard to reach places will go up but it will also kill a lot of companies uh inevitably so everything is about politics and money and just by geography and how much of that is kind of political and economic manipulation is that easy to say with the exception of certain really expensive cables to lay that have to be paid back and therefore they're always going to be expensive to be on it's almost never the cables it's almost never the distance it is almost always local politics and local competition in particular it has to do with who owns the landing station for the cable in some countries the incumbent is the natural person to own the landing station and if the incumbent owns the landing station it means that no competitive provider necessarily can get access directly to the international carriers they might like to go to the landing station and say i want to buy you know a gig of transit from level three or from Singapore telecom or or Deutsche telecom um but they can't get it they say no sorry you've got to go through us we're reselling we have the license on the landing station for 20 years and we're going to resell that transit to you at the market price which was approved by the regulator and that's the story there are other places like Bahrain where they have made a very concerted effort to have more landing stations on more diverse cables and to put them all in different people's hands so that you actually do get some choice it's always a private almost always yes because you have well you have to have somebody to operate the landing station which is a technical operation um and typically those franchises are granted for a period of years and they can include exclusivity and under those circumstances prices don't tend to drop um there's there are examples in the world where for example flag lands twice in Aqaba and so Jordan telecom has one landing station and one of the competitors I think it's VTEL operates the other one and yet when we look at it we never see uh measurements go through the second landing station whenever we try to go to Jordanian targets uh to reach them with a trace or something or a ping uh the traces always go through the the primary and we don't see VTEL on the path so why is that we don't know it's there are mysteries like this that remain where I was going with the question was um I want to get a sense of you the degree to which the governments as well as the businesses but especially the governments recognize the internet as a rising tide that lifts all economic votes right and and work not to protect the incumbents whatever they may be and their parochial interests in a really rather narrow business right that they operated the expense of the rest of the economy I think it's hard to talk about governments all through this region as having monolithic intent or strategy in many cases there are internal uh divisions where one group may be interested in promoting security which is much easier if you have a single strong incumbent another group might be interested in promoting competition and reducing prices for consumers which happens better if you have a distribution of control and so they have different strategic interests and often in some of these countries you'll see that play out as a matter of domestic politics how is that balance going to tip in the abstract I think everybody recognizes the value of information in the internet now uh concretely uh you know often the the the major internet providers in a country are some of the largest companies in the country and so getting them to change direction or strategy uh is is a maybe a frightening proposal yeah a few years ago then we know that Russia cut the natural gas pipelines towards uh Ukraine I think it's not possible if Russia has quarrels with the companies that depending on the infrastructure of the internet is Russia possible to cut off those services yes and any fiber that is you know in any country is subject to the laws of that country and can be turned off gas pipelines you know there aren't as many of them as there are fibers that carry bits and so they are more vulnerable to disconnection because if I know that you're going to just route around me I may not bother that's why people start to look at this strategically and they say well so I'm I need you know I need three fast paths between London and Beijing and so I need them to be fast and reliable and cheap that's impossible so maybe I'll take one of the main submarine routes out of the London internet exchange go all the way around through Egypt through the Suez through the Straits of Malacca up around and into Beijing but then maybe I'll take a second route and I'll get one that goes through Russian territory it goes along the old the Siberian railway and and heads right out and that'll be lower latency but it'll be slightly higher price that's the trade off then they may say well you know anything could happen to those two I need a third one and so the third one today might be I'll follow the Roman roads to Istanbul I'll ride across Turkey on either Turk telecoms or Turk cells fiber backbone I'll hand off to my partner in Georgia I'll ride through the Caucasus I'll reach Baku who knows what happens from there you know maybe maybe then I'm happy with my Russian path maybe in the future something else develops to the south it's this continuous game and companies really do play it when they have the information trying to figure out what risk are they exposed to how do I hedge that risk appropriately because it used to be the case in especially in the Gulf states that you could have a single fiber would break somewhere off of Egypt and it would take you out for a week or two weeks the whole country and people recognize that that was not really consistent with with economic development or with an IT industry that functions or with banks or with anything these days in Korea have any role to play in this region or are they just little self-contained places Japan hosts for example NTT so one of the big tier ones that can offer you service anywhere in the world but geographically Japan Korea are pretty much at the end end of the line from that side and don't wait nobody would send traffic east from from the Middle East through Japan I've seen when there were outages in the in the Mediterranean I've seen Gulf states send traffic over satellite to the Hong Kong teleport so you're not but not terrestrial well mostly on submarine I mean and typically we call those the wrong way around the world routes so I'd rather get the short route to Europe and if I can't get it well I will go the long way but you're talking about hundreds of milliseconds so try doing voice try doing high latency or low latency trading you know over a link like that and just does it work a second question when you're talking about the Gulf states going left line for a week doesn't that really mean switching to satellite or some other high-cost provider for the week at a greatly increased cost for some of them it does so some some autonomous systems will retain connectivity on contract for for cases just like that so a bank for example might have a satellite link that they could use somebody like the African Development Bank in Tunisia has satellite connectivity that they maintain just in case something goes wrong with with Tunisia's actual connectivity but most people these days look at the internet and and by people I mean corporations will look at the internet and say wow I can get a cost savings orders of magnitude over you know before I had to buy a wavelength or a fiber path or something going under the ocean now I can just buy generic internet for my local provider and it will be a hundred times cheaper so people look at that and they don't go back and in satellite internet is is even more expensive than any of the other alternatives on earth historically so they often they they tear those down and they get rid of the earth station and they forget about it they don't want to look back so I'm curious about the the connection between the content layer and the transparent layer right so one of the things I wondered when you showed that great map of influence of of russian sourcing of the internet right is to think about so so I we have some data that shows that Russia Russia and China in particular are mostly internets in any case they overwhelmingly look at content hosted within their own networks right and so I would be curious to know if there's any relationship between between exporting of content to exporting of traffic for Russia and China right so there is a coupling between content and traffic clearly in the case China and Russia are interesting because they are both their own center of gravity for their own language and so they tend to have their own content in country there is a Russian diaspora all through this the the former Soviet Union and so there is a lot of traffic that comes to Russian content from there there are other the other big players it's kind of interesting actually the big guys are the ones who have idiosyncratic or or or local language populations where they host their own content Turkey and Iran predominantly but it's actually interesting because a lot of the a lot of the traffic is driven by things like YouTube videos YouTube and Google in general don't host everywhere they they can be fairly selective about where they end up where they end up has enormous impacts on the local internet cost structure if you get large content to appear in a local data center suddenly everybody locally will want to be at that data center and interconnect this can overcome for example some of the problems that you have in relationships between incumbents and competitors in a small country if if you have local content problems regulations that forbid you from hosting certain kinds of YouTube videos for example you may find that that content will not settle itself inside your jurisdiction as soon as you've lost the content in your jurisdiction well now you're just another guy at the end of a soda straw who has to go get his video content from Europe or from California and then everybody pays and so there is a very real coupling between content regulation and infrastructure quality and infrastructure cost and infrastructure delay by chasing the big content out of country you're basically forcing everybody to to go where the content is to get it specifically curious about China as well because China seems um politically they're clearly very focused on doing essentially what you said Iran is trying to do which is just going in internet right don't really care about outside content they just want to all try very hard to make all virtually all the content be local but at the same time you paint a picture here where there's sort of power and influence and money to be gained by exporting the traffic itself right so there seems like there's a tension there between there is building connections for to export them but keeping the content yep um Iran is the same thing on a small scale you know Iran wants to be an island but they also export the internet a lot of it here is is geography so Iran exports because they can they have easily accessible neighbors Russia has its diaspora population and they have contiguous territory so they can there's also railroads pipelines roads in place to allow them to to connect easily and so they do and they do good business doing internet service to their neighbors China is less so China really is more more of an island even in a place that you would expect like for example Vietnam where you might think there would be significant internet traffic between the two you do see that China telecom for example provides internet service to some of the providers in Vietnam but when you actually go and find out what do the Vietnamese providers choose when they want to go to the internet meaning we study all of the all of the active measurements we study all of the routes we try to figure out how much Vietnam really goes to China to get traffic to get content it's very small the Vietnamese providers apparently really just want to get onto the big tier one global carriers backbones and get out and get YouTube videos you know they don't it's not clear that there's a lot of Chinese content that is that is really on their radar but China is much more self-contained there's a lot less let's call it export to neighbors the kind of kind of no that's very interesting because it shows that there's like a there's a major economic and political influence cost in China's decision to keep YouTube up right yes and they're they they have this very powerful control control very well but divides the sort of cute cats that folks want but the cost is they can't export YouTube this is that's true mine apart and they don't they don't have the political influence of transit right anybody who doesn't host big content on your own network in your own country you're a consumer you end up having to go out and get the content and you know that traffic flows are really asymmetric i make a tiny little request for a video and then i get megabytes of video i get a gigabyte of video streaming in at me so those are scenarios where you lose you lose power if you don't have a balance of traffic if you're always the guy who has to receive the content and you have nothing locally to make that easier on you it really constrains growth and i think we've touched on a few of the ways that it affects the ways that incumbents choose to compete with or not with competitors don't some countries iran for example and in the news lately china may be trying to actually do that intentionally because they don't want their internal population to be exposed to all this yes that's the question um that has been stated by iran at least back in may as an explicit intention we just we'd like not to go get that content and you kind of have to ask will will the iranian population sit still and just be happy with with iranian tube and with iranian search engine and with iranian facebook i don't know the companies that work there certainly need unfiltered access to the real internet not because they want to go to youtube because they actually want to connect to people and and and sustain commerce and get financial data and so forth so can you have it both ways can you grow an internet economy where the businesses are happy but also build a restrictive consumer internet where nobody looks at all the good stuff i don't know if that's socially sort of sustainable in the long run do any of the of the operators the carriers say based on what you just said jeez we'd really like to get google here um and offer them hey look we've got conditioned power we've got real estate you can give you space or if that stuff happens is it so below the table that nobody knows about it um so i'm not i'm not authoritative on this question because i'm not google i think that that i don't think google guys ever have to buy beer i think that people make make are very friendly and want to demonstrate the benefits of a place but ultimately i suspect that it scrapes up against not only power and stability and so forth but you know friendly regulatory environment and i don't know it would be if you have that as a berkman center luncheon i will come to that one if you hear google comment on this anything to say anything but the guy we know drinks wine they'll be happy to come here and let us buy them in beer just a procedural question at the beginning you just you said that you get data from lots of isps what do they get in return like why wouldn't isp give you the information right um this question the answer to this question has evolved over 10 years in the very early days um i participate in a lot of the network operator groups the guys that actually i don't do this but these guys do this so i i'm i've done program committee work for example with nanog in north america uh with minog the middle eastern network operators group uh with minog the eurasian network operators group uh in the early days it was very much about uh you know uh can we help you measure and we could talk to routing routing engineers and they would simply turn on sessions to us uh we would give them we still do give them uh some feedback in the form of complementary subscriptions to some of the tools so that they can look at the engineering tools and see you know why are my routes withdrawn or to try to figure out some of the pads that that their traffic might be taking so there is some concrete benefit there that they get um as the set has grown um we used to we used to go out and try to meet specific providers and really you know please connect to us because we need your perspective now we're up to um i think it's about 400 different unique sessions and we now have coverage in almost every country and with almost every provider that is sort of substantially in the way of large amounts of traffic and at that point we've reached i won't call it a complete picture but something which which which reliably fails to disappoint when we try to ask it questions about regional connectivity so that the stress is no longer there to go and get more and more connections for this this observation system but the everybody who does join up does get a some complementary subscription to the the data services which they find useful sure how big a market is there at the engineering firms that actually do the work of laying the undersea cable are there a few large providers so it's a fairly open market you know i'm not totally read in on that there a project like that they don't happen at the rate of hundreds per year you know they have they happen they tend to happen in peaks and valleys there'll be a big burst of cable laying and then there'll be a desert for a few years where everybody says wow we really built too much cable i don't know what engineering firms i know alcatel lucent has done a lot of work like that especially in in the gulf region they lay a lot of cable they tend not to be the companies themselves you know laying the cable that that operate the cable there's sort of a separation of concerns but there can't be too many layers because frankly there's not a lot of money left in pushing bits around the planet so everybody's pretty tight what sort of a order of magnitude cost would there be just lay sort of an undersea cable landing i don't want to i don't want to miss quote and and my knowledge may be may be way out of date so let me get back to you so i'm very curious about you set earlier about the that egypt was able to shut down its internet because egypt telecom controlled the physical infrastructure might with with absolutely no grounding my story i've i've told you about is that it's just a matter of a dozen telephone calls to companies that egypt has some influence over so but that's actually a very different story what's very interesting so i'm curious who do you have what sort of evidence what you know about that um yeah so i don't have definitive evidence on either side of that the the the majority of the reports that we had received said it was one of those two scenarios either it was 12 phone calls or it was a physical shutdown and what we did which i had the slides was we in the blog you can see it we looked at the time series of outages to see how fast it happened and then we asked ourselves i think every route went out within 20 minutes except for the providers that were unaffected and we said well is that a timescale that makes sense with phone calls or physical infrastructure and initially i said that has to be phone calls 20 minutes and that's you know an eternity on the internet and somebody pointed out to me that um if you have uh ups gear so battery backups for your routers and they're in a rack and they're in some dark dusty room in the basement of an exchange building and somebody turns off the power what will happen they'll some of them will will immediately die because they've never been serviced and you know like my battery backup at home probably would not work right now if the power went out others of them will will have been refreshed and maybe you get three minutes maybe another one you get eight minutes and after about 20 minutes they're dead so all we have is i think anecdote and inspection of the outage data but i'm reasonably convinced at this point that it was in fact power related probably at a central point anybody have better evidence i'm here scouting for evidence as well it's an unusual thing i mean the level of we look at a lot of these countries we come up with scores for them based on how many different people actually have international access that's usually a fairly good metric of what's going to happen under duress if you have a large number of domestic providers all of whom are able to buy directly from european or asian providers that's usually a sign that everybody has access to the cable landings but everybody can write the contracts that they need to it typically means that there's nobody sort of being the bottleneck and so based on that alone i had predicted better survivability for egypt during the troubles because there are several independent providers there all of whom at a logical level in the routing in the paths look like they have perfectly good direct relationships with international providers but we underestimated the degree i think to which telecom egypt still owned the exchange infrastructure controlled the power owned the dots owned the fiber all the things that you had to have in order to make the connections to the international providers so it was it was a hidden lack of diversity so i guess my sorry for dominating so my other question i have is about um edge peering which is this problem that if you're if you're monitoring um routes by just looking at routes now right at route announcements in the core of the network basically right and there's no way for you to see the little peers who are just the networks on the edge who are just announcing peer connections in between one another that's right do you feel like you have a grasp of what those connections are or you feel like they just don't matter just don't matter for your purposes that is one of the systematic errors in this approach any approach that's based on observing paths routing paths uh you're right so if you imagine the internet as sort of a big mountain snow capped mountain like erot so the the absolute edge those of us with dsl at home we're at the base of the mountain looking up at the summit and you buy from a provider who's at base camp one and maybe he buys from a provider who's at base camp two and on up to the top at the top of the mountain there are about 12 or 13 providers holding hands with nobody above them the default free zone the tier one of the internet if it exists and all they do is pass traffic to each other basically settlement free that's how it always worked before and what has happened over time is a greater and greater tendency for people halfway down the mountain to string fiber to each other from base camp to base camp and deliberately so that they don't have to pay to carry traffic all the way to the top of the mountain and back down again when will renaissance see that well we see it whenever we have a relationship for collection with one of the providers who is below the provider of interest right so if I have if I'm if I have sessions from two or three people down at base camp one and I want to find out who base camp a guy way up at base camp four is peering with I'm in good shape because they are his customers and I hear from them what he is doing I will see the peer routes to his peers edge peering is the hardest case where people way down low who historically never should have had enough traffic to make it worth it are peering it makes it very hard for renaissance to get a collection relationship with either of the parties and in that case we will not see that peering edge it's a it's a known problem if that if these are small guys who are just peering for fun because you know I'm the University of Iowa and you're the biggest computer repair company in Iowa and we should be peering it may not matter if the people at the edge are content providers may be emerging large content providers then yeah we're missing some of the traffic that's implied by those routes but that's what active measurement is for as well so in the last couple of years that they said we've done a lot of work to get dozens and dozens of vantage points from which we can do active trace route to every prefix in the global table the entire world basically so that we can try to find some of these paths in traces even when we don't see them in the routing and we can sort of put those two views of the planet side by side the physical router connectivity side and the logical routes side and say oh this is interesting there's a there's an edge being crossed here in this graph that is not actually crossed in this graph and causes consternation and gnashing of teeth and then we figure out what it is are there research projects here that are thinking along similar lines uh if there are research projects that would like to have the answers to this question well so you've made a a compelling case about the geopolitical power the potential for power but how often do you see manifestations of this I mean you've also talked about the accidents the grandmother with it and the anchor so accidents are not that and the self-inflicted damage is not that right but how often do you see people threatening to shut down or actually shutting down access is that happened I have never seen it attributed and you would think that yes if you were going to use something like that as a lever of power you should talk about it you know here's what I'm gonna do here's what I just did and we haven't haven't seen a lot of that there was we we were I remember during the the brief Russian Georgian skirmish that went on we were actually watching Georgian routes through turkey and because we were curious what would happen there was some speculation at the time that somebody would launch attacks and maybe take out the pipeline in order to create some some economic chaos well it never happened I think there were some near near misses but no specific attacks but there would have been an impact on global routing at that point it would have been accidental though as a byproduct of somebody who's trying to to exert influence through blowing things up the the internet would not have been the target as they say the internet is harder to use as a weapon I would say at an infrastructure level just because there is so much internet and the credible threat of turning something off goes away if everybody has two or three backups you could use this against French Polynesia to great effect they only have you know a cable snap that cable now you're under our control weren't there real incidents there was something in Estonia a while ago there was the thing during the Georgian-Russian war every once in a while during unrest in some of these other countries a lot of things get turned off sometimes with prior announcement there there was an incident and nope who knows if this is connected or not there was an incident back during the Georgian time where the Russian army blew up the cable landing station at Poti which was I think not yet complete and the fiber was not yet lit but it was going to be a vast expansion of the caucus's cable system it was going to carry large amounts of traffic to Varna across the Black Sea and the landing station got taken out fog of war who knows why I think a lot of the other incidents that we kind of we all hear about in the blogosphere are our D-dosses and things that are at higher levels I haven't really heard a lot about infrastructure maybe because fewer people comparatively understand the infrastructure you know it's kind of magic it's down under the streets as we say and it's not understandable as a target I mean if you take out a government website it means more than if you turn off an internet connection in terms in the press in terms of influence which is I guess good right we don't want the internet infrastructure being used as a geopolitical weapon we want it being used to to grow economies and foster communication connectivity all the good stuff we don't want it being used as a lever for exercising power how much does the uh I've heard speculation that the the routing redirects through China recently were actually signaling of a willingness to use routing as a tool you want to comment on that stuff speculate read my blog oh I heard that speculation too so speculations are cool but they're not you know they're not falsifiable but the the technical details of that as and we looked at that in a lot of detail it looked like the kind of thing that was really more more compatible with an explanation by accident than deliberate signaling of the ability to redirect but you know speculations are as I say they're unfalsifiable I think that's uh I think that's good uh thanks for the talk