 Analysis of the Centrelink level three outage. And this happened on August 30th of 2020. Today is August 31st of 2020. And we're kind of break down what happened. This is a great postmortem up here by Cloudflare. But Cloudflare had visibility to tell us what happened, but they weren't the cause. They were just, well, they're the first person I found a good postmortem on is to determine why the internet broke and why a system built for resilience had such a catastrophic failure. It was not an attack. It was, well, a mistakes were made, essentially, that causes. So it was not nefarious in nature. It was more, oops, the internet got broke. We're gonna break down those details if you can take a second and click the like button and first. If you'd like to learn more about me or my company, head over to laurancesystems.com. If you'd like to hire a short project, there's a hires button right at the top. If you'd like to help keep this channel sponsor-free and thank you to everyone who already has, there is a join button here for YouTube and a Patreon page. Your support is greatly appreciated. If you're looking for deals or discounts on products and services we offer on this channel, check out the affiliate links down below. They're in the description of all of our videos, including a link to our shirt store. We have a wide variety of shirts that we sell and new designs come out, well, randomly. So check back frequently. And finally, our forums, forums.laurancesystems.com is where you can have a more in-depth discussion about this video and other tech topics you've seen on this channel. Now back to our content. So CenturyLink, a major ISP and internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare's customers, as well as a significant number of other services and providers across the internet. While we're waiting for a post-mortem from CenturyLink, I wanted to write up a timeline of what we saw and this is from Cloudflare. They break down the increased errors they see, how they tried to mitigate this. So they tried mitigating it, but I'm gonna explain shortly how BGP is really a difficult piece to mitigate when something goes wrong. So they break down all the stats, each little piece of this. Now, what really happened here was the CenturyLink system is huge. CenturyLink and level three after their merger probably represent one of the largest backbone providers for the internet. That being said, when they make a mistake because everything is so interconnected, which we'll get to a little bit later here in the video, the scale at that mistake, a small mistake scales outward and breaks the entire internet very quickly because of our dependence on all these networks communicating with each other using a protocol called BGP. So this was also exasperated by CenturyLink's network not honoring route withdrawals. So what exactly happens here, and of course this is the result of it is the CenturyLink outage map, is BGP is the way any two different networks have to talk to each other. So you go through the routes of how everything is connected and we have the CenturyLink net, we have Google's network, we have Comcast, we have Wide Open West, many different providers, AT&T, and BGP essentially is the protocol that says this network belongs to this particular group. And there's lots and lots of these groups. So what happened was they made a mistake and are using a tool called FlowSpec over at CenturyLink and somehow they well pushed the wrong routes basically saying instead of the networks they're connected this way, they're connected a different way and somehow that was not validated. And as you can imagine, once you've invalidated a route and you cannot get to something, it's really hard to withdraw that and send the right routes back. There's mitigations against this, but apparently everything failed. So that's why these postmortems are important to try and dig into this. And this is what happened not that long ago when I covered this before how Verizon also broke BGP and knocked out large part of the internet. So let's talk about what is BGP and what this actually means. Now these are not IP addresses. These are all those autonomous systems and the way BGP looks. Each one of these tiny little dots represents a separate network. So we're gonna zoom in here. And we can go and nail down one particular one we'll zoom in and see it's interconnect. So here's this little tiny belongs to KimaDual. This is coincidental. I clicked on this, I have no relationship. And it shows all right, these are the interconnects this particular one has or maybe this particular one. But then we look at something like cogent communications. There's quite a few of them. And then we look at level three. So when you look at something at this scale, so it's quite big. We'll go over here. This is another segment of level three. This is a segment of level three. This is another piece of backbone, not by level three, you can see how much smaller they are. So you talk about just these couple from level three and looking for each one of these, that pretty much tells you that they run the internet. They are the biggest part. And I mean, everyone says, okay, how big is Google? Well, let's search for Google. And we dig into like right here with Google. Yeah, Google has a lot of interconnects, but where else does Google interconnect against? Level three, level three and cogent. And then hurricane electric. And then we have some smaller ones over here part of the TI net SPA backbone. So yeah, Google's big. What about Amazon? Aren't they big? Amazon also big. What's in between Amazon and Google? How do those two communicate with each other? Back to level three and level three. So there's those interconnects and XO communications as well. So you take these large pieces and we're back to showing how these interconnects work. So once someone, the scale and scope of level three makes a bad round announcement, you can see by taking these out, if you were to invert this and say, what happens when level three is not connected to some of this greater internet? Well, you can't get places. And that's gonna include things like Comcast as well. So for example, right here, we dig into some of the Comcast network, which once again, Comcast has a lot of pieces here. Here's a lot of different communications, but where's the big nodes again with Comcast? Well, they're still here. There are still our big level three interconnecting them. And because this is the glue that how, basically holds everything together with those analysis being bad, that's what happened. So this is kind of a short summary of BGP. I'll leave links to this so you can read and dive deeper into what happened, why it took so long to fix. Basically, once all this lights up, and of course this affected me because here on a Sunday morning, when this was occurring, we got all kinds of alerts, all of our clients were down, we're trying to sort it out. And we find that we can't get too many of our own communication tools. And right away, you jump on Twitter, you jump on all the other places, you go, all right, there is a large outage and well, it turns out to be BGP. When you're dealing with small networks, usually DNS is your headache. When you're dealing with something, the scale and scope of the internet, it's gonna be something like BGP is your most common flaw. So I'll leave links to this so you can do some further reading and I'll leave link to this visualization because boy, it's kind of fun to play with, but by the way, make sure you're doing this on a fast computer because rendering all this is challenging for a slower computer, a render a lot slower. All right, thanks. And thank you for making it to the end of the video. If you liked this video, please give it a thumbs up. If you'd like to see more content from the channel, hit the subscribe button and hit the bell icon if you like YouTube to notify you when new videos come out. If you'd like to hire us, head over to laurancesystems.com fill out our contact page and let us know what we can help you with and what projects you'd like us to work together on. If you wanna carry on the discussion, head over to forums.laurancesystems.com where we can carry on the discussion about this video, other videos, or other tech topics in general, even suggestions for new videos that are accepted right there on our forums, which are free. Also, if you'd like to help the channel in other ways, head over to our affiliate page. We have a lot of great tech offers for you. And once again, thanks for watching and see you next time.