 the next talk that we have that we have and I guess how to discuss how the this kind of presentation you know it was ironic because when I when we were reading the CFPs we were looking at the CFP and I'm looking at the description it was really eerily familiar it sounded really really familiar because does anyone remember the not too long ago there was a D-DAR a distributed distributed Nala service attack again dying and it took down some of the big websites back in October of 2016 it also affected all of each and every one of my courses as well I write coincidentally two weeks before all that mess happened in October of 2016 which took out some of the biggest serve web biggest services on the internet I was I was putting together content and material to talk about distributed distributed Nala service attacks and I was looking one of my resources that I that that was just absolutely fantastic was on IP spoofing by by Merrick and sure enough when we opened up when we read the abstract with of course the name and the title were the name and the title were were covered once I opened it up as they were so spooky like sure enough here it is so now it's very coincidental and I guess things happen for a reason Merrick I think happened for a reason and that's why we're here ladies and gentlemen from cloud firm Merrick but no one said my surname is easy alright thank you for having me here first thing this is the room is fairly packed so if you have a free seat near you can you rise your hand so people at the back if they want to sit they would know that there are a couple of seats in front especially left so yeah please do grab a seat this is going to be a long talk yep there are still seats in front and also at the back is super cold so it's much warmer here alright so let me get started so this is talk about IP spoofing and I dare you I will say the phrase IP spoofing at least 100 times so if you are not in technology you can at least count if I'm I'll do that and I promise that I will not say cyber any anytime so it's not about cyber it's about AP spoofing and if you if you look at this talk kind of from the higher level point of view you can sum it up as how the internet works what's broken in the internet fundamentally and how we as the internet community can fix it eventually that's the optimistic view but more more specifically this talk is going to be about IP spoofing what it is how it allows the Lord attacks and finally how to fix it so with a positive note at the end alright so what is IP spoofing I'm sure everyone here in this room knows this is this is pocket hacking village but still let's let's just give couple of slides of introduction what I'm going to talk about so by IP spoofing in this talk I mean the capability of our network cards of our of our devices internet devices of transmitting packets over the internet and on our phone on our laptop everywhere on every connected device the network card itself doesn't know anything about the higher level protocols so from the hardware point of view you can transmit whatever the hell you want over network and we often forget about it you know people program in JavaScript these days and whatever Python and they forget that network card inherently is it's a device that we control so you can basically transmit whatever you want over network and that includes the headers all the headers that includes the IP header so once again fundamental you can put whatever data you need you want in the IP header and by IP spoofing in this talk I mean specifically putting the fake or or malicious IP address in the source IP port on the IP packet so basically overriding the the source IP in the packets and you can totally do that this is absolute legitimate feature of our network stock so what's the problem why this is bad why this is something we have a talk about it's bad because it's fundamentally enables in purse impersonation so from the receiving host point of view if you ever receive any packet from the internet basically there is no way you can find out if the packet was actually originated in the source IP it it says or was it actually fake and deliver and created by someone else some potentially modern most actor basically there is no way from the receiving side to know that and that's pretty bad and we learn that is pretty bad years ago this is a photo of Kevin Mitnick who is who used to be or still is one of the most famous the famous hackers in the universe in the galaxy and so one of the his his best attacks was against Mr. Shimomura and in 1995 he basically took over Shimomura's TCP session and injected some some commands in his TCP session therefore granting himself fruit and he was able to do that only because he was able to spoof packets is because he was able to fake the packets put magical commands there and grump himself access that was in 1995 so quite a while ago since then IP spoofing was a problem and number and number and number of different attacks different exploits different kind of internet fundamentally internet issues in 1996 we had a wave of sin floods which I would say kind of the problem of sin floods was solved only about two or three years ago but since lighting is a different story in 1998 we have my favorite attack sorry scanning technique called idle scanning this was invented by Antires who now is working on this but if you haven't read if you don't know what idle scanning is I highly recommend it is a super interesting scanning technique again based on requiring IP spoofing then we had a whole bunch of problems with TCP implementations I think the most well known are the ones when most attacker by forging reset packet was able to drop BGP connections so that could that could that led to pretty serious problems in the internet and then there was another ten years and in 2008 we all know the famous Dan Kaminsky's DNS bug and yes issues which again were only triggered could only be triggered if you could do IP spoofing and I would say since about 2013 you have a new wave of attacks of the big denial of service attacks which again are all caused by these poofing but you know that was on the attacking side was on the defense side are we making any progress as the internet community is is there is there anything happening of course the first formal definition that you know IP spoofing is something but it was in around 1998 and around 2000 there was a famous BCP 38 published it basically says yep the internet work as it works but we should not allow and hosts to forge packets to send packets from the IPS they don't belong to them unfortunately BCP 38 is fairly naive so then BCP 84 four years later came in and said you know it's harder than we thought but still we should do that and there are many working groups and many kind of initiatives that that's right now try to fight a spoofing so they there is the ETF working groups called savvy there is man arrest a working group in the industry and then there is the spoofer.kai.org project which tries to track down the IP spoofing around the world so this is a chart from the spoofer.kai.org project and it basically shows that around 56% of ASs of networks on the internet do not allow spoofing so it's good not everyone allows spoofing but still the whole problem is not solved yet there's around 27% of ASs of network that do allow IP spoofing so you know the problem still exists in the internet even though it was first problematic in 1995 and not only that there is the whole industry around amplification around attacks using IP spoofing so if you if you use your Google skills properly you'll be able to find services like this one so spoofing enabled offer servers when you can for Bitcoin by bandwidth not even bandwidth by servers and do whatever spoofing you want so this is a problem this is a proper industry this is a problem by the way in the terms of service they do tell you that if their ISP disables spoofing if the ISP tries to enforce filtering they will not give you the bitcoins back. All right so what's what's the big deal about spoofing well in my opinion it enables the large attacks so if you open new outlets and read about the big you know ransom emails requiring you know banks to pay up or kind of attacks that actually were influencing some institutions in most cases not all but most cases those big attacks the biggest attacks will be caused by a spoofing in one way or another. Now there is there needs to be a disclaimer here I'm not this is not about the mirai attacks so like we can speak about the mirai and dine at the end in the question section because they were a bit different but except for this case the major attacks you've heard were almost all about IP spoofing. All right so who am I why do I care why do how do I know anything about it. So I work at the cloud and we operate a global reverse proxy network we have our servers all around the globe we operate on almost all continents we have customers from all backgrounds from government agencies to social social portals so and we also have plenty of customers so we see a quite a good cross section of what's happening on the internet. This is a chart of daily attacks that we categorize as denial of service attacks. This is per day and you can see that some days are quite quiet with 50 or 60 events that we categorize as denial of service attacks and some of them are more busy with up to 1400 events. So we see a good chunk of the weird things on the internet and I won't lie most of the attacks that we see are fairly small they are not really anything exciting but some of the attacks that we do see are super large are gigantic and over the years we've published a number of of blog posts describing those attacks in big details and and in this talk I'm going to show you two major types of attacks we've seen and I'm going to also emphasize why they are problematic because of the IP spoofing. All right so one of the attacks I'm going to talk about is called I will call direct attack and the other one it will be amplification. So let's start with the direct attack. So what is it? Why do I why do I put it in a separate category? And so this is an attack when the attacker is just transmitting packets directly to us directly to the target without any amplification and a reflection in the middle. It's just packets from the source the target. That's it. Fairly simple. But we cannot trace the attacker because the source IPs of the attack are spoofed so we don't know who actually originated them. About a year ago we published a story which we dubbed the winter of attacks in which case the attacker had about 400 gigs of capacity of spoofing capacity. These are the charts that we published. They look fairly fairly shady but basically we were hit with about 200 on the peak 200 million packets per second of the attack traffic and in the peak as well about 400 gigabits per second of the attack traffic. So it was fairly substantial and once again it was direct attack. So there exists an attacker in the world at least a year ago that had 400 gigabits of spoofing capacity. So the interesting things about these type of attack is that they are usually originated in only couple places and couple of I would say big data centers or on big ISPs. So when we see them we see them only in a couple of places around the world. So that's how we notice. Okay so once we see such an attack what do we do? Well first we have to make sure that we are kept online, that our servers are operating, that our customers do not feel the problem, that we are, that the internet still works for majority of our customers, for all of our customers. So what do we do? Well we obviously try to figure out what is hitting us. So we run TCP dump or some other tools and try to figure out what is in the packets and in this case those 400 gigs this was a SYNflat. So it was very very very large SYNflat. And it's interesting because it had to be, SYNflats have to be direct. There is no way to reflect SYNflats. So basically if you see a SYNflat this is direct attacks. In many of the SYNflats the IP packets, the IP sources are not spoofed so it's just flood of SYN packets with a source IP address belonging to the attacker. But in that case it's easy to recognize who is attacking you. In our situation the source IPs were obfuscated, they were just random numbers. Okay so how do we keep ourselves online when we see 400 gigs of SYNflats? Well we mitigate it. And our favorite mitigation techniques is called BPF. This is an example of IP tables command with BPFs inside. So what is a BPF? BPF is a bytecode. It's a program that you can stick inside your firewall, inside your IP firewall, and run it within IP tables. In the bytecode you can read data from the packet and you can basically have an almost fully fledged program running inside your firewall which is super cool. So on this command line prompt you can see a byte of numbers. What are those numbers? As I mentioned there are program. Here is the same program is just decompiled to more assembly like format. So once again it's almost fully fledged program. And by the way if you ever run a TCP dump or wire shark you should be familiar with BPFs because that's exactly the interface between TCP dump and the kernel. So if you use TCP dump and type something like port 80 this will be compiled down to this bytecode. Alright so more specifically we can't really use the TCP dump expressions because they are a bit too limited. So we wrote a series of tools that we actually use to mitigate the big attacks. They're all open sourced so on our github you can find BPF tools and that's exactly the bytecodes that we run to to mitigate attacks including synflats. Now there is a disclaimer here all the IP tables is fast and BPFs are super fast and it all works reasonably well. There are still some limitations. IP tables run fairly late in the kernel network stack. So you can do much much faster by doing some magical tricks like kernel bypass. But I won't go there because there's a new development in kernel in the Linux kernel called XDP which is express data path. It's available from kernel 4.9 and right now you need to have a magical specific network card only a couple of vendors support it. But it's super cool. It's really great but that's it. I won't say much because tomorrow 2 p.m. my colleague will give a talk exactly about XDP. So stay tuned and be here tomorrow at 2 p.m. Alright so we mentioned how we keep our services online, how we drop decent packets on the floor and sometimes we want to identify who's actually behind those attacks. We want to dig deeper into who is actually sending the traffic where the traffic originates. So basically we need to identify, since the source IPs are spoofed, we need to identify from which physical cable the buckets were delivered, like where they are coming from. We have no number of cables connected to our routers. So what we do is we log into our routers and we chase the graphs. This is a chart, this is a couple of charts from one of our routers and here you should be able to clearly see that there are some inbound spikes hitting whatever 10 gigs on the inbound on the bottom left chart. So it's easy to identify from which interface, from which physical cable that is coming from, even though the source IPs are obfuscated. Alright so then we follow the wires, then we look on our router carefully and we try to figure out who's on the other side of the cable, who's actually sending us the traffic. And there are generally three types of cables that our routers have, three types of connectivity that we have on our routers. This is basically how the internet works by the way. So the first type of cabling is called PNIs, direct peering or cross connects. It's basically a direct connection, a direct cable laid from our router to someone else's router, when someone else is an entity similar to our company. So it could be Google, it could be Akamai, it could be AWS, it could be any other number of providers. So basically another entity on the internet. The second type of cabling that we have is a cable that goes from our router to local internet exchanges. I will speak more about local internet exchanges but basically internet exchanges are a collaboration of local ISPs and it's a single place where they all connect. And finally there is the connectivity to the transit providers or internet carriers, where basically it is paid for traffic, we pay for to deliver packets to the internet. So it's kind of normal connectivity to the internet. All right, so let's imagine that the attack came from direct peering. So what do we do in this case? What do we do if we know that the attack packets came from this specific cable, which has this other entity on the other side? So it's easy, it's easy because we can just call up them and say, you know guys you're living as junk, please don't attack us, please don't do that. And the good news is that this usually works, this usually works well. So the relationship between us and other part that we cross connect to is usually very friendly. It's in both our interests to, you know, to keep the relationship okay. We do have sometimes situations when the ISP on the other side does respond to our calls and say okay I will kick off the customer, it's actually attacking you. And then the problem reappears a couple of days or weeks later. But this is fine, like we can call them again and in the worst case we can just deep hear them and force the attack traffic to go over the internet and increase their costs. So basically there is some discussion here, so that's great. So what about the other two options? What if the attack comes over the internet exchange or what if the spoofed attack comes over an internet carrier and transit providers? Unfortunately in this case we cannot do much. So this is a pretty sad story here. So let me explain to you why this is the case. Okay, so let's focus on internet exchanges for a while. So this is a picture of I think Seattle internet exchange. It's basically a gigantic layer to switch with plenty of cables and with local internet providers connected. And so what's the issue? The issue is that internet exchanges are layered two switches, so there are ethernet switches, while routers are layered three beasts, layer three entities. And from router point of view, router is there to route packets, so it takes packets, it throws away the ethernet frame as quickly as it can. That's its sole job, it's to route packets on the IP layer and then it routes it back to our data centers. And that's an issue because from the router point of view all the router knows is that yep there are packets, they are back packets, they are coming from internet exchange. That's all I know, I don't know anything else. So basically we cannot really figure out which of the local ISPs, which of the local connected parties is actually originated in traffic. Now if you think about it, this data is there because it is in the ethernet frame and the source MAC address usually should point out to the router that is belonging to the entity that is sending the junk traffic. But unfortunately as I mentioned, our router doesn't really look at that because it's on different layer. So this is pretty sad. Okay, so what's in the third case? What happens if the attack comes over internet carriers or transit providers? What can we do in such case? Well we often call them up and say, you know guys, you are delivering us packets which are just junk, which are just involved packets. And they say, well it's because you pay us to do that. And that's true, like the internet carriers are there to deliver packets. We paid them exactly for this job. And by the way I don't think that is their job to filter the packets. It's censorship, it's not really their job. So it's kind of fine, but still they should be able to do something about it, they should be able to inspect their network and figure out which of their customers is actually originating the traffic. Unfortunately that's not the case, they cannot really do that. So again, pretty sad story, we see packets flying in from our internet carrier. We have no idea who is actually originating them. So that's that all leads to the conclusion that basically in most cases tracing back these spoofed packets is impossible. And that's pretty bad. And that's pretty bad because we cannot really push back against it, we cannot really figure out which data centers are evil, which IPs are evil, because we just don't know. Okay, so let's dig into a bit more details into the attacks I showed, which is the winter of attacks, which I mentioned already. I can actually share a couple of more details about this specific attack. So this attack was on the third category. This attack, this 400 gigs, was coming to us overpaid for internet carrier. And in this specific attack, the source IP address we're belonging to an AS number belonging to Hurricane Electric. Hurricane Electric is a big internet carrier. They are one of the more competent ones. But anyway, from what we know, all we know new is that we see attack traffic, we see sinflad, and the sinflad comes from IP ranges belonging to Hurricane Electric. Okay, that's pretty bad. But then we noticed that actually the attack was coming to our Los Angeles data center, and in Los Angeles we actually have a direct connectivity to Hurricane Electric. So that leads to two conclusions, to two options. One option is that, okay, the attack was actually coming from Hurricane Electric. You know, it is possible, but in that case it should be delivered over direct link to us. It shouldn't be delivered overpaid for internet. So maybe they misconfigured the routing. If so, they should fire their router admins because they just paid for a big attack traffic. So let's say that's not the case. So the other interpretation is, of course, that our internet carrier has a client, has a customer who is spoofing packets and pretending to be Hurricane Electric. But this is a nice situation, an explanation that our internet carrier actually doesn't know, does not know if the packets are valid or not. From the internet carrier's point of view, these packets are originating in Hurricane Electric, and that can be okay, that can be fine. It's just we know that that wasn't the case in this situation. All right, let me show you a couple of more examples of similar attacks. But before I show you more details, we need to draw, we need to have a vocabulary, just speak about those attacks, those IP spoofing attacks. So this is how we're going to show it. So this is a picture from XKCD by Randall Monroe, from about 10 years old, and he attempted to draw a map of the internet. It's not easy, so what he did is he used a mathematical thing called a Hilbert curve. And the Hilbert curve is a way to draw a line over two-dimensional space that will kind of group things together. So in each of the squares you can see here, there is a slash 8 subnet. So you can see top right corner, there are some reserve blocks, again kind of top right corner, there is a multi-class groups. So basically all these slash 8s are kind of close together in a in a square. If you look at slash 16s, they will also be in a smaller square and so on so forth. So it's a pretty cool way of visualizing IP space, basically. Okay, so let me show you one of the attacks. So this is a Hilbert curve drawn by me. You can see some harsh out blocks. These blocks are basically unrautable IP addresses. And in the middle there is 127 slash 8, so kind of reserved loopback IP range. Okay, so in this attack for each of these source IPs that we saw, there was, I drew the black dot. Okay, so plenty of IP addresses scattered all around. Is that interesting? What's exciting about this? Well, not much until you realize that these are IP ranges belonging to China Telecom. So again, this is the attack, this is China Telecom. Attack, China Telecom. So the interpretation of this is fairly hard. Like I'm not saying China Telecom attack does. I'm saying the IP addresses belonging to China Telecom were used in the attack. And from my point of view, it's super hard to actually figure out what's happening behind the scenes. So there are two interpretations possible. One interpretation is that someone has a large bot net in China. I know it is totally possible that someone has a gigantic bot in China is just using it to attack us. That's fairly sane. It's possible. In such case, the IP addresses will not be spoofed. There will be properly legitimate IP addresses belonging to malicious bots in China. On the other hand, if you look at that again, they seem to be quite well distributed. Again, I'm not an expert. Maybe China Telecom IP addresses are distributed evenly around the China population. I know if that's the case, not as possible. But it could also be the case that someone has just went through the IP allocations and is just spoofing the China Telecom IP addresses. There is no way to find to figure it out. Okay, so another example of exciting attack. This one, this is my favorite. So in this case, you can see that the source IPs are just uniformly distributed across the whole IP space, including the reserved blocks, including 255, so the top one, including zero, so the bottom one, but also including reserved blocks like 127 slash eight. So if you ask me, do I see attacks from loopback? Yes, absolutely. Every second of the day, I see malicious packets coming from 127.01 attacking my network. So you know, there is that. There are then variations on top of that. So for example, this is a very similar chart with uniformly distributed source IP addresses across the whole IP space, but with the reserved blocks skipped. Now, the question is, why did it happen? Was it the attacker that is smart and when they have the routine that is forging the source IP addresses, is the attacker just filtering the IP addresses and say, okay, there is zero, so maybe let's not put the obviously wrong ones, but I don't think that's the case. So I think what's actually happening is that their ISP is filtering the traffic, but again, there is no way for me to figure it out. There is next one, next very uniform distribution. Why this particular pattern? I have no idea. Maybe someone will know. And another one, this may again look like a fairly nice, either gigantic botnet from all around the world or just someone spoofing some weird stuff with a bit more density and kind of the lower IP ranges, but no. If you look at this particular chart carefully, you will notice, if you're a network expert, you will notice that those dots correspond exactly to the non-routable IP ranges. So in current internet, almost every IP address belongs to someone these days, but there are still plenty of IP blocks that are not routable, that don't exist in the global BGP routing table. So in this case, the attacker was fairly sophisticated. The attacker went for the global BGP routing table, inverted it, so figured out, okay, so what blocks are not present there and only spoofed these. Why such? I don't know, but here you are. All right, so that's all I have on the direct attacks. When I actually can see the source IP addresses, I can actually argue about it. Again, I cannot really track it back to anyone, but at least I see them. But then there is a second category of attacks called the amplification attacks. It's about two weeks ago, three weeks ago, we published a story about SSDP amplification. These charts are not as exciting, but basically they show that the attack we received to our network was of about 40 million packets per second, and the volume was about 110 gigabytes per second. So you know, a sizable attack. Okay, so what is an amplification attack? Just a couple of slides on it. The general idea is that there exist UDP servers in the world, UDP protocols in the world, with a basic request response protocols. And the idea of those protocols is that the client creates a request and the server responds. That's it. And you know that because every day you use DNS and it works exactly like that. You know that because NTP works exactly like that and a number of other protocols work in the same fashion. So what's the problem? Well, the problem is that if you can spoof packets, you can forge the request. So you can convince a legitimate server to answer a request that wasn't really asked in the first place. And therefore you can cause the server, legitimate server, to trigger legitimate answer to a target. And that way you can basically amplify the attack. So the main idea is that in many of those protocols, the responses are much, much, much larger than the requests. For example, in DNS, in DNS the request is usually small and the response is usually much larger. And if that's the case, if this is the case in the UDP protocol you're using, you can create an amplification of a big, big size. So in this case it's like 10 times, but it's not uncommon to see 20 times or 80 times amplifications. And this is great from the attacker point of view because possessing one gigabit per second of spoofing capacity, you are able to generate much, much, much larger attack volume going to the target. Also, you never use a single UDP server. You have to scale it out. There is no single UDP server that can generate too much traffic. So that's what we saw those couple of weeks ago. So what we saw was an attack which, again, on our side had about 112 gigabits per second. There were about 490,000 reflectors used, so a sizable chunk, a big number of reflectors. And we believe that the attacker only had about five gigs of spoofing capacity. That's what we can know from kind of looking at the protocol deeper. All right, so again, what do we do in case of session attack? Well, the first thing, as always, is we try to keep ourselves online. So how do we do that? There are a number of techniques we're using, but one that the most interesting one is that we are, this attack is nicely distributed geographically. This is because all of those reflectors, all those 940,000 reflectors around the world, they don't belong in a single place in the world. They are scattered all around the globe. And this is good from our point of view because the attack traffic is nicely delivered to all our data centers. We can do that because we run an anycast network. So we advertise the same IP ranges all over the world. Therefore, the attack packets will be delivered to the closest data center that is to the reflectors servers. So again, our architecture allows us to basically split the traffic very evenly across our data centers. Okay, so what do we do next? Well, blocking these attacks is usually very, very simple. These attacks usually happen, usually come from a specific protocol. So if it's SSDP, the servers will always send packets from port in 1900. So you're just dropping port 1900 with the protocol UDP on their firewall will do the job. So it's really fairly simple. It's a bit harder for things like DNS because you can't really drop all DNS traffic. But still, fundamentally, that's a fairly simple thing. Okay, and how do we identify the sources? How do we know who actually is attacking us? And the answer is that we don't and we cannot do that and no one can do that. And that's pretty sad again. So the issue is that the packets that are delivered to us are legitimate. The reflectors servers deliver us responses to the requests we never asked in the first place. But still, the packets were legitimate. The source IPs are absolutely correct. So if you wanted to trace it back, we would need to ask the internet, dear internet, who in the internet forged our source IP addresses? And there's just no, you just can't ask this question. All right, so what about the amplifications? So in the another blog post that you published, you mentioned a couple of other protocols used. But these are usually the usual suspects. So it's usually NTP. It's SSDP is the biggest one we saw. There's also DNS and a couple of other protocols. And most interestingly, there are a couple of gaming protocols. So I think we will see a new interesting attacks that are using gaming protocols, like call of duty or source. Because these protocols are not audited well enough. On the attack sizes, the amplification attacks are usually fairly small. So the average size of the attacks we noticed in the last six months was about seven gigabytes, gigabits per second. So, you know, not gigantic, sizable for small internet players, but, you know, not gigantic. The maximum was 100 gigs, as I mentioned, so this is a bit outdated. But still, it's not uncommon to see a DNS attacks, a DNS reflection attacks, going up to 44 gigabits per second, or NTP attacks. I think the maximum NTP one we saw was 64 gigabits. We said just over the last six months. All right, that's almost it. So basically there are kind of three themes in this talk so far. One is that IP spoofing is generally bad, and it allows all kinds of problems, and most importantly, the big attacks. The second thing is that in order to survive the big attacks, you need to have big network capacity, and that's pretty unfortunate because that means that only big internet players can survive the largest attacks. And also the third one is that fighting against those attacks is basically impossible. There's no way to trace the attacker. There's no way to put pressure on the attacker. We just don't know who's on the other side of the wire. Okay, so how the internet can fix it, how we can influence our colleagues, how we can buy bond with, what can we do as internet community to fix it eventually? Okay, so let's tackle it step by step. So first, we have to understand that these attacks are possible only because of IP spoofing. We would not speak here about amplifications if it wasn't about IP spoofing. I wouldn't speak about SYNFLUTS if it wasn't about IP spoofing. Yes, SYNFLUTS would have existed, but I would be able to just sue or put pressure on the people that are actually sending me the malicious traffic. So what can we do about IP spoofing? Well, fundamentally, we cannot read too much because that's how the internet works. But we should continue the usual efforts. We should continue promoting BCP38 and basically promote good network practices. These things are not dead. We just need to spend more time on it. But we can do even more things. There is this PooferKaida.org project which I mentioned already. If you want, you can download an application which will run in background and will basically test network that you connect to for IP spoofing. This is pretty cool. So you can basically, as you go around the world, you can basically see what networks allow spoofing to what degree. And finally, vendors can do a lot for example, right now the RPF, so reverse path filtering, sorry, forwarding, is not enabled by default. If you buy a new router, it should be enabled by default. There are reasons why you want to disable it, but still by default, for most of the users, it should be enabled. But again, we have to appreciate that actually doing filtering is hard. Filtering must be done close to the source. So it's basically the anti-SP's responsibility. And many ISPs, as I showed, 56% of the ISPs already fixed their networks. But there are still plenty of ISPs around the world who just either don't know that this is a problem or don't care. So we will be left for next year with the incompetent ISPs. So we will be left with the problem. So we need to figure out other strategies as well as we go. Okay, so the second thing is network capacity. What can we do to avoid everyone to have gigantic pipes in order to intake the amplification attacks, for example? And here I believe we have a technological, technical solution which is called the BGP FlowSpec. I'm a big fan of BGP FlowSpec. BGP FlowSpec is basically a protocol created on top of BGP which allows you to serve firewall rules up the routing chain. So basically your router over a BGP session can tell to our routers, by the way, here are my firewall rules. I don't want some of the traffic. I don't want the SSDP traffic. I want the amplification traffic. Drop it as close as possible. And this is great because it allows you to relieve the congestion on the last link and basically push firewall rules up the chain. Now, truth be told, we did have a couple of issues with FlowSpec early on. So a couple of years ago, we had a number of voltages caused by that. I mean, sorry, FlowSpec worked perfectly. It dropped all the attack traffic. 100% of the attack traffic. Unfortunately, it also dropped normal traffic. But these are technical problems. And I think I'm sure they are fixable and I'm sure they are all fixed already. This particular outlet was caused by us putting IPv6 IP addresses and FlowSpec rules, which was not supported at the time. And once again, the FlowSpec shines because it can cross AS boundaries. It's a firewall language which can basically cross ASs. So that's again great. You can basically move your rules up the chain. Now, again, I won't lie. Most people that use FlowSpec use it within one domain, within their AS number. It doesn't cross the boundary. So it's not as useful, but I think it should be deployed as the cross AS thing. And then there is at least one success story in Russia. So Raskom, I think, a year ago created a service which basically enables their customers to send FlowSpec, FlowSpec firewall rules, up the chain. So there's at least one internet carrier that does this thing right, so it is possible. So next time you choose your bandwidth vendor, you choose who to buy the bandwidth from, ask about FlowSpec. Okay, and finally, what about tracing bug? Can we ever find out who is behind the attacks? Can we know that? The answer is yes. And again, there is a technical solution which is called NetFlow, or nowadays it's called IPvix. And again, I'm a super fan of NetFlow. So what is NetFlow? NetFlow is a protocol which allows the routers to send packet samples, actually not packet, but flow samples to a central location. But this time we win one AS number within your domain, so it doesn't cross. So there's kind of no privacy issues to cross data across ASs. And this data can be collected and can be analyzed. And with this data, if configured properly with good toolchain, ISPs and internet carriers and T-run providers should be able to answer questions like which of your customers actually originated the shitty traffic, who is actually behind it, who is actually being paid money to deliver this poofs traffic? So yeah, so we could just call them in such a case and ask and they will just run a simple command line tool and basically be able to answer those questions. But once again, the point is right now it's impossible the ISPs generate the not-support-flow spec and that's pretty bad. There is the privacy aspect of that. So people say that okay, using NetFlow and basically logging traffic on this ISPs is bad for the privacy. This is true. I'm basically asking for more logging, but there are ways around it. There are ways to find a compromise. So for example, I'm only interested in the really, really, really big, big, big attacks. I'm only interested in the gigantic spikes which threaten the stability of the internet. I really don't care about your customer connections. So you can set the sampling rate today lowest or largest depending how you look at that maximum value. So which is one I think in 64,000 of flows. So unless you are sending 64,000 of flows a second, your privacy should be reasonably preserved. And second thing is I care only about recent attacks. I care only about last, whatever, 48 hours of traffic. So the logs can be aggressively rotated. This is fine. Okay, so there is a tangent that it is a bit harder for the internet exchanges. But again, there are technical solutions to that. One of the solutions is called MAC accounting which is a feature in our routers which allows us to do something and get more statistics about the MAC addresses on that network. And the second thing is that internet exchanges, so this kind of no man's land between ISPs, should play more active role in guarding the internet. Internet exchanges should provide tools to figure out where the attacks are originated. All right, that's it. So a quick recap is in order to prevent the IP spoofing, we have to promote VCP38. This is the root of the actual problem, but unfortunately it will not fix it in short time. In the meantime, we should definitely promote FlowSpec because this allows smaller players to avoid buying gigantic, gigantic network capacity. So basically this will enable smaller players to survive big attacks. And finally, NetFlow is necessary, is required to figure out who's behind attacks to trace them back and basically solve the problem by just putting pressure on the attacking parties. Thank you very much. That's it. That's all I have. Actually, I'm lying. I have 30 more slides with more technical details, but I think we should stop here. Thank you, Magal. Anyone have any questions? Please. Anyone? I have a t-shirt to give away. So is it ever worth it to call up ISPs and chain, okay, so this ISP, whatever transit partner you're getting it from, they don't know where it's coming from, but they know which of their transit partners it's from and you go ask mom and then you ask dad and then you ask cousin and eventually find the end. Is it ever worth trying to chase that despite how difficult it is? So the question is, is it ever worth kind of asking your ISP and following the chain? The answer is we do that, we do that eventually, but the issue is they just don't know. Our providers, our tier 1 say, yep, the buckets are coming from someone and so... And we just don't have the logs. We just have no idea, we have no visibility in our network because I don't know. So there is a tangent here, which is the tier 1 providers cross-connect between each other. So it is totally plausible for people smaller than Clouser that the answer will be, yeah, we see the spoof buckets hitting you, but they are coming from another tier 1 and then could you ask another tier 1, even if you are not paying them directly money? I don't know. But in our case, it's much more local. Most of the traffic that we see is crossing only a single tier 1. So if they were competent, if they had better introspection in their network, they would be able to answer the questions, but they just cannot. But the more networks you connect directly to, the better chance that it's just coming in over your private link anyway? Absolutely, the more cross-connects we have, then the more we know, absolutely. So the better connected parties know much, much more about how the internet works, who is actually already in traffic. And this is one of the reasons why we are actually trying to cross-connect with shady entities, is to figure out, okay, so now it's you. Thank you. I didn't notice in your amplification table, but are you seeing much in the way of LDAP amplification? The question is about LDAP amplification. Yes, there was, I think, I don't know, three months ago a blog post by our competition, I think by Akamai. It's about specific, it's not normal LDAP, it's a connectionless LDAP, so it's kind of a deviation on that. Yes, there were a couple of attacks on LDAP. Yeah, they're smaller. So not as interesting, but yes, we do see connectionless LDAP as well. So you had a slide there that said that some percentage, 56%, do not allow spoofing. And can you explain how those networks do not allow spoofing? How do they manage to do that, and why are the others not able to do that? Perfect question. So the question is about the number 56.4% of network that do not allow spoofing. The answer is I have no idea, and the reason is because it's super hard to actually figure it out. So the particular data comes from spoofer.kaita.org project, so read on, read what measurements they are doing, read what methodology they are having. My understanding is that they have probes and they run probes on the internet. I think nowadays they run their software on your machines, so if you install their piece of software, it will basically probe network and upload the data. They have some privacy concerns because they don't want to reveal your IP address, so there's kind of a bigger subject. But basically they do some active probing. But what specifically spoofing they are talking about is spoofing within the IP ranges of this ISP, or globally, I don't know, but it's there. So spoofer.kaita.org is the data source here. Just curious, so the bandwidth attacks, the DDoSes get the big numbers in that. What, do you have any idea like relation? Usually the app falls first, right? If they're at... Okay, the question is about what is the app falling first. Yes, in this talk we spoke specifically about layer 3 attacks with spoofed IP addresses. So a very specific niche of attacks. There are tons, tons, tons of more attacks, L7 attacks. Interesting tangent, for example, is DNS attacks. So we at Clouthware operate DNS, and DNS is Connectionless, it's UDP. So we see packets which are kind of both L7 and both L3 because they are spoofed, but on the other hand they are literally not DNS packets. And yes, the answer is usually yes. Our DNS application will never be able to do 100 million packets per second, which we often see on DNS layers. Sorry, we used to see. So yes, so for that we are using Grace Limiting on IP tables layer, but again, BPF is the main methodology. So for the Connectionless stuff, for the UDP stuff, it's usually fairly easy to block, and you see our DNS, sorry, BPF tools. For the layer 7 things on TCP, it's completely different subject, and you always see the proper IP address there. So if the worst come to the end, you can always block on the IP addresses. So it's way easier to mitigate. You can just blacklist the whole internet. I mean that was actually kind of touched on my question, which was some of the stuff that was hitting TCP services on layer 7. Could you look for connections that were not being set up properly and just blacklist them using some sort of automatic tool? The question is, does the problem exist on TCP? No, if there is a TCP connection handshake, then I know the other entity IP address, so. Well, you had showed a TCP dump where you were showing connections on port 80. These are SYN packets. So SYN slots are, it's the first bucket in the connections handshake. So SYN slots are fairly normal thing, like everyone knows these deal with SYN slots. So it's basically a layer of free attack. So it's not a layer 7 attack then on SYN slots. Sorry, but I mean could you do stuff to mitigate based on, sanity checks, whether or not the incoming data was representing a legitimate user versus, do you understand what I'm trying to ask? Not really, but if you are asking about the details on the SYN packets, then again, BPF tools, BPF tools is what we use for. I wonder if I can do it in apps? Okay, so the uplayer. So what about uplayer? Higher level. Okay. Can I ask myself a question? So the Dyn attacks, I mentioned this talk is not about that. So from my point of view, it's not super sure what were the Dyn attacks. They were the biggest published attack on the internet with 620 gigabytes per second. So why have I spent more time on that? So first is that it's not obvious what exactly was hitting, hitting, it was crabs before Dyn. So it was hitting crabs, this particular attack blog. But from what we do know is that it was GRE packets, which without spoofed IP addresses. And also what we do know is that the attack had large post uploads. So it is possible this attack was not spoofed. So that's why I haven't mentioned it here. So yes, sometimes it happens that there are big attacks which are actually from legitimate IP addresses. But then I would argue it's fairly easy to block them. Thank you very much.