 So hi welcome to our talk. I'll introduce myself first. My name is Nick Saunders. I am or I'm responsible for cyber security for Viasats government group. I am also, you know, an engineer. I've worked in engineering for 15 years You know, I've done I started in embedded systems. I you know recently got to launch a bootloader That I wrote into space, which I was pretty proud of You know more recently I've also kind of worked into you know upper layers of the stack Oh, we're a lot of different hats in this context though talking about these events. I'll be sharing my role As one of the primary investigators for the k a sat attack And so we will be going through these different series of attacks And so mark my colleague as well will be Going through and giving us an introduction on these and then I'll be taking it over for some Technical investigation and analysis and really telling our story for what this looks like Hi everyone, I'm Mark Kulaluka. I am a network and security engineer by trade. I did a lot of work on Layer two packet processing on some of the design of our first and second generation satellites Now I am the sysso at viasat, which in this context meant one of my great jobs was dealing with the outside world to kind of Give a protective bubble for all of our engineers and forensic responders time and space to do what they needed to do So I'm gonna go pretty quick through an overview of the satellite network itself just to baseline everyone I apologize if this is a level of knowledge many of you are familiar with but it's good to set the context So when we get in the detail of the two different attack variants you understand the parts of the network Okay, sorry is that better? Sorry. Sorry guys It's not and that's our time. Yeah, it's not moving at all. We made we made the arrow buttons the There we go hey So what are we talking about? February 24th 2022 this is a graph of our terminals online on a particular section of the KSAT network The way the network is segmented Actually, we have kind of by geography, but also to load balance at different segments of the network and you can see up until about 0300 or so UTC about 60,000 modems online and the two arrows there represent the two different attack Vectors that we're going to talk about today the first one you can see a gradual decline in the number of modems this ended up being a Result of some network-based attacks that we're going to get into in detail they commenced there But they went for several weeks in Intermittently in different fashions and you'll see how the attack actually morphed and that ended up being a DHCP based attack that manipulated both the protocol and the packets and some other things the second which is probably the more widely Publicized and discussed is the modem wiper events. We're going to talk about the reverse engineering of the malware strain And the toolkit that came with it and kind of a little bit of the lateral movement that led to the deployment of that But that resulted in somewhere in the 40 45 50,000 modems being affected There was a bit of a long tail on this as modems came back online We had some instability in the network because of the activity as well This is what the network is each of those are spot beams Those are where the users live so you can see it covers most of Eastern what Eastern and Western Europe and some of the Middle East Satellite itself was launched in 2010 it Was used for both broadband and satellite TV access and at the time of the events There was about a hundred ten to hundred twenty thousand commercial modems on there We also support commercial aviation and government customers at the time in a separate partition One of the messages I want to emphasize here as we go through this the priority Was and is on the customer base that we have if you notice from that graph There were still 20,000 customers in the affected partitions still online still using the service The other partition had you know upwards of 40 or 50,000 at any one time So we had a pretty big focus throughout the incident response in the forensic analysis on Preserving the stability and availability of the service and that's a guiding point. You'll see Nick highlight throughout And with that I'm gonna talk Turn it over to Nick to talk a little bit about the background and some of the things that we were seeing early days And then he'll walk you through both attacks All right, so first word of warning. Don't believe everything you read on the internet We're gonna go through a couple of things that you know as we went through this attack We were also reading the news as it was going out there and there was you know For good reason in some cases we were not sharing information because there is information that was you know Close we're our main priority as Mark mentioned was keeping users online And you know ultimately patching any any findings that were out there But one of the things that was kind of going on simultaneously was a few different pieces of reporting that we're also here to want to get Information out correct the record share get you know ultimately get The details around what this attack was and was not so a couple things You know one was that the attack was so effective that every one of the case at ground user terminals That was turned on at the time shut itself off and could not be powered up That is not accurate. We'll go through why that's not accurate today You know also attackers were able to enter a ground-based satellite network in which via sat's k a sat Users were running by exploiting a vulnerability in the fortinet VPN there was no evidence that we found or that any partner is found and Forensic evidence directs us elsewhere. It was not a zero data any knowledge or a vulnerability that was known on the forti gate There was in fact a forti gate VPN. We'll go through that But that was not a vector and then the the third one malware was you know part of a larger supply chain attack Maybe if you defy supply chain extremely extremely liberally then sure anything could be a supply chain attack But there was no evidence of any compromise or tampering with a viasat modem software firmware images No evidence of any supply chain interference for supply chain to viasat. So I want to correct that and then the last one is that You know, this is my favorite one They deployed wiper malware to erase the hard drives of the modems disconnecting them from the case at network Modems do not have hard drives for the record. I actually kind of like this one because this is you know to be fair This is the more accurate one of these four, but modems do not have hard drives within them for the record the on a more kind of personal basis my My experience as a as an engineer on this an investigator on this was I got a call from my boss at 645 a.m. So thanks a lot You know Who here like 7 a.m. Daily stand-ups can you raise your hand? Yeah, yeah, was that your idea? We had a bit of European partner and time zone issues. So unfortunately, I couldn't accommodate everybody's request for a noon pacific So maybe next time it'll be different. I don't know if I publicly shame him But you know so as as our teams, you know a number of different cyber security people got roped into that call because it was an Apparent it's appeared to be a cyber attack. So, you know, every everybody was really eager to help Let's get involved. Let's go dive in figure out what was going on and What we found out very quickly on a practical basis was you know mark mentioned in the case at network, you know That was operated by you know partner that was operating that network So on a practical basis we as engineers and responders didn't even have access You know we had to log into we had to get access to log into that network and that network was actually through that partner Operated as we were later finding out, you know operating other networks as well through that same the same data center And so there was some hurdles that had to be worked through and our partner graciously did Accommodate you know bringing us in as an extended part of the team to investigate and so, you know Here's our hopefully chart, which is you know This is our wish list that we wrote up on the whiteboard of like a couple other things, you know later But you know the initial ones we just wanted to get access to a jump box get us into the network You know get us 20 port 22 access to terminals so that we can investigate so we talking through You know We wanted to be able to connect to a terminal over the air log into it and diagnose what what what was going on for some Of the terminals that were online if I can add there to them I was very unpopular with the engineers because one of the first actions we did was hit the giant red button which only not only Severed all of our management access to k a set but also to other via set networks around the world because we weren't sure If other ones were vulnerable or at risk or under attack So I was extremely unpopular with a large amount of the ops and the forensics teams because I made them crawl through broken glass To get access no comment So I'm gonna take you through our knowledge what happened initially, so what did we know? What did we not know so mark shared that graph earlier? That was one of our initial pieces of knowledge The next thing that we learned was that there are complaints from distributors that some terminals would only light up with a solid White LED which is not normal and then the second piece of knowledge was conflicting Which is that other terminals would light up with a blinking blue LED which was another abnormal case But that they do not provide communications or enter the network and so in both of those cases Those were not normal conditions. I didn't know what these lights meant and so you know first RTFM Like let's take a look at the instruction manual and figure out What does it actually mean when you get a blinking blue light or a like a solid white light? And so if you look on here solid white means initial power up. That's an early stage. That's not good You know the other case that we reviewed was the the pulsing blue case the device is busy But working normally so that means it's booting up. I guess but you know it's busy So I didn't really know more detail than that to begin with and so I'm gonna step back a little bit before we go Into a ton more because one of the things that we had to do that I had to do that other incident responders had to do was get familiar very quickly with a legacy network architecture We've got multiple different generations of different network architectures, and so we had to come up to speed So I'm gonna bring you up to speed I knew a lot of this but you know we all had a level set and so we're gonna level set in this room together on a satellite Communications network kind of this is a little bit biased toward the way we do it But in general we're trying to stay a little bit, you know provider agnostic in this view So typically excuse me typically you have you know customer premise equipment That's like an end user device that connects over a hard wire or Wi-Fi to a terminal or a modem That interfaces usually with a home through a router that's also coupled with that modem Then you have an antenna that communicates over RF to a satellite. I like to think about a satellite as a big mirror I mean it basically does frequency translation from one frequency to another and then you know beams that back down over a different Frequency band, but really it's the same signal and then it goes back down to a ground gateway system Which then is responsible for converting RF Into a Mac layer and then you end up with a core node That's often providing depending on the network maybe layer two layer three services and then you have different sophistication levels of a you know Of your core node as we call it That include different things and so going through those different elements. You have a control plane Which is you know for us a control plane really means signaling to a device, you know, you know communicating with the device Getting the network to function Making changes that's controlling of a network so that might be instructions down to a terminal That might be communication of a terminal to authenticate itself with the network and that's where you see our first bullet AAA So every single modem on our network on the case that network in this case But on every network that we're running has an identity That identity, you know terminals have their own private key. They have their own public key Authenticate with the network through triple a server that is responsible for making sure that when entering the network a terminal is authentic And as a service plan and all of that you also have DHCP DHCP as we know is a protocol that's used for Giving an IP address for example, but many other things as well if needed So you can do configuration of different settings and so there's a lot of different parameters around DHCP That is actually one of the areas will we'll be diving into throughout this Throughout this presentation and then on the management plane side you have other things So you've got device management that might be things like pushing software images down to terminals Through distribution mechanisms through image servers and things like that that also includes You know as I mentioned earlier getting SSH access to terminals We were able to use SSH to servers to terminals things like that to Interface down with different systems within this network and then server at router administration typical things to be able to Manage the different systems that are involved in that core node and ground network And so that's that's kind of what we call those is our ground network is kind of the summary of that little Element and then those connect a user to the internet or in some cases, maybe private networking setups and so That's that's an overall topology. I Will pivot toward the case at more specific network layout and so kind of knowing all that You know now what that what a data plane what a control plane are what a management plane is This network was made out of you know if you look at the left-hand side of this diagram to orient you again Are the users there's users with modems or terminals And those users in the k-asat example were served by different partitions of a network And so we called those bandwidth aggregation points in the k-asat context those are That one been with aggregation point one served a regional set of users Shown on the left those are commercial users back to was also a different set of users And then back the the last one was a viasat operated mobility core Which was also it included you know different systems and one of the elements on this was there's a partner operation of that back one and then back to and then we were Responsible and there's pretty much a different core node network that was assigned to that Management network of the viasat operated one shown down on the bottom for mobility and government cases So how many people in this room know which side of The network the attack came from so I'm gonna ask audience participation Raise your hand if you think that the attack came in from the left-hand side of This diagram Okay, raise your hand if you think that it was on the right hand side of this diagram Okay Everybody's wrong because I didn't see the same people raise their hand from the second. Maybe maybe one person. Okay So I'm gonna go through first Everybody's familiar. I think generally with the reporting around the wiper attack So we're gonna call that method number one the wiper attack. That's been very widely reported So we'll go through this we're gonna go through our investigation of this but the other side of this was a different view of Crafted attacks that came from terminals and we haven't gone through those widely and so we're gonna break those apart And what those crafted attacks looked like which were focused around denial of service activity on the network so We didn't have I'm gonna be fully honest that oh We'll deal with it. Sorry guys The right side of the room is now in the wrong Okay, so I'm gonna be fully honest this this team structure Developed over time as we learned so as I mentioned we only knew a couple things at the very beginning We knew the lights weren't turning on correctly, right? So we had to create these teams dynamically on the fly But I'm gonna I'm gonna talk through these different efforts because there were a lot of subject matter experts Engineers involved that all deserve a ton of credit for the work that they did and so the analysis that we're gonna go through I'm gonna represent the work of a lot of others that were Investigating on these different investigation efforts and so that was our kind of typical like to high-level team layout for these different response efforts So we're gonna dive in to the wiper attack investigation first And so, you know the first the first step that we wanted to do was get terminals brought into the Brought and brought in from Europe and into our Carl's bad labs. That was in California We wanted to bring them in ship them in from Europe get them in and so we shipped them We had to get them through customs Friday supposed to get them customs attempt number one failed, you know There's somebody got turned away through customs and honestly come back tomorrow was I think the answer that happened and so Not knowing all the details our first attempt to get these terminals in the country failed And so we got them on Saturday and laid them out all on a conference table in a room So what we did on this conference table was there were different sticky notes There were different pieces of information kind of annotating these different terminals you mark had requested Kind of do you want to go through kind of what you requested from the the distributors? So since we didn't really understand what was resident on them or what it talked to them We have similar type modems on other networks. Also. This was a completely air-gapped room But we wanted to make sure from a protocol standpoint that we did not tamper with or adjust or Inject anything that wasn't there when we received the modem so I Required them to establish a protocol specifically for example of no transmission pins were being enabled when we did the Reverse engineering or the analysis because we wanted to make sure that what we were Getting his output was exactly what was there and we hadn't tampered with or altered it in any way cool So as these modems were arriving one of the things we were also doing was what what's our plan of attack? Like what do we want to do when we first get these modems and we constructed a plan wrote it up on a big wiki page and Then said okay here when we get these terminals. Here's the very first thing We're going to do here's the second thing that we're going to do and so the first thing that we decided to do was we want to Engage with these modems through console. We just want to pull logs and so we had to do some hardware outfitting to Change these terminals so that we could connect to them through console shown on the video on the right is actually a modified version of UR connection because of one key constraint that we developed in our plan We did not know any state about these terminals. There was no wiper write-ups on the internet Nobody knew anything about this so kind of putting yourself through our lens We did not know if there was code on there that as soon as you connect a tx pin That might transmit into it might wake up and then adjust the way that is behaving wipe itself You know what we did not know what was resident on these terminals that we got in so we wanted to be Very careful not to lose any state because we all know how hard it was to get something through customs So we didn't want to mess that up. So that's a three-pin. You are where you might otherwise see four And so what we first did was we pulled logs And this was one of the terminals where it was a white light terminal And so that white light terminal meant it was an early boot failure What we saw in our experience was that the on the left-hand side the not working version of A terminal actually did not spit out logs past a very early stage of the boot sequence And so what we did was okay that that informs the way that we're going to think about these further our next step in the process That we had identified was we're going to look at JTAG And so we're going to connect to these boards through JTAG. So that's exactly what we did We got our hardware teams to outfit these boards properly. We There are a number of people that made some changes to properly get us You know a JTAG connection so that we can do a boundary scan do a JTAG connection to these boards and to pull Pull the the the chain of what was actually there on the the That was all connected to the JTAG chain. So we did that connected There are a couple of different versions of terminals And so we'll talk about the different variants of those terminals later But we had to do some different outfitting for kind of the different versions of those terminals but the approach that we took Was to modify you boot to dump flash we wanted to see what was in there We wanted to get everything out of that terminal because we knew that like the early stages You know if there's a boot failure the boot the boot sequence would ultimately read from flash We knew that we needed to pull flash content. So that was the first thing that we decided to do from there So after this we started to pull content from flash and we were able to see okay This this is a little bit of a weird set of content The audio will be connected to this Okay, so I'm gonna there is a On this side This is the very first time Where we compared the content of flash from a non-working and a working Bodo and so what's shown on this is we're we're looking at the left-hand side Computer that you'll see here on this screen is a working terminal You can see there's multiple different bootletters and there's multiple different stages of bootletters in a different file system so there's a bootloader file system that bootloader file system included the contents and so That would those are images that you boot would ultimately boot into for stage two and a stage three boot sequence On the right-hand side You can see the flash content of the first stage bootloader and then you can see at the very end You saw it before but as we scroll through this you can see a Decrementing value of the flash content and so this was our initial This is the first time that we ever saw flash content on this board So that's a that that had a couple of unique things about that does a decrementing value. That's not just you know a Something that could have gone wrong There was a decrementing value that was the content for the flash so that looked wrong And then the last thing that you'll see as we scroll to the bottom here There are also no contents in the file systems There's no partition table for either of the file systems that those should then read from and then boot into the next stage of The boot litter and so all of that looks wrong So that was our first readout and that happened we learned that on Sunday night So the attack happened on February 24th for this wiper attack and then on February 27th Sunday night We worked through Saturday to Sunday Thank so many thank yous to every engineer that was involved on over the weekend people were diving into help But what we did was get a good understanding of the content of life One one thing just to note that that Nick just mentioned too is the removal of both file systems That's pretty typical of modems of that generation to have like last known good So both of those file systems appeared to be Dropped such that if the modem rebooted thought it had a problem if it had the ability to get past first stage It would go back to a known good image So it appeared that both of those might have been Then removed or at least altered Cool mark do and take this one. Yeah, I'll go really quick. Yeah so During the weekend while we're doing the hardware analysis on the modems Tool kit was found on one of the management servers if you remember that management plane in that in the cornered the tool kit was a that's the actual name of the file if you see it and it was dropped to us on 342 a.m. Sunday morning. It was retrieved. That's central time and that tool kit contained a number of scripts as well as some I'll say suspicious looking binary files and so We ended up kicking off one of those other threads that Nick mentioned kind of the reverse engineering and analysis teams So Nick's gonna walk you through what we did with the tool kit Which includes figuring out the sequencing of the contents of the tool kit reverse engineering the binary itself And then a detonation step to actually verify that it produced the same kind of outcome that you just saw on there and in Parallel we actually as part of our standard incident response mandients our forensics and investigation partner they convinced On March leaf Monday, February 28th is when we engage them And I think the very next day is when we gave them contents of that as well So they could start their investigation and forensic analysis cool so as I mentioned there are a couple of key questions and and we didn't know a ton in the very beginning so we had a number of different questions about The tool kit that was found and we had we had a lot of questions and a lot of those questions affected the way that we were thinking So some of those questions were what other capabilities did the tool kit have did it have evasion capabilities? Was it selective over targets? How did it erase flash? Could anything be done to stop it? Could walking dead modems be recovered could itself propagate? There's a lot of different questions that we had about this tool kit So we wanted to understand what the what what was actually there? So this was we spun out our tool kit reverse engineering team which consisted of a gentleman I really do want to give credit for credit to Jonathan Wyatt and a couple of his colleagues Andrew Lamar These these gentlemen were able to quickly use an NSA open-source tool Ghidra to investigate We'll go through the the investigation that they did in a second from that tool kit But one of the first things that they did before that was to do an emulation through Camu and so they installed the One of the key binaries that was found in this toolkit I'll go you'll see an image of what was contained in that toolkit on the next slide But they emulated the way that this malware or this binary would actually behave Through Camu initially and so they were able to see a distinct pattern We recognize that pattern from before from what was actually recovered from the modems that were in the field And so another key question that we had was was this malware? Actually the toolkit or was that a decoy? We didn't know so we had to pair those two things together So on this slide This is a picture of what was in that toolkit from a summary perspective So as you look at this toolkit, there are a series of files and scripts all kind of centering in around use of SSH and SSH credentials to Distribute an executable down to a terminal one of the key things that I want to highlight about what was in this toolkit Was there were a couple of different ways that these attackers were? You know interacting with their scripts. There is a series of files Some of them had gotten changed initially it seemed like and we're speculating Initially it seemed like there was a hardware list We don't know the order in which these scripts were were executed But there was a hardware list on this network based on what everybody in this room now has seen There's a satellite communications overview. This is not one layer to broadcast domain So that you know doing an art-based look up for terminals doesn't actually even work on this network So looked like that attempt was then you know changed into it in favor of a different attempt And so there was a series of different Scripts and files that were all centered around distribution of this Executable down to a terminal and so we wanted to figure out what was actually in that Well, so we want to figure out what was actually in that executable And so as I mentioned we use the NSA tool Ghidra to understand what was actually in the executable What did it do? How did it work? Does it make the same pattern? And so one of the things that we quickly did was it takes if anybody's used the tool before you have to Spend a lot of time you know renaming functions, you know understanding syscalls understanding numbers And so this is kind of the derived version of syscalls mapped back up into an understanding This was created through an internal document By the the internal document was finished on March 2nd Really proud of that team for their work The next thing that we did was we also let's detonate it Let's let let's take let's get some sacrificial terminals And so one of the things that's really cool about this Event was the way that the teams rallied to be able to respond We were able to quickly acquire terminals from a lab that we had on a shelf and Detonate so sorry terminals you guys You're the unlucky ones Some some of these terminals were then pulled into our lab our response room And we detonated we distributed the the kit down or just the executable down onto those terminals It was a MIPS binary We executed that MIPS binary on the terminals and we're able to see that it does actually have the same effect as what so so now We had three different independent signals that all correlated together to understand that this toolkit was in fact the toolkit That really did help us to understand what was actually there what what what everything was So just to kind of summarize our initial attack conclusions We went through and developed an understanding of you know the attacks happened at three to four a Mutc I'll go into the other attack which happened around three a mutc on Thursday 224 Modems arrived on Saturday Sunday night. We had discovered flash content We had read that flash content understood it and then our teams then were also able to reverse engineer the toolkit and Understand it over the series of days. They were kind of streaming in information on that That reverse engineering process because oh cool. This is what I found Okay, it's it's really looking similar to that and that internal final publication of that document or that final document was Was kind of created by the the mark second time frame Kind of put the conclusions to a picture For us Based on the forensic analysis and a little bit that came afterwards But essentially attacker through known tour node exit relays Unauthorized access to that VPN with compromised credentials Pivoted to a management server which actually a different set of compromised credentials Went to a network ops server to do some recon during this time period That's the stuff about trying ARP and hey what what modems are alive or can I reach which parts of the network? And then pivoting to the place where we typically do distribute software images but just using that particular place to stage the toolkit and This is it kind of pictorially from a timing perspective what we saw actually The first attempts were about five or six o'clock in the evening trying to get access to the VPN at several failed log-ins first successful compromise with unauthorized access about eight o'clock in the evening then Took about two hours. I don't know if the attacker or attackers needed dinner put kids to bed Whatever went to a management server hung out there for a little bit Then went ahead and said I'm gonna go check out the network right around 10 or 11 o'clock that evening and then about 1130 through 1 a.m. To 2 a.m. Went to that that server Started staging the toolkit and then started executing attempts But it looked like about 0 4 15 as you saw the arrow with the precipitous drop is when Essentially sent out this over the satellite network just going modem to modem down the row and sending the toolkit Placing it on the modem and then executing it once the binary was executed It performed the flash memory overwrite that you saw with that pattern and then attempted to reboot the terminal Which would put it in an inoperable state Okay, now we're going to talk about the other the control plane attacks if you remember these started at roughly 0 300 UTC and I want to emphasize while they were going simultaneously Throughout that morning. There was some intermittent behavior And they also continued for several weeks as some of the methods and means that were being used We had mitigations deployed they would pivot to a new technique So I think we've seen this slide already just to orient now We're talking about the second method and so this this sec the second method was crafted attacks that came out of terminal So we're going to decompose that and what were those crafted attacks and what when we learned from those We have since gotten the chance to apply a lot of all of these fixes on to our networks And then I've also been working to generalize the understanding to say okay What other attacks could look like this and so that's what we want to get out there and inform the community of I know I'll add to that This is one of the places that through private sharing forums and direct communications with other satcom and service provider This this is a more I'll say a more similar pattern that other service providers might use to provide control plane or signaling or IP addresses So even though these are specifically manipulating things we use in our network certain DHCP option fields, etc We thought it's generalized enough that we should share So this is some of our earliest sharing both directly with that com operators as well as through our government sharing partner One thing also to call out on the prior slide is that this fell then into the other track that we had formed as part of our incident Response which was the network and infrastructure analysis and response teams And so those teams were a series of subject matter experts and security engineers that were all kind of responding ultimately to these Network-based attacks and so this was a distinct track from the others But we were generalizing the learnings and understanding and aggregating them into a composite picture the whole time Yep, so as we go through this we're gonna look at this first attack example So in this first attack example, we've attached a timeline to Initial DHCP request volumetric attacks that we saw starting at 3 o 2 am UTC on February 24th If you recall the prior graph that mark mentioned you saw that initial drop-off and then you saw a steep drop-off This was around the same time of that initial drop-off of Terminals when you started to see that was due to effects to the network entry process They were caused by these crafted attacks coming from terminals and Two other points. I just want to make on this one one if you see the legend in the top right I mean these are legitimate modems with valid subscriptions all throughout that beam pattern that what there was no Localization like they were all in one place or on these are just essentially random modems that appeared to be doing this The second thing to note on there's that gap in the middle. It looks like it kind of stopped Well part of that was due to the instability in our network because of the volumetric attacks Some of the elements that Nick's going to describe stop the responding crashed reboot Analyze take it offline take a note out of service Cool, so now I'm going to dive in I'm going to decompose a control plane a little bit more than we have so far So you have a customer premise equipment. You have a device of an end user. You have a terminal Then you have a gateway so that terminal communicates ultimately through this satellite through the gateway And then that gateway makes a decision depending on whether that traffic is control plane or data plane And then that that traffic ultimately flows through an asn is a role Within a system architecture for the network. This is a virtualized asn That then that system then acts in the DHCP context as a DHCP relay That's an extension to the DHCP RFC And then there is DHCP servers that are facilitating DHCP requests and responses There's a triple a as I mentioned for authentication and servicing of that and then there's a couple other systems that are involved so As we decompose this this is a little bit small It's an eye chart, but I'm gonna I'm gonna talk you through this And so this is attack number 2.1 of three different attacks that I'm gonna decompose that we saw happen on our network That were these crafted attacks from terminals so the precondition for this attack first is that a terminal must have Valid access and an identity within the network. So you must have a terminal specific private key You you have to be authenticated within the service the service. Yeah, you're real terminals It might have been stolen or haven't been provisioned yet They don't yet have that kind of key and they go to a walled garden for an installer to go install it So that you can't get to this place in the network if you will unless you're a valid subscribe, you know A terminal with a valid subscription Prior to that if you hadn't paid your bill or a service was suspended triple a would send you somewhere else So this is this is a pretty important point. This is a Authenticated itself to the network using its key and it's got a valid subscription So kind of from that precondition now you see the next you have a user device Which issues a DHCP request up to a terminal and then that terminal then since that DHCP request on This is a terminal that in this case is not acting as a router There are sometimes cases where if you've got a router involved then the router might act as a relay for that so this is the case only where It's assigning an IP address essentially to like think of your home router Basically the home router gets an ISP IP address that is that request right so that DHCP request is then filled out It's sent across to the gateway the gateway then propagates that to a VAS in And then that ultimately then gets relayed to a DHCP server So in that DHCP request in the DHCP RC, I think 2131 It defines a client address as really like it's the CH outer field is essentially an opaque key That opaque excuse me that opaque key Can be anything that you want. It's essentially a hardware address in this context for a modem That then Might include there's kind of two variants to this attack if you want to execute it one would be to Set your client hardware address to any And then associate a requested IP address, which is DHCP option 50 with the IP address of a target terminal So this is what we saw from the attacker where there is IP addresses of target terminals that were in these volumetric attacks You could also when we reviewed this you could also do the reverse And so when we have since then applied that into a patch We've had to think about and generalize what are the other cases that we didn't see but could also then be used and then Kind of winding ahead to the DHCP server You can see a DHCP request to a DHCP server that says the request is invalid It doesn't match an active lease and then a knack is then sent from the DHCP server to the VAS in That VAS in system then is you're responsive to that knack It shouldn't have been in this case. That was the problem So the VAS in takes an unsolicited knack and then is confused into Issuing a command on that control plane to disconnect a terminal from the network and that ultimately would knock a terminal offline So it was a fairly sophisticated way to use multiple different essentially DHCP system roles to confuse multiple systems operating together and so if you think about this within buzzword zero trucks of trust context you have to think about What are the ways that you might be able to? Confuse systems where the DHCP server is implicitly trusted from a VAS and so you have to think about how might that affect something else And how can then you ultimately prevent that from happening so I can wind pretty pretty quickly through the rest of these You have a second variant of this attack, which is a DHCP decline attack, which we saw Which is ultimately issuing a DHCP decline for a target terminal That then causes a DHCP server to really not care about that that actually confuses a VAS in to then disconnect the terminal This was we responded to this and then we're able to just disable this request And then lastly same exact attack, but through a DHCP release method as well So this is a bit of a cat and mouse game from a response standpoint. You know, we saw one attack We saw a DHCP request attack We saw a DHCP client a decline attack and then we saw a DHCP release attack every time we would make a change We would then see You know within days response and then within hours as we got better at it And then the responses would change over time and then later we applied this into the threat modeling context And then ultimately hardening of these different systems so I will say just The attacker or attackers also displayed similar Responsiveness time when they had definitely had ways of testing and viewing and observing what changes or behaviors that we made and they responded pretty quickly So what specifically did we do to defend? Wouldn't an adversary like to know but the things I will share here is That we you know quickly applied filters to control plane systems to intervene disable these requests We stood up a dedicated control plane slice We had engineers dedicated to this to set up a dedicated slice That was a Ukraine only control plane slice to keep that segmented from the rest of the networks on the fly And so the control planes were then separated at that point That was a pretty prideful moment when we had that new control plane slice online and you know dynamically responding to those address And then lastly I'm gonna just kind of give a shout out there have been a lot of different engineering teams that have been involved in RF attacks This is not so we've seen attacks from different cases of it, you know attackers That are out there around You know RF attacks as well That's not it's not these just these domains that we've had to worry about and so this has been a very ongoing Series of events for us. We've seen eight different changes of RF interference over the last year So this has been a pretty active domain So our ask yeah Yeah, so our ask, you know service provider networks, especially calm or especially complex You know Leo Gio, whatever that is, right? They're they're complex systems So as you think about those, you know, we know what complexity introduces in the community, right? That that introduces vulnerabilities And so you have to think about the system complexities And so one of the things that we want to share with this community is our ask Slide our ask we we have a traditional bug bounty program through bug crowd We typically have the campaigns that are our kind of user-facing things like billing systems or customer support portals in some cases It's our corporate infrastructure so we're announcing an expansion for the research community kind of if we get vetted researchers through bug crowd We're gonna ask for some help by providing terminal kits and service to Selected researchers to be able to test over-the-air methods and attacks for disrupting a particular canary terminal in there This is the kind of thing that we would love to engage the research community to think of ways that maybe we haven't considered yet to manipulate or interact with a terminal so basically interest in resume portfolio for our responsible disclosure program will take on submissions in September kind of review those and select some Select some researchers hopefully to have some fun with our network So I want to thank everybody for their time and attention today. I appreciate it