 Tom here from Orange Systems, and we're going to clear up some Misconceptions like that you can stop a DDoS attack by putting Cloudflare in it And I know that but I don't know the next answer of exactly how you stop a DDoS attack But this guy does Ray or Cine how you doing Ray? Good Tom. How you doing man? so Ray is an expert and He runs OIT VoIP and Ray is the reason my phones are ringing now Which makes me happy, but I was actually joking that it didn't make my staff completely happy You're like, you know, we did kind of like it when phones didn't ring But I did remind them if the phones don't ring the checks probably won't cash either. So, you know, we sorted that out pretty quick Clients expect to reach you for some reason So this is kind of a follow-up I did a video the other day on VoIP MS just kind of explaining what we know about the situation The only thing we know more about the situation a few days later today is September 24th 2021 is that it's still happening which sucks. We don't have any inside information But I did want to you know a lot of questions came up on that video like how do we mitigate a DDoS attack and things like that? So we thought we'd just clarify how that works how some of it works Ray's gonna share his expertise on it But you know in full disclosure. We are a active customer of rays. We were using Ray before I've been on his channel tech bar, which by the way, I will leave it down below and Summer ray staff has a problem with me because he made them do some hot sauce That was unfair you had an unfair advantage Yes, I was chewing Sean was chewing and you didn't tell us not to chew that would have made all the difference So if you're wondering about a hot sauce video, we'll I'll leave a link to the tech bar I was on and of course you could find all the other tech bar episodes It's a lot of fun if you want to hang out with Ray and them but also the hot sauce video and things like that So we've been working with Ray for a while and Ray is a little bit different what they offer at OIT VoIP They are a full service as I would call them but company where they do full deployment and everything else They aren't just a sip company, but they jumped in to help us because well The situation sucks and it's you know, we're not going to throw VoIP MS under the bus or anything like that We just don't know what's going on there Outside of we know as much as you the audience knows of whatever has been posted on Twitter And we're not here to arm-chair it but I thought I would do the clarification of Exactly like how to mitigate it and Ray was kind enough not only to come out and hear and talk about it but go a step further and He used my favorite program and draw and he drew some cool graphs So we'll talk about that mitigation and clear things up related to that. It's a Little bit more in-depth because when you take a mitigation from something like Cloudflare or put something on a high-powered content delivery network when you're Backing up a website. Well, that's easy. It's static content in it If you know if my website was being attacked I could throw it up on Cloudflare and replicate my website and my static content would be delivered But that doesn't really work with a voice phone call does it Ray? No, not at all and that's the problem with real-time communications because anytime you add anything else in the mix You have latency. So while like WAPMS now has Cloudflare in front of the web portals And it'll ask you if you're human and all that stuff you can't really do that with a phone call So it's much more complicated. Yeah, there's a there's a Just a lot of confusion. It's because whenever you're managing real-time applications So for what you learn here today You can also extrapolate this out to really any type of real-type application service A lot of these things apply to it if you have something that needs that back and forth communication You can't just statically put something in front of it We're gonna keep it focused on VoIP because that's we'll raise a long-time network engineer But currently president of VoIP VoIP so there's a lot of words expertise is but he understands this deeply I understand that it can't be done, but Ray's gonna tell you how it can be done So where do you want to start Ray? You want to start with the graph or do you want to talk about some of the services? No, I just I want to talk about like some of the ideas and the concepts that go beyond first before we get deep dive into the graph Okay, and just some my background my background is in complex and distributed networking. That's what I've been building for 20 years So multi-city multi-site multi-continent. Those are the networks. I've built my entire life. That's what I love doing But like you said like when it comes to real-time applications voice Video anytime of that, you know real-time messaging. That's much much much more complicated and you know one of the reasons I was excited to get on a video with you is because There's so many misconceptions going on right now of like and people asking questions. What can I do? What options? Why don't they just do this? Why don't they just do that? Right? And again, not not just VoIP MS I want to be real clear. It's just like a breach. It's just like ransomware. It could happen to anyone, right? This is just they got hit it really, you know, it's really unfortunate and our sympathies go out But with all the people that are affected and the questions they have we want to go over What really happens the way what really happens when the stuff happens and what what options are available to them? I just want to comment that a lot of this is referred to as mitigations And that's well the reason that words used as opposed to this solves the problem magically Which if anyone tells you that to throw them out the door or march that salesperson right back out the door The reason they use mitigation is because you're mitigating some of the problems But you're not absolutely if the pipe is big enough you can get taken offline I remember I think it was Brian Krebs a couple years ago Someone really hated him and he was getting one of the largest D-DOS ever attacked against a single individual He said he set some records when they were trying to Mitigate it and even though he had cloudflare cloudflare I think passed them off to Google to avoid it So any one of these are it is a scalability problem and as more of these people won't patch stupid things on the internet The botnets get bigger and more powerful and toasters are gonna be the destruction of the internet So we don't we don't know the numbers behind VoIP MS, but You were bringing up the one for the UK, right? What was the number you said for their throughput or 50 gigs and I that was quoted by Steve Gibson on the last episode of Security now where he talked about the VoIP MS attack in the previous episode. He had mentioned how big the Marist spotnet was I don't know if those are coincidental things that he announced one week the biggest botnet Of course bleeping computed it as well. And then the next week We're talking about a large-scale attack on different VoIP providers And I I don't you know, we don't know until the only people know who's on the receiving of this is the people receiving it but either way these these botnets are dangerous and They cannot companies offline, which is what brought us to this conversation And that's the thing right like when you're scaling out your you know purchasing a firewall and you know You're talking about throughput for your IDS IPS and whatever other filtering you're doing They have limits to what they can handle and so when you think about you know That's the thing that comes up now in MSP world where you know, you have multi gig available to you at a client site nowadays It's not that uncommon But go find a firewall that can actually filter in real time that kind of throughput right now when you say Even when you get a data center, you know, we may get a 10 gig a couple 10 gig pipes or multiple You know or or even a 40 gig Most of the connections that most providers are using is 10 gig at 40 gig in a good day So when you say 450 gigs of throughput, you got to imagine the kind of equipment required to just to be able to filter that and That's what that's what we'll get into on the diagram which keep in mind that kind of what's happening with the DDoS Is it's just bombarding your equipment where it can't do anything because it's it's stuck, you know, yeah I would absorb it essentially Yeah, exactly. And so and just like anything else, you know When you have real-time filtering and if you do email filtering, for example, right a lot of the email filter providers They receive the email first they examine it see what it is And then if it passes their test then they pass it over to you that prevents that presents latency Right where the email gets you a minute or two later, whatever it is and that's the problem of weight You can't really do that with weight because you know, you can't the conversation will be delayed and the user will notice We have milliseconds to respond anything to give you an idea. Good. VoIP is under a hundred milliseconds There are acceptable VoIP is under a hundred milliseconds. Good VoIP is under 30 milliseconds So every little fraction of a second really makes a difference when you're talking to somebody but if you want to pull up the the diagram Says first I want to go to go over. How does VoIP actually work, right? Because like there's oh, yeah, there's some confusion there And so at its core, it's pretty simple You have your phones and they'll register over to whatever VoIP provider you're using, right? And they may have multiple Locations around the world and that kind of stuff that's pretty common and you're gonna want to pick the one That's closest to you again to avoid that latency, right? And then they'll do their call processing and all that stuff all inside here And then when you're gonna call somebody that's outside of their network, right? Because on that if you're calling another customer on the same network, it'll stay there But if you're gonna call somebody that's you know calling a cell phone calling something like that That'll go over to what's called the public switch telephone network the PSTN is what we'll call it and Those connections more often than not are happening over the internet too, right? Most of the most of the Clex the competitive local exchange carriers the people that actually own the phone numbers They have internet they're on the internet just like everybody else and they may have private connections public connections but they're gonna have their connections out to to these providers and They're going over network resources. They're it's IP transit just like anything else So as you can imagine every one of these in the VoIP world, they're called switches in the telco world They're called switches Switch is something that's actually switching Calls from one resource to another adding in applications like your auto attendant your call queue your music on hold So these soft switches that are processing the calls You're connecting to that over and of course you're gonna have their routers or whatever they have here and then Trafficking it there and they're gonna do their job and send it off. No big deal but what's happening is in the With the DDoS attacks is you have all these other people doing the exact same thing Right, you have all these other Agents sending traffic to the same routers Congesting the path and you know where they may have a couple one gig pipes a couple ten gig pipes or whatever it is There's that these guys, you know, we take the UK for one because that's the only one we have the real numbers for They're sending 450 gigs of traffic through and even if that was split out to 45 gigs per pop or whatever You want to call it it's saturating the rest of your calls can't get through now remember what I said you have It's that the carriers that are you know, the public switch telephone network to get out to dial a cell phone dial You know somebody else's line on another phone provider. They're also using those same internet connections To the same soft switches. So if the call can't get if you can't connect to it They also can't connect to the psd. Now. There's other strategies. I want to be clear You can have private separate links, right? Like if you're building a sand You're gonna have different now internet connections or different network adapters for the data Separate for the storage separate for the compute You're gonna have different things and there are strategies like that for VoIP I'm just talking about this general because I don't want to get really, you know into the weeds on it But at the end of the day, it's this it's these connections here these giant traffic These giant blobs of traffic are just clogging the pipes for lack of a better word and none of the traffic can get out So when people start saying well, can't they just forward my number? Well, how can they forward the number if the call switching the call routing that happens here? They can't change anything because it can't send the traffic it needs out to the PSTN if they can't say okay Well, and what a common strategy would be is, you know, take this phone that's going over here to pop 3 Move them over to pop 2 But if you saw Tom's last video where he brought up all the pops and you know those public IP addresses are very easy to find it's not and What's happening with the current attack is they're just rolling through the different pops and they're just you know They're taking one out and then they're moving to the next one. They're moving to the next one So you may have good traffic going on at that point in time and an hour later. You're you're slammed again Yeah, it appeared to be kind of like a rolling attack across where people say it's up It's down and I'd seen people showing different plots and things like that It's kind of maybe they don't have enough bandwidth to attack one because they have certain mitigation that can absorb X number of Connections per second, but that's why they roll it now between different ones because you're still disrupting the service and They probably do it based on time zone like you know, they know that these ones are more busy in the eastern time And these ones are more busy in Pacific time Whatever they can do to cause the most pain whoever's doing this. That's their goal is to create pain Well, and they're very smart about it, right like we saw that With the big, you know MSP hack that happened over 4th of July weekend They were planning to do it over the weekend because they knew it was gonna You know people were gonna be off and nobody was gonna be working that Monday or whatever was it just happened to go early You know and so these these attackers are very very smart and I agree with you I think they're looking at time zones. They're looking at the most active times. They're doing this and again We don't have any insight. We haven't contacted any of it's pure speculative But it is to me. It's clear. This is being done to make to cause pain This is being done in the way that is the most destructive method possible. This is not a you know Oops, we took over your your oil pipe by mistake. We really didn't mean to my bad. This is a this is absolutely a directed attack and Like I said, it could happen to anybody, you know what I mean It's just you know, and that's what you you want to be careful with but there's mitigation strategies, right? Like that's what the people really want to know that what can I do about this? so With the mitigation strategies as much as I'm sure Tom and I would like to get really really really technical about it It's not as complicated as you think it is basically putting a referee in front. It's saying it's having the referee say No, I'm going to Scrub all the traffic and that's what they're called. They're called scrubbers and they're these I should probably bring that up front They're these giant big iron routers with these multi hundred gig or bigger connections aggregated with these ridiculously fast asics with the sole job of Identify the traffic filter out the bad traffic let the good traffic pass and so when it comes to the boyp world The common strategy is Instead of having these what are called pops these points of presence Whether it's a soft switch whether it's their routers whatever it is instead of having them directly exposed to the clients right as we had here The phone registers directly to the pop What'll happen here is the traffic will go through the scrubber and the scrubber will say this is good traffic This is bad traffic. It'll identify it'll put stuff on RBLs It'll black hole routes once it identifies. Okay. Well IP one dot one dot one dot zero One there we go. There you go. It's just DNS whatever. Um, they can handle it, right? So but what'll happen is okay, they'll identify this traffic's coming in it looks to be malicious It's from this IP address We've also seen the traffic from all these others because these are all have working in tandem and when I show 3m These companies that do this there's really big companies to do this. There's Arbor we talked about and that scout owns them Path.net Carrero their cloudflare is the biggest most common one now cloudflare doesn't do VoIP That's why we're not talking about in this context, but These giant enormous scrubbers They're talking to each other as well. So there's this orchestration that says okay Well scrubber one identified traffic from here. That's bad. Let me take that same IP list Or let me take that same ASN or let me take that same prefix And I'm now going to tell scrubber three or scrubber two give them the data So they can also practically block it and they get faster and faster and faster And that learning period and that scrubbing period. That's why we call it mitigation. We can't do this all the time Because now imagine you're passing traffic over to the scrubber then over to your wave network and the reason VoIP providers like myself have so many of these pops is we want to be as close to you as possible And anytime you introduce an extra data center an extra hop You're going to introduce some latency and you know the potential for any kind of route congestion any kind of anything else that could happen But what happens is, you know, these things are ridiculously good at what they do Especially the ones that are designed for VoIP They'll scrub the traffic and instead of having these devices these soft switches on the public internet We'll have g re tunnels typically um I'm gerry tunnels, which is basically it's an encapsulation thing. It's basically like vpn, but different But I'm gonna do a video on g re because if people ask about it I want to do a video on it because it's there's some good use cases for it like this. Oh, absolutely. Yeah, absolutely, right? Um, doesn't provide the encryption, but it does provide the encapsulation And a lot of times you can combine ip sec with gerry, but that's a whole that's a different looking forward to that video Yeah So we'll have established gerry tunnels and so what'll happen? We may be hitting that 450 over here, but all that's really being passed over here is the good traffic So maybe that 10 gig or that one gig or five gigs or whatever's coming through And then they're able to go out to the pstn And they may have a private connection back to it to the pstn carriers Or they may have another gerry connection or even if they're just going back out to the scrubber to go back out to the pstn Not a big deal because the scrubber is cleaning all the work for you It's your security guard and it makes life a lot easier I hope that was as clear as clear as mud You know what? I think it makes a lot of sense, you know the data coming in and the other thing too is As you do this the learning period that these go through And they build these blacklists like how they are identifying things as I mentioned earlier like the merris botnet those are a bunch of Specifically microtik devices make up that particular botnet And the goal is to collect all those addresses because these are usually solved on static addresses And these are fairly powerful devices the microtik have a relatively good processor So it allows them a lot of attack and as you said the collectors are doing all the collection and burning that so once those Things have collected and learned all the bad IP addresses The only thing left is the good IP addresses coming from clients now Hopefully they're not one and the same. Hopefully the client's not part of the botnet, but I bet that happens. That's probably fun They're like hey, by the way, you're sitting behind once or twice. Yeah, you're sitting behind with these infected devices that's doing the attacking That's a weird coincidence Right But yeah, and then you get that email from your isp saying hey, we notice you have port 25 open and sending traffic Or are you doing a dns reflection? Yeah Bad stuff and and what happens with the clients. I thought I should bring this up real quick um what happens with the clients the reason All this can happen in the background the clients actually don't have to worry about registering to a new place is because And this is where you have to have some preparation. Uh, this is not just something you can do on the fly You know void providers such as myself We do plan this stuff out just like, you know, whether it's an ir strategy for anything else You have to have your plans for this and what will happen is we will have we will use bgp and Um these scrubbers or these ddOS mitigations will start announcing our prefixes So when you go and you resolve pop 1 dot void dot network What'll happen is instead of resolving over here It'll resolve to an ip address And that ip address instead of routing over here it'll route over here that way they're getting the traffic We're not telling the user. Hey now you need to register to pop 2 instead or pop 3 It's completely transparent to the user. Um, and that's just built into the way the internet works. Thank you bgp and Yes, yeah, and essentially what's going on here is you can slide these in with the bgp route announcement So you you've prepared this you know what the route announcement is to put these in You know what the route announcement is take them out And as you said in the beginning to keep the voice latency at the absolute optimal Which wipe is kind of unique in this when you're thinking about your presence Voip always has to have almost like a city level wherever they have a lot of customers You end up with one of your points of presence being there to keep the latency at the absolute lowest And you don't leave this turned on all the time because once again You want to keep the latency at the absolute lowest? But this is really where the planning comes in phase to make sure that well We see it happening guys flip the switch and do a bgp route announcement if you dig around on my channel Um, I covered the verizon. Do you remember the verizon pittsburgh? Uh incident. Oh, yeah Yeah, yeah, I covered how that happened and cover a little bit of how bgp route announcements go wrong And it happens too all the time where like somebody else will announce the wrong route or You know when you're announcing you're announcing routes They may announce a prefix that didn't belong to them. Yeah, and so stuff will start getting routed through croatia or through whatever else Yeah But yes, it can Yeah, we could get a whole the whole another topic and I could bring some more people You should have like a bgp round table would be fun because I have some friends stories I've learned a lot from them and I'm not I don't even I'm not the bgp expert, but I I'm talking to them. They're pulling their hair on a regular basis But nonetheless it's gotten much better than it was a couple years ago And once you have this in place you flip the switch the bgp route announcement by how long do they take from from I see something going on and you flip the switch. What's the time frame? We're looking at ray from uh, these not installed So the announcements you can do them as quick as is 60 seconds. Um It is very very fast as long as you have everything Prepared in advance, right? Um, they need to know what your ip or what your prefixes are which your subnets. They need to know what you have They need to know, uh, who you're peering with They need to know what the acceptable traffic is what the data centers are using you have to have the g re tunnels ready to go Um, those g re tunnels typically stay live 24 7 The differences were just not routing traffic through them until something bad happens Um, and this is not really different from what how an isp handle sings either. It's not uncommon for them to take an uh Taking a subnet and instead of announcing it in their cleveland data center They'll announce it in their you know datan data center. It happens very very often. Um The difference is with void. We don't move things around nearly as often just because You know we want things to stay nice and stable and it's you know, if you're google.com takes a little longer to refresh You're not going to notice Um, if your phone call gets choppy you're absolutely going to notice and we're going to get a phone call or an sd email or something Yeah, exactly. All right. So we got this part covered now. What's next? So one of the things that um, I kept seeing online, right? And I know we talked about this both of us kept seeing this Is well, why can't I i'm just going to get a backup sip provider, right? I'm just going to get a backup way provider That way I don't lose my phone calls Which Okay, this is one of those things that maybe right and it doesn't really work that way because what'll happen is just like, um And I don't I don't want to get super deep dive into this but just like ip addresses have their bgp routes that Or which are how you find that the ip addresses and just like dns. You'll look for dns and dns will resolve the ip addresses Dids phone numbers right and the north america numbering plan is 10 digits, right area code Exchange and then the last four So what'll happen is those phone numbers have what's called an lrn It's a routing number and the and when you look for i'm going to call tom and you know one one one one one one one one one What's going to happen is we'll do a look up and we'll say okay Well, what rate center does that belong to it equates to this lrn belongs to this rate center This rate center is currently pointed to verizon. Let's call it right So and then verizon being the sip provider for you will send the call to you You cannot move that number you can't say okay. This number is now going to be on You know bob's telco without a porting process where that we basically it's telling the lrn to now point to Bob's telco instead of verizon So when you go through porting you're moving carriers. That's what happens Um, if you had a second void provider say you had verizon and you had bob's telco You can't get your phone number to just magically come through bob's telco if bob if verizon's getting slammed That number is not going to work. You can get a second sip provider for doing outbound right That's not a big deal. You can absolutely send the call out That's not going to help you with people trying to get a hold of you. So If you wanted to say hey, this is my temp number You know call me on this number. We can still receive to the office. You could do that It's not much better than just saying here's my cell phone number Which honestly it's what I would do um And now i'm only saying that that's only for local phone numbers. Um, so Um toll free numbers are a little bit different toll free numbers are a lot like bgp a lot like dns and that toll free you can actually modify Modify their route every five minutes um, which is why you know We got very lucky when we were helping you guys out and and moving your numbers We happen to have a lot of the relationships on the back end with the same underlying carriers So we were able to move the numbers the same day Um, but with toll free that's always the case toll free. You can move it like that That's actually one of the benefits of toll free why it's sold to enterprises Is because they can load balance calls on toll free and they can actually migrate them to where it's most cost advantageous for them or where it's more advantageous for them because of Any kind of provider out a journey like that. It's actually one of the really cool things about toll free It's one of the reasons we liked it years ago and it worked in uh enterprise With that's one of the reasons we set the toll for you is because uh, we had some call capacity issues that we run into And that's how we would mitigate those and solve them Yeah, it's it's weird because d i d's local numbers right d i d's direct and we're dialed but it's the common name for local numbers Um, they'll be tied to one provider with toll free numbers Especially in the enterprise space and the marketing when you're in the marketing and sales space It's not uncommon for a toll free number to hit multiple carriers at the same time. Um, that it's actually pretty typical Um, it's one of the things a lot of people don't realize when you call out We're actually trying multiple underlying carriers. We're doing real-time calculations. What is our lowest cost route? What is the route that's going to have the least amount of? The least amount of hops or that's going to get the fastest connection And so like when you make a call and you hear it's a little bit quiet before it starts ringing That's called post-style delay. That's what the carrier is doing on the back end when they're seeing. What's the best route to get there? Um and toll free you just have a lot more magic you can do. It's just a lot easier That is cool All right, do we cover all of it sir or is there something else we want to throw in here Let's see here. We talked about d dos. We talked about um voice applications Uh, we talked about real-time We talked about second providers um toll free Uh porting out. Um, that is okay. I want to be really cautious about this about how I preface this because A lot of the providers thought Mike and I'm I have to save wait by mess here. I'm not calling them out I want to be clear on it But a lot of the people that I saw on the twitter feeds were saying I can't even port out because their services are down That's absolutely not true. Now. I want to be clear I'm not saying when your carrier has a problem jump right jump ship right away In most cases porting out takes a couple days or a couple weeks. So, you know, there's a negative return and They may fix it. So I'm not trying to get anybody scared and saying you should port out immediately But the way porting works is there's a central repository M pack is the one that's responsible for the phone numbers and the phone number routing when you say You know, I want my numbers to be on tom's telco or bob's telco and so That company is completely outside of VoIP MS or anybody else So that porting process does still work now You may need a copy of your bill wish if you can't log in the portal Or you can't get a pin number or your you know, that stuff is fine Even if they're so jammed up that they can't respond Because there is a timeline they have to respond to port out requests With the way the FCC has has set it up They don't get a pass to not respond to port out requests because they're having a DDoS attack That's not the way that works the FCC truly and honestly doesn't care Yeah So they do have their timelines to have to respond Most in air in most cases if they don't respond the number is going to port out automatically no matter what So if you're in a position and again, I want to be extremely clear I'm not saying telling everybody I'm waiting for message. I'm ship. It could happen to anybody Right, I'm sure they're working night and day to get a resolve. That's not what I'm saying here But so you know what's available to you Regardless of what whatever incident happens. You always have the ability to port out no matter what Because I saw that come up over and over again and It's more about telling that process and making sure people are clear that you can do that And this is actually one of those questions that's come up before And once again because I I did some My first phone system I worked on was a 99 on nortel systems and things like that But so I they're probably still running Yeah, they didn't they never die But um, they turn yellow, but they don't die. It's all they do is kind of admit that weird yellow patina But they uh, nonetheless the one thing that is come up for us like well What if you're with this company and they go defunct or whatever and you can still get your number back out Because that's actually happened over the years Um, maybe one day me and ray will talk about some more stories I have a crazy one about being involved in a voice company around circa 2005 one of the big voice companies It started up and fell apart because of some funding problems. But anyways, you can get all your numbers back out of it Um, it kind of it defaults. Let's say that company any company. Um, it just doesn't respond for reasons unknown Eventually the it's kind of like fail over to the inbound porting where the request came from correct So that's how it works Yeah, the local number porting act local portability act Local number portability act. I'm sorry. You're gonna have to edit that We were doing our best to not have to edit anything. Um, the the local number portability act is very clear In most cases, there are very few there are a handful of exceptions But in most cases The user of the phone number has the right to move it anywhere else they want I'm not saying they are the owner of the number. The owner of the number is a whole different thing That belongs to the rate center that belongs to the spin. That's assigned by by n pack But the user of the number has the right to move it however they want to wherever they want And in most cases as long as there's you know, it's not some ultra rural telco That doesn't have ss ss seven interchange or whatever Um, in most cases they can migrate the number and it's not a big deal. Um, so you always have that available to you Yep So hopefully this cleared up all the questions and now this will be the video I reply with for all the uh Cloudflare people that just say did you use cloudflare? Which tries me nuts. It's an over simplification of a very complicated problem You see how far we respond back on twitter and they'll say now we have web service Yeah And make this whole video wrong because they'll they could do the same thing But all these complicated lines this shows the complexity of it. That's why you can't just say use cloudflare There's a little bit more to it than that and that's what we wanted to share with you. Um, that's why I have Ray on here But me and Ray are probably going to do some future videos together as well, but I'll leave all the links to oit voype and uh me eating hasas array and suffering a little and Poor simon I have gone back and and had a few more pieces with my family Obviously, there was you know, it was at a family event and there was good drinking involved to to build up the courage to have that again We're not at tom level where you can just I don't know how you pound a down man. It is impressive So On this little memory card is the one chip challenge. I have to edit that still we did it here at the office I don't recommend it We we ended up employees like we had to clear the calendar. It was bad. It was rough for us I was watching a tiktok where they were talking about how to make your own chip Seasoning and stuff like that. So when are we going to see like tom's like fiery chip sauce or like, you know what I mean? Like when are we going to see your kettle chips? Like I can only imagine how bad that would be at some point we may we may go there, but Nonetheless, all right links down below to some of the things I referenced like of course everything that rated and If you want to listen at g rc episode by steve gibson He covers it and um some of the other bleepy computer articles I mentioned I try to make sure we cite all the sources where we have this information from There's always more reading to do I don't want you just to watch this video and say this is it There's always more and it's a rabbit hole and we've read all these articles that get to where we are so we can share the knowledge of you and uh, thank you all for joining us here And uh, thank you ray for coming on and helping us out here and big thanks because my phone's ringing You're welcome and i'm sorry. Uh, no, thanks for having me on tom because they ring a lot when once you fix it Man, we were like oh grab just ding ding I would like to know how was that first person that first phone call you answered Like were they like surprised that the phone was ringing like how excited were they like they won the lottery, right? I answered some of them because they were ringing so much, you know, I was like, I guess I gotta help now Sounds good, man. Well until the next one. Yep. All right. Thanks And thank you for making it to the end of this video if you enjoyed this content Please give it a thumbs up If you like to see more content from this channel hit the subscribe button and the bell icon To hire a sharp project head over to laurance systems.com and click on the highest button right at the top To help this channel out in other ways there's a join button here for youtube and a patreon page where your support is greatly appreciated For deals discounts and offers check out our affiliate links in the descriptions of all of our videos Including a link to our shirt store where we have a wide variety of shirts and new designs come out well randomly So check back frequently And finally our forums forums.laurancesystems.com is where you can have a more in-depth discussion about this video and other tech topics covered on this channel Thank you again, and we look forward to hearing from you in the meantime check out some of our other videos