 with Eric instead. Go ahead. Eric, we've got a few questions already in the queue. I'll start off by asking you one here. Watch your talk. It was great. Thanks for doing the work you did and releasing that tool. It's going to be helpful for the attackers. Myself as a blue teamer, some of the things we do as tricks is to look for, you know, you talked about how to detect it on the wire, essentially network traffic. Have you looked at anything like on an endpoint? Like if there's any artifacts or anything that would indicate that there's a problem, maybe not necessarily that the domain is bad or that it's being fronted, but anything that would show there's a problem. Yeah, I think it's going to depend on the tooling that you use when you actually implement this technique. If you're using a test client or something like the demo C2 that I had, you'll obviously have an EXE executing. If you bake it into something else, maybe you inject. If you're running out of SVC host, you know, and it's supposed to have network connections and you're fronting on something that might look okay, that's probably going to be the best bet for red teamers. But yeah, I think the last slide of my detection piece was, you know, just do the good old-fashioned police work and look at the endpoints and use that EDR. Okay. And you mentioned that a few, a couple of years ago that some of the major players were essentially removed this capability and Azure is only the last standing big cloud provider that does this. Is it possible for Cloudflare to decide the same thing and they say, hey, we're going to block this? Can they do anything about it? Yeah, definitely. They could turn it off tomorrow if they wanted to. I think the way that would work is once they decrypt the server name indication, they could check to see if it was sent to the IP that they have registered for an A record for that domain. And if it doesn't match, then they could drop the connection. But I don't think they're going to do that. Nick Sullivan, their head of research has been aware of this technique ever since Robin would posted it last year. So it's not super new. And I think Cloudflare is kind of a progressive, freedom-loving company. And I think besides, if you not run an HAN or Daily Storm or something crazy, you're probably pretty safe with Cloudflare. So I think it's going to live on. And unless governments really turn the screws, I think this has some shelf life in it. Okay. So anything, like when you said you specifically talked about, you know, how you can detect it on the network, what would that look like? Like, so if I'm using this technique, you showed how it's happening. Again, my team is, or my perspective is the blue side. How would I see this? You know, tell me, you know, give me some hints, how would I detect this and make sure that it's not happening on my network? Is there any quick and dirty tricks? Yeah, I think the best way is going to look for those packets that have both the encrypted server name and the regular SNI. If you're making a TLS connection, you use both of those. That's not normal. It's not expected. And it's probably not anything good. So I would drop any packet that's trying to do that and then investigate that endpoint for sure. If they're only using an encrypted server name indication, then it's a choice to either block all of it or allow all of it. So that's where it gets tough. And there's no easy button in any product right now to enable you to do that. And you mentioned custom rules. You could create your own custom rules, obviously. Yeah, I think in Snored and SecureCata, you're going to have to get into the content parser and really pick up those, the TLS extensions themselves. I'm not a blue teamer. I haven't dug into that. And it's super in depth, but I suspect we'll see some rules pretty soon for this kind of stuff. Maybe tomorrow. Is there any other technologies besides WebSockets and HTTP that could be used to perform the same type of attack? Yeah, definitely. So this works right now at the TLS layer. So anything that you can wrap in TLS would also work with this technique. So I showed HTTP and I showed WebSockets working, but there's no reason you couldn't wrap arbitrary protocols in that. And that's kind of what Cloak is doing, although it uses WebSockets to do that. One thing you could also do is investigate Qwik, which is a UDP based protocol. Cloudflare supports that. And I have a suspicion that this would also work using Qwik. Okay. Somebody's mentioned from the chat, you know, why if if the ESNI is the problem, the traffic, they're asking why not just block it all or decrypt it all, I guess blocking it all obviously would have some impacts, but speak to both I guess. Yeah, you could you could block it all that is an option. As as adoption grows, I think that's going to be a more difficult option. Right now, you might be able to get away with it since it's, you know, only a smaller percentage of the traffic TLS 1.3 itself is, you know, 25 to 50%, I would guess, depending on the network. With ESNI, it's still a draft. So it's probably much lower. It's in Firefox and things like that. But you could probably get away with blocking all ESNI traffic right now and your users wouldn't yell at you too much. As far as decrypting goes, I don't think that's really an option. You're going to have to get in between the DNS and then put your own keys in there. Maybe a vendor will come out with a solution to do that. But right now, it's going to be messy for sure. Yeah, probably a function of how big of a server you have to run how much volume of data that's coming through. And so yeah, you mentioned there was a little comment in there, a little in parentheses, the AI and ML reference. Do you care to speak to that? Is there any sort of statistical analysis machine learning that could be used to protect the system? I think there's definitely space in there for solutions that do their machine learning and their anomaly detection, especially. Like I said, if there's a computer that maybe an accounting or something that's all of a sudden, every 30 minutes, it reaches out to doesn't really matter what domain it is. But all of a sudden, there's a new pattern of behavior from that machine that should be suspect. And if you're if you have a good solution, maybe you get an alert on that. Okay. All right, have you had heard any response from you know, any sounds like you've been talking to some most of these place people, businesses, and providers have they mentioned that there's anything coming up or other than, you know, just expect expectation that once they see this on the on the YouTube and Twitch, that it'll spark them to create some new rules. Yeah, I haven't I haven't really communicated with the firewall vendors much. But I suspect that as adoption grows and as this becomes more of a problem, probably once the first major ransomware uses something similar to this, people will start taking notice. But yeah, nothing nothing right now, I don't have any insight on what's coming down the pipe, but I look out for it for sure. Okay. So let's say I was especially interested in your your problem you were seeing with Paul Alto and the mismatch on one one support for one three, did that get resolved after you recorded this or Yeah, so I don't think that's it's not really an issue. It's just the default decryption profile that they have in place. If you just click on like, do HTTPS decryption for me, it just doesn't include 1.3 by default. But it's if you're setting that up, you're probably going to want to fine tune that. So all you do is you check the extra box and then it works. But it was a little bit surprised when I tried to just do the easy button solution and turns out that it just let everything through it did log an error for every connection. So your logs would fill up pretty quickly. And you should notice something, but a little strange that it's not included in the default profile. So what if I'm a, I'm a pen tester, I get paid for success. And I have my own little tool suite that I'm using, how would I utilize this technique? Like how is there? What did you have to go through to tailor your tool to work with? You know, Yeah, I think there are two options right now for what you can do. One is you can use the kind of cloak proxy shadow socks method that I demonstrated with cobalt strike. So if your tool is proxy aware, you just bundle it with those two, it's going to be a little bigger and make a little more noise. But you can use this technique today with that. If you want to integrate it, if you're using go for your language that you write your tool set in, then you can their instructions on the GitHub on how to integrate it. It's not too bad. Just a couple changes. If it's another language, you're going to have to dig into the TLS library itself. There are two requirements. One, you've got to support the ESNI or the draft ESNI extension. And then two, you've got to kind of adjust that to do things that maybe it's not designed to do by spec, like maintain the ESNI or maintain the regular SNI, if you use an ESNI, if you so want to do that kind of decoy SNI stuff. So digging into the actual TLS library of your language of choice might be required if you want to integrate it smoothly into your tool. Okay. And asking for the red teamers on the call, because I'm sure they want to know if they're trying to be secretive and not use, you know, even though it's you're hiding in the in the noise by using Cloudflare. What if I wanted to find something a little more rare or make it so that it's less likely to get detected since, you know, most most of the early rules are going to be based on essentially the examples that you had. How do I check to see if there's what other tool or what other domain fronting options there are? Yeah, I didn't actually show us in the talk, but there is a tool inside the Git repo that I call find fronts. And you can feed it a CSV of domain names and it will spit out which ones are available to use with this technique. But if you want to do it yourself, it's super easy. All it's doing basically is curling the domain and then looking at the response headers and anything that comes back with a server name of Cloudflare has a set cookie with a, you know, underscore underscore CF, which is a Cloudflare value or has an expect CT header that includes the Cloudflare domain pretty much indicates that that that that site is behind a Cloudflare worker or some other Cloudflare service and it's available to be used with this technique. So everything that's returned that so far has been has been great for this technique. So I'd do that. Okay. And you used Cloudflare as the example. Had you tried it with other other services like Azure? I haven't worked on it with other services mostly because most of them don't support the draft ESNI standard. So you need a CDN that supports the standard and then kind of also allows arbitrary IPs to be used for arbitrary domains that are hosted on it. Another thing is that the bigger the better, right? So you want it to be painful to block and Cloudflare being the biggest CDN. They supported all the things and they were the biggest. So kind of a one-stop shop with that one. Yeah. And does your tool then, would that you mentioned there's a couple of caveats there, would your tool then work through that, you know, knowing that all those other conditions exist and tell you this is legitimate website you could use? No, the fine fronts isn't going to isn't going to do that for you. The fine fronts is just looking for specifically Cloudflare protected domains. So if you if you find another CDN that works with this technique, you're going to have to come up with your own method to detect which sites are available to front with those. Okay. Thanks for clarifying. Let's see. I got another request or question here. Let's see. The Tor project uses domain fronting. Are you aware of this being of your tech? I think it's your technique being adopted or are they looking into it? Yeah, I don't I don't know that the Tor project is aware that this kind of branch of domain fronting is available yet. Hopefully this this will spark their interest. They use the MEC project to do domain fronting and they're kind of reliant on Azure right now since it's the only major CDN that still allows it. But I would love to see them roll out with this technology and same with signal. I think they do some domain fronting in restricted countries as well. So besides being a red team tool, I'd also love to see it be used as a censorship bypass to give people free and open access to the Internet. Yeah, can you speak a little bit more about that? We have another other request that's you know, you walked right into it. That's exactly what they want to know. How how would this be leveraged to essentially become a communication, you know, a tool for communication, I say secretly, but you know, to bypass censorship. Yeah, I think it's probably set up a system similar to one of the demos that I had where you have Cloak or the Cloak fork in the Noctilucent project, a shadow sock server, you put that out on a VPS or a company sponsors a large instance. And then you run a Cloak client locally with a shadow socks client locally. And then you have a socks proxy running on your host that tunnels arbitrary TCP or UDP traffic out to the open Internet and you can bypass any great wall or other restrictive government that's trying to block specific sites or really anything, all your traffic is going to appear to come from that VPS. So it's like you're browsing from that VPS. Okay. All right, it looks like I don't see any new. There was one other question that said, apart from the fact that using domain funding must also host the site on the same domain. What are the other downsides to this? Are there any solutions for detecting and blocking it? Yeah, so original domain fronting that was that was popular back in like 2018, it required you to not only have your domain on the same service, but you'd have to run a service or a VPS on that provider. So if you were using Azure, you had to have an Azure VPS running on that same thing. If you were using Google Cloud, you'd have to have a Google app worker or something like that running on the same service. And all those required, you know, full name to sign up credit cards, phone numbers. It was it was a little burdensome plus expensive if you had a lot of traffic. I think the the tour project has some stats on how much money they were putting into domain fronting, but it wasn't trivial. It was it was an amount with Cloudflare. All you need to do is you sign up for a free account, which just requires a name and an email, and they will take disposable emails. And then you point your domain, register our name servers to Cloudflare, and then you define your IPs in the in the Cloudflare web dashboard. And that's it. Those domain that those IPs can point to anywhere you want. It can be Digital Ocean, Amazon, Google Cloud. It doesn't matter where they're hosted. As long as your DNS is run from Cloudflare. I thought your example of the mail forwarding with the postcard in the envelope is a good good analogy to help help with the C-suite understand what the threat is and how it how it works, essentially, you're a you accept the letter and blindly open it and hand it off to whoever it needs to go to without inspecting it. Exactly. Yep. No, so you did watch the show. I did. I was just joking that I didn't watch it. Let's see. I had another question. I cannot for the life. Oh, the speed. I found that very interesting that the speed was so high. I was not expecting that. Is that do you think that is a function of Cloudflare's capabilities or is it just a function of the protocol that, you know, because it's used on the Internet, that it's there's a focus on making sure it's the throughput's high because you wouldn't want to slow people down to get their right content. And most of the time it's being used for video or those types of things anyway. Yeah, I think it's a function of kind of every piece of the chain. Web sockets are great for bi-directional communication with low overhead and then Cloudflare is just really good at delivering content. I mean, that's their bread and butter. So even when you're pushing 100 megs per second, you know, fronted through them, they handle it no problem. So yeah, everything from the Web sockets to Cloudflare's infrastructure. I wonder what would happen if I, you know, paid for the higher tier AWS. What I could I get even more? I don't know. Maybe that's a future research project is how fast can you do this? But 100 megs a second is pretty solid for everything I have to do it. Yeah, yeah. Yeah, but yeah, I was surprised at that when you showed that. So any other ideas for future research? Are you, you know, is there any other techniques or tricks? I saw that the the enhancements with the Cloak was cool. You know, as a blue teamer, we're always looking for this type of stuff, both, you know, we realize that there's techniques that could be we won't see them. And so we have to combine network analysis with user behavior with you know, anomalies that are detected on the end point, you know, like we're before we were on this the chat, we're talking about, you know, somebody in finance running PowerShell and, you know, maybe the PowerShell is running in the middle of the night, which is unexpected those types of detection capabilities. And, you know, to me, what worries me is as when I was a consultant doing, you know, the pen testing stuff, I felt that a lot of the businesses we were working with the small to medium sized businesses, which are, you know, 90% of the businesses in the country or in the world, actually. There's not a lot of, you know, not a lot of hope for them in detecting this level of stuff, because they just don't have the resources, you know, it's difficult for them to one understand it, you did a great job of, of providing a way to understand generally what is happening. But, you know, if the tools if you go to a vendor and the vendors tools don't can't do it, you know, out of the box or very simply, it's, it's, it's, you know, it's a daunting task for them. Yeah, I think in this case, it might be more of a, you don't have to outrun the bear, you just have to outrun your friends situation where this technique is probably only going to be used by, by the most advanced actors or red team that you hire. So, you know, first step one is detect that the fact that PowerShell is running on the accountant box in the middle of the night and then start worrying about the more advanced stuff. But are you ready for the more advanced stuff? It's, it's there for sure. Somebody's scratching at the door. So yeah, I want to check. Yeah. Let's somebody made a comment about that. Cloudflare has something called Bandware Band With Alliance where you aren't charged for your VPS. I don't know. It's just a comment, I guess. Is there anybody any reason why you chose Cloak over some other pluggable transports like V2 Yeah, there are a bunch of good projects out there. Chisel was another one that I looked at, which is also written in Go. I liked Cloak because it seemed to have the widest variety of potential secondary services that you can kind of glue onto it. So I use Shadow Sox in my example. But the fact that you can tunnel UDP traffic through it, you could probably get WireGuard VPN running, you know, fronted through a Cloudflare, which is just kind of crazy to think about. But yeah, and at decent speeds as well. So I went with Cloak because it seemed to be the the most censorship unfriendly because the finger printing, although I think I kind of broke that when I just used the standard Go TLS library and then the widest variety of things you can use it with. Yeah, I think that's to me that's what the biggest any of these types of talks where you're kind of pushing the envelope and coming up with something new and building, you know, a concept is is the where do we go from here? You know, soon as somebody shows, hey, look what I found over here. And then there's this explosion that happens in the next six to nine months where development kicks off and then somebody will do a speed, you know, a presentation next year based on what you presented and kind of taking it to the next level. Yeah, I hope to see, you know, C sharp TLS libraries that that use this and see libraries and, you know, everybody kind of modifying their low level TLS libraries to enable these options. That'd be really cool. Yeah, do you have any other somebody's asking about research future research, any avenues you think would be fruitful? Yeah, I'd look into quick for sure. Q U I C. It's the UDP based protocol that a lot of providers are even Microsoft is not pushing quick and Cloudflare, of course, supports it. And I would be surprised if you couldn't do the same technique using quick. So you wouldn't you wouldn't be using a TLS and TCP, but you'd be using UDP and using that quicker protocol potentially higher speed and potentially harder to detect if you're if you're allowing UDP out to port 53. Maybe you can, you know, slip some traffic through that way. OK, all right. Well, we're getting close to the end here. You got about seven, eight minutes. You have any other comments or ideas for people to. Yeah, for sure. Anything anything you want to let us know any new tools or updates you want to improve on or no, I would be. Go ahead. Sorry. I'd be lying if I if I didn't have a if I told you I didn't have a DC 29 folder on my computer already with with a notes document in there, thinking about about the next talk and how to take this the next step. But yeah, I think people should just keep up with the latest news either on Twitter or Reddit, NetSec or anything like that. When you see a little nugget, like I saw Robin Wood, it was the one who originally kind of showed that this was even possible. You can kind of run with that and productize it and take it to the next level and then explain how it works and and that's good enough for a talk. So yeah, definitely stay stay with your ear to the ground and keep looking for what's going to be the next thing. And who knows, you know, I was I was at Defcon. I think it was like 23 was my first Defcon and watching speakers talk and I thought that man, that's crazy. These guys are, you know, next level. And then you keep working on it for a couple of years and you find your little niche, you know, you don't have to you don't have to understand everything. Just expand the universe a little bit and here you are. So speaking to that, let's say I'm this is my first Defcon, you know, I for whatever reason, I couldn't afford it or couldn't go and now I find myself quarantined at home and I'm watching you. And I'm and this is something I find interesting. And is that is or I know you posted your contact information on the end of your presentation. Is it okay for people to call you and or maybe I call you but you know, get a hold of you, thank you and say, you know, hey, I got this idea or can you explain something or is there any place they can go to find more basic information, you know, a little more of the one on one? Is there any, you know, of any known or, you know, common sites we can push them to? Yeah, sure. I mean, you're always available to hit me up at bad sector labs is my personal Twitter handle and six gen.io if you need commercial support. But for basic stuff, I would I would look at the kind of the introduction CTF intro level CTFs like over the wire is really great. And those those will kind of walk you through kind of the basics of how do you get into this mindset of, you know, there's something that's supposed to work a certain way. How can I get it to work a different way or how can I do things to it that the author didn't intend? And that kind of mindset is just if you apply that to everything that you see that comes out with cyber security and even just the latest technologies. Yeah, I think you'll go far and honestly reading the RFCs as boring as it is. You know, that's that's where a lot of gold is. So stay up to date on the latest RFCs dig in, look at implementations. Are people doing it correctly? Probably not break their implementations. And there you go. Yeah, I think the RFCs is is a key piece, because it may be not a new feature. It may be an old feature that people forgot. And they nobody's looked, you know, under the covers to know, Hey, look at this feature, I can use this. You also mentioned WireGuard. I think that's another, you know, cool project that's come about in the last few years. Can you speak to that? You're using that in this kind of attack? Yeah, I didn't I didn't test it out with this tool, but I love WireGuard. And I think it's going to be the future of VPN. Super fast, super easy to set up small code based auditable props to the author on that. That's an amazing tool. But yeah, if you can tunnel UDP traffic, then you can probably put WireGuard behind this kind of fronting or hiding. And therefore, the sky's the limit, you know, once once you get out to the open internet, and then you can run your VPN through there. Censorship is kind of they're playing catch up for sure. Yeah, I'm I'm I mean, to me, this is, you know, from a hacker's perspective, I think a good application of this would be tunneling out through a sensor network and getting access to Netflix so I can watch, you know, the latest, the latest, whatever hot lava show or whatever. And that that's one of the last questions here is. So he mentioned if I think we've talked about this a little bit, but they're asking what if government made it demanded that domain fronting illegal? What, what would break? I guess, you know, obviously making it illegal is one thing. And then you have to have the capability of stopping it because you can say it's illegal. And if people continue to do it, you know, but I'll let you speak to it. Yeah, I think I think we saw this happen in 2018 when the Russian government turned the screws on on AWS and Google and they eventually capitulated. So it's a possibility that, you know, enough governments with enough power, turn the screws on cloudflare that they say it's not worth fighting this and we're going to, you know, succumb to the demands and they're not a government. So they can't really fight that fight. But I think that they're going to try their hardest to to maintain the ability for any of their IP range to go to any of their domains. I think it's, it's one of their business advantages, honestly. So turning that off might be detrimental to even the just the basic operating of cloudflare. So while it's possible, I think they're going to fight it as hard as they can. But if the government makes you do it, I mean, we've we've seen it time and time again, eventually, I think these companies will succumb to what the government forces. This isn't like a Bitcoin or a magic blockchain solution to anything where, you know, it's going to survive no matter what happens. But we'll see. Yeah, yeah. And things continue to evolve. So, you know, where they might, this might get fixed by the industry, and then a new technique comes up, you know, so, so we're about out of time. Thank you for speaking here at DEF CON. And it was a great talk. And I appreciate all the questions you had. Yeah, thanks, guys. And thanks to all the goons who made this possible. I think a lot of people tuning in don't realize that it was it was a major effort to get everything coordinated and work. So thanks to you guys. Thank you. All right.