 Hello everyone and welcome to domain fronting is dead long live domain fronting using TLS 1.3 to evade sensors bypass network defenses and blend in with the noise My name is Eric Hanstead, and I'm the CTO and adversary emulation lead at six gen a full spectrum cybersecurity company based in Annapolis, Maryland Today, I'm going to go over what domain fronting is including HTTP and HTTPS basics I'll talk about TLS 1.3 and encrypted server name indication and how this can be used for domain hiding I'll show some demos of this in action from basic bypasses to bypassing an enterprise TLS decrypting firewall and Finally, I'll talk about some ideas for defenders who may want to detect this in their environment Domain fronting 101 if we're going to discuss a new method for domain fronting Let's all get up to speed on what domain fronting is why it works and what happened to it Like many things in security to understand the cool stuff you have to dig into the basics to understand domain fronting We really need to understand HTTP and HTTPS Specifically, we'll be focused on server name indication Which is a special extension added to TLS that allows multiple sites to be hosted on the same server But TLS 1.3 this extension can be encrypted and combined with encrypted secure DNS We can do domain fronting 2.0 or domain hiding Okay, let's start with HTTP. I'll skip over the super low level stuff as it's not relevant here The first external request involved in an HTTP connection is a DNS request This DNS request gets sent out unencrypted via UDP to port 53 The DNS server responds also unencrypted via UDP with an answer that contains the IP address of the domain from the request Now that the user's computer knows where the domain is hosted It can issue an HTTP request to that IP address with the host header indicating the domain the user wishes to browse This request uses TCP, but it's unencrypted The web server responds with content in this case some simple HTML This was fine for the early days of the internet when it was just a few research institutions But as soon as people wanted to conduct business online the obvious issues with completely unencrypted traffic had to be fixed Plus without encryption all your data is free game for any number of actors that are positioned between you and the web server HTTPS was the answer to this problem, and it starts off the same way with the DNS request The DNS request and response are still unencrypted when using HTTPS Unless you use a different system, which we'll talk about later on. Here's where HTTPS is different The connection to the server starts with a TLS connection Specifically a client hello in this packet the server name extension is used to tell the web server Which website the user is requesting this allows the web server to return the proper certificate for that site The server returns the certificate in a server hello packet And then the client and server can agree on which encryption algorithms to use and exchange session keys All traffic after this is encrypted, but both the client hello and server hello are unencrypted Which leads the domain and certificate to anyone between the client and the web server And we know there are plenty of organizations interested in that data With the basics covered, let's talk about domain fronting Domain fronting is a technique to get around network defenses or sensors by connecting to an approved server while hiding the true destination of the HTTPS request one of the biggest restrictions of domain fronting is that the front of domain and the true destination domain Must be hosted by the same service provider Typically, this is a content delivery network. So how does domain fronting work in practice? Well, it starts with the same DNS lookup as before and the TLS handshake is the same as well However, in the handshake the client connects to the front domain not the true destination domain Once the TLS handshake is complete with the front domain the fronting begins The client sends an HTTP request to the front domain wrapped in TLS But with a host header of the true destination domain When the content delivery provider receives this request it inspects the host header Determines that it's for a different domain hosted on the same provider and dutifully forwards the request to the true destination domain You can think of a domain fronting like a postcard inside an envelope on the outside of the envelope the client writes The address of the CDN, but on the inside the true destination domain is on the postcard The network filters or sensors are like mailmen who can see the outside of the envelope and they deliver it since the CDN Is an approved address however when the CDN opens the letter they deliver the postcard to the true destination internally This was working well on all major CDNs until April of 2018 when the Russian government put pressure on cloud providers to stop it Since the popular messaging app telegram was using both Google and AWS for domain fronting Both Google and Amazon stated that the reasons were technical and domain fronting was either Never supported or a violation of terms of service and other providers follow suit to include cloudflare Microsoft's Azure platform still allows domain fronting for now as do a few other smaller CDNs Domain fronting is a great censorship or network bypass technique But it has some major problems for any network censorship bypass to be effective It has to be really painful to block and since the largest provider shut it down It lacks the punch that it once had Also, the true destination domain has to be hosted on the same service as the front domain Which limits the potential front sites and imposes cost on anyone wishing to set up fronting With the background out of the way We can talk about TLS 1.3 encrypted server name indication and what can be called domain hiding First let's talk about TLS 1.3. It's seeing steady growth on the internet today This is data from Kuala's SSL labs, which tests the most popular 150,000 sites as you can see TLS 1.3 is on the rise and is supported by 31.7% of those sites tested last month Cloudflare is seeing 59% of all traffic using TLS 1.3 again showing a steady increase Before we dig into the mechanics of a TLS 1.3 handshake with ESNI We first have to secure the DNS process if a network defender or sensor can see our DNS requests They can easily be blocked Also secure DNS is a cornerstone of ESNI which we'll see in a moment The first solution is to simply wrap a DNS query in a TLS connection To make things even more difficult for network defenders or sensors DNS via HTTPS is a newer standard that is seeing more widespread adoption Once we have an encrypted way to perform DNS queries We can use this to retrieve a public key for a domain Which we can use to encrypt the server name in the client hello packet in this screenshot from Wireshark You can see the encrypted underscore server underscore name TLS extension and the encrypted SNI value Is it's just an encrypted hex-string No longer can defenders use the server name value to block traffic based on a plain text value A TLS 1.3 connection with ESNI relies on having the server's public key in order to encrypt the server name in the client Hello, and the current standard for how to get this key is to use DNS and query the underscore ESNI text record for that domain This request returns a base 64 encoded public key the client can use to derive a session key that encrypts the server name Providers that support this should rotate this key to provide some forward secrecy Cloudflare for example rotates this key every hour It allows a few hours of buffer in case a client has a slightly stale key from a previous request or from caching Armed with the server's ESNI public key The client derives a session key and uses that to encrypt the server name and sends the client hello The server has the corresponding private key and can derive the same session key Which it uses to decrypt the server name and performs the same steps as standard TLS This session key generation ensures that the key is tied to this session that generated it and prevents replay attacks Another difference between TLS 1.3 and previous versions is that the certificate returned by the server is encrypted This is possible because in TLS 1.3 the client hello includes a list of supported ciphers And the client also makes a guess as to which key agreement algorithm the server is going to choose And it preemptively sends a key share for that algorithm So all the plain text portions of HTTPS we saw before have been mitigated However, there's still weak spots where sensors or network defenders could stop the connection Even if the DNS request itself is encrypted inside either TLS or HTTPS The data returned by a resolver may be poisoned Imagine a scenario where a country or enterprise returns their own key for all underscore ESNI requests No matter the domain to enable man in the middle DNS sec solves this problem by signing the record set and allowing the chain of trust to be checked The details of DNS sec are outside the scope of this talk, but I encourage you to check it out But what if encrypted DNS is completely blocked? The solution here is to bootstrap the connection by pre-loading the current ESNI keys by some other means Maybe encrypted DNS is blocked, but what if I have a script that updates a github gist with the current keys every hour? Or you could just bake the keys directly into your binary if you know it's going to be executed in the next hour or two Okay, let's say we get a valid ESNI key But the IP address of the server we want to connect to is blocked The IP may be shared with other sites, but it doesn't have to be Is there a way to route from arbitrary domains and IPs to our true destination domain and IP? The answer is no, but let's see how close we can get So far I haven't introduced anything new But we have covered a lot of ground and identified potential issues and solutions around domain hiding with TLS 1.3 and ESNI Let's tackle that last point and get around IP or domain blocks We'll do this by leveraging the world's largest cdn Cloudflare was founded in 2009 and has had explosive growth It's now the world's largest cdn with the most internet exchange points as well as being an authoritative dns server hosting over 26 million domains Cloudflare is on the forefront of internet technology and supports TLS 1.3, ESNI, web sockets, quick and dns sec This new technique, which I'm calling domain hiding, accomplishes the same goals as domain fronting, but uses different technologies A TLS 1.3 connection with an ESNI of the true destination is made to any cloudflare IP And the underlying htp request also has a host header of the true destination This enables any cloudflare owned IP to act as a front for any domain hosted by cloudflare dns This technique was first shown to be possible by robin wood, so props to him for the discovery Today I'm releasing Noctilucent, a project that enables ESNI and goes crypto slash TLS library As well as exposes two additional configuration options, ESNI server name and Preserve SNI It can be used by any project currently using the standard crypto slash TLS library as it's backwards compatible It comes with a demo client application that allows you to control just about every part of the TLS connection As well as the HTTP get request, so you can play with what's possible using this technique It also supports web sockets and through the magic of go, it compiles cross-platform to a single binary with no dependencies Here's what a standard TLS 1.3 connection looks like with Noctilucent We've specified the TLS host in red, the server name indication in yellow, and the HTTP host header in green The output shows what we expect. There's no ESNI. The SNI is set to cloudflare.com We're connecting to cloudflare.com on port 443 and our get request to cloudflare.com returns the expected 301 response Nothing fancy yet Let's enable ESNI and set the ESNI server name Everything is the same as the normal TLS 1.3 connection we looked at on the last slide Except this time, the ESNI value is set to cloudflare.com and there is no standard SNI Progress, but no fronting or domain hiding quite yet Now I'll change the ESNI server name and HTTP host header to a domain we control In this case, defcon28.hackthis.com, believe the TLS host as cloudflare.com Notice that this connection is still going to cloudflare.com, but the HTTP request successfully goes to defcon28.hackthis.com, which returns hello defcon To anyone between this host and cloudflare, it appears to be a TLS connection to cloudflare.com. We've successfully achieved domain hiding Now the fun begins. Here we send an ESNI and standard SNI, where the standard SNI is cloudflare.com Because cloudflare ignores the standard SNI when an encrypted SNI is used, we can set it to anything The HTTP connection works as before and this time anyone between this host and cloudflare, looking at SNI data, thinks this connection is going to cloudflare.com In addition to the underlying TLS connection actually connecting to a cloudflare.com IP What I've demonstrated is the ability to arbitrarily front or hide any IP behind a domain that is using DNS provided by cloudflare The true destination IP doesn't have to be a cloudflare worker or other cloudflare service In fact, all the examples today are IPs hosted by DigitalOcean Best of all, the requirements to sign up for cloudflare DNS are minimal and it's free What domains can we use for this hiding? It turns out a lot. Over 21% of the top 100,000 sites are behind cloudflare and allow this to work Some notable examples are shown below, including three of 11 domains which are both on the Palo Alto decryption whitelist and behind cloudflare I'll demonstrate why this is important in a minute There's a little bit of everything on this list from security related sites, banks, sports, higher education, streaming services and even government sites And porn. So much porn Here's a good example of how the variety of available domains to hide behind can be useful This is a request to www.octa.com, a single sign-on and identity provider used by many enterprises But the actual request is going to a server eye control To a sensor or network defender, this looks like standard octa traffic, with the exception of the ESNI extension being used The logical next step is to try out Noctilucent with some web filtering and see how it fares It turns out most web filtering is done by inspecting the SNI and making a filtering decision based on that Here I set up the untangled firewall with federal government settings and explicitly blocked defcon28.hackthis.com On a host behind the untangled firewall, I made this request, which succeeded I set the unencrypted SNI to bypass-untangle.com to see if it appears in the logs anywhere After repeating the request 30 times or so, back on the firewall, we can see that the top domains include the decoy domain that we sent with the request Simple SNI filters were easy to defeat. What about HTTPS decrypting firewalls? These are often seen in enterprise environments and work by breaking and then re-encrypting HTTPS using a root certificate placed on endpoints behind the firewall You can think of it as a corporate man in the middle that allows the network defenders to inspect full encrypted packet data Kazakhstan actually attempted to implement this nationwide for a few weeks back in July of 2019 before major international backlash forced them to abandon what they called a, quote, test I headed to AWS and spun up the latest and greatest in enterprise firewalls, Apollo Alto PAVM version 10.0.0 Released in June, this version touts the ability to decrypt TLS 1.3 with a two-time speed boost Imagine my surprise when it was all set up and the default decryption profile didn't include TLS 1.3, instead it let it pass through while logging an error about a version mismatch Once set up with the decryption profile that enabled decryption of TLS 1.3, I pulled the list of default domains that are allowed to bypass decryption for a variety of reasons, such as having a PIN certificate Here, the firewall interface is on the left and a Windows VM behind the firewall is on the right We can see that decryption is enabled and we have no other decryption rules that would allow traffic to bypass decryption Normally, in an enterprise environment, categories like banking or healthcare bypass decryption due to privacy laws, but this setup is as strict as possible, no explicit exemptions On the Windows VM, I'll run a PowerShell script that checks the certificate hashes of major sites against hard-coded, known fingerprints We see that it fails, indicating that all tested sites are being manned in the middle Additionally, opening a browser and going to Google.com shows a lock icon, but inspecting the certificate shows that it's the firewall certificate Switching to the command line and using the Noctilucent test client, I'll set the TLS host to Mozilla.org, the host header to defcon28.hackthist.com, and the unencrypted SNI to Mozilla.org, which is on the decryption exemption list The SNI is preserved and set to Mozilla.org, a connection is made to Mozilla.org on port 443, and we see the hello defcon28 response we expect from the test server We'll run an NS lookup of Mozilla.org so we can look for the connection in the logs Looking at the firewall traffic logs shows a connection to one of the IP addresses for Mozilla.org Moving to the decryption logs, we see DNS over HDDPS, which is the SNI key lookup that is performed before each TLS connection But no traffic to Mozilla.org's IP To make sure we didn't miss it, I'll add a filter for the two possible IPs No results, because Mozilla.org is exempted from decryption by default We have successfully defeated an HDDPS decrypting firewall by using built-in default exemption and domain hiding This bypass is great, but every time we want to send data, we'll add an entry to the traffic log, which could stand out What if there was a better way to hide? This technique works at the TLS level, so any protocol that Cloudflare supports underneath would also work We saw that regular HTTP request work, but create a new connection for each request Web sockets are a technology used by many streaming services that enable full-duplex communication channels over a single TCP connection And we can take this a step further by using one or more of these web socket connections to tunnel arbitrary TCP or UDP packets to a proxy beyond the firewall Here's the Noctilucent test client sending and receiving via web sockets to our test server, while hiding behind www.octa.com Web sockets are the perfect candidate to use as a wrapper for a proxy protocol that allows us to send arbitrary TCP or UDP traffic using a helper program while only making a single TCP connection The Noctilucent project includes a fork of the Amazing Cloak project by Andy Wang that enables two new options in the client configuration ESNI server name and preserve SNI, which operate just like they did on the test client Cloak is a universal, pluggable transport that cryptographically obfuscates proxy traffic as legitimate HTTPS traffic disguises the proxy server as a normal web server and multiplexes traffic through a fixed amount of TCP connection while providing multi-user usage control With the additions from Noctilucent, it can now do domain hiding as well Here's the same Palo Alto and Windows VMs from before, with a public IP of 3.14.44.10 This time, the Windows VM is running a Noctilucent Cloak client and a local Shadow Sox client The Cloak client is connecting to a VPS running the standard Cloak server and a Shadow Sox server Firefox is configured to use the local Sox proxy provided by the Shadow Sox client running locally, and I can browse the internet as I would normally The certificates of websites in this proxy browser are legitimate as they're not being intercepted by the Palo Alto firewall and all traffic is appearing to come from the DigitalOcean VPS The speed is surprisingly good During testing, I was getting speeds up to 100 megabits per second using 4 TCP connections with Cloak If we look at the firewall logs, we only see the 4 connections to Mozilla.org, which are non-tycripted while the client has been browsing freely, with all traffic remaining encrypted and uncensored Switching to the decryption logs, we see the DNS over HDPS requests, but no request to Mozilla.org I'll filter for the Mozilla.org IPs, but we'll get no results What we see here is arbitrary traffic at high speeds using only 4 TCP connections to what Palo Alto thinks is Mozilla.org Can we use this technique for our pre-existing Red Team tools? If they support proxying, you bet Here's a Cobalt Strike listener with a Sox proxy setting for localhost port 9999 For fun, we'll set Cloak to use bitdefender.com as the front domain, which has the IP address shown here After starting the Noctilucing Cloak client and local Shadow Sox server in Sox4 mode, we'll double-click on an innocent-looking EXE We get the callback immediately, and it works as expected And of course, the first thing you do when you get access is pop a calc Switching to the firewall and looking at the decryption logs, we only see entries for cloudflaredns.com I'll paste in a filter for the 2 possible IPs for bitdefender.com, and we get no results Looking at the traffic logs, we see 4 connections to the IP of bitdefender.com, as well as lots of DNS over HDPS Our Cobalt Strike beacon is still functioning as normal, and any network defender is none the wiser Packaging up Cobalt Strike, Cloak, and Shadow Sox into a clean package, possibly communicating over namepipes instead of sockets, is an exercise left up to the viewer Red Teamers are no doubt wondering how much work it would be to integrate this with their existing tooling And I can say if you're using Go, it's almost no work at all Here's a new C2 framework called Demo C2. It was released just 5 days ago and is built using Go and ViewJS I've set up an instance here with an HTTPS listener with a public IP on port 443 Going into the listener settings, we can view the source code for an HTTPS agent It has the C2 host imported in the code, and the agent will connect directly to this host Switching to an editor, I've made a few modifications to get this to work with Noctilucent First, I set the host to bitdefender.com, and I've added two new variables, FrontDomain and ActualDomain Which are set to bitdefender.com, and DayMos.hackthis.com The actual code changes needs to get everything working in just 30 lines First, we have to query for the ESNI keys and parse them If we wanted to, we could bake in the current keys here to avoid the network traffic related to querying for them Then I define a TOS config and set the minimum value to TOS 1.3 I'll define the two new options from Noctilucent, ESNI server name is set to the actual domain, and PreserveSNI is set to true I define an HTTP client by hand so that I can tell it to use this TOS config when dialing the host Besides that, the only change to the agent is having the HTTP client use the actual host when it makes its post requests On our Windows VM, I'll set a filter and wireshark for the IP of bitdefender so we can inspect the client hello I'll double click this new innocent EXE and switch back to wireshark In the client hello, we see the unencrypted server name is bitdefender.com And the encrypted server name is just a bunch of hacks Back on the C2 server, we have an agent calling back Notice the external IP When using this technique, the reported external IP will be from one of Cloudflare's ranges Just like with Cobaltstrike, everything works as normal You can run who am I, pop a calc, and check the public IP Besides the external IP appearing to be from a Cloudflare range, the agent functions as expected Back on the Windows VM, I'll search for traffic to the C2's server IP No packets. There's no trickery here, but just a small modification, this go-based agent is using domain hiding So what is the blue team to do, given that we've just seemingly bypassed the most sophisticated network defenses? The easiest thing to do would be to block TLS 1.3 But is that likely anywhere from 25 to 50% of the TLS traffic leaving your network? It might make a few users unhappy You could block Cloudflare, but dropping 21% of the most popular 100,000 domains probably is not a good idea either What about just targeting client halos with ESNI's? It's possible, and one vendor has had the ability to do that for 6 months now However, since you can't determine the destination site, it's all and unblocking There's no way to only block selective domains if you block TLS connections that have an ESNI I was surprised by the lack of support for this across major vendors, but perhaps it's just not an issue yet The most sensible way to detect this specific technique is to alert on packets that contain both a server name and an encrypted server name TLS extension There are no default helpers for TLS in Snort besides SSL underscore state, which can help you get the client hello packet and TLS.version, which can get you TLS 1.3 So you'll have to get creative with the content parser and look for the extension types SecureCata has a bit more horsepower than Snort, but only has a selector for TLS.SNI and not TLS.ESNI So custom rules will be required here as well It's going to come down to tried and true techniques that work regardless of traffic content Things like beaconing detection and anomaly detection can help show clients that all of a sudden start sending regular traffic to destinations that perhaps don't make sense There are even some open source tools that do just this A network defender should be asking if a traffic pattern to a given endpoint makes sense in context Perhaps that Windows machine and accounting shouldn't be sending gigs of data to your single sign-on provider TLS fingerprinting with JAW3 and JAW3S is another technique that could be effective JAW3 looks at the specific options used in a client hello that can sometimes uniquely identify a client and JAW3S does the same for servers This will work until attackers tune their tools to be identical to browsers and common web servers and work is already underway to do this in projects like Cloak and UTLS As always, a layered approach that uses strong instrumented and managed endpoint detection and response and other techniques is the best way to prevent compromise or as Microsoft so helpfully says just don't let untrusted code ever run on your server To wrap up today, I've discussed the basics of HTTP, HTTPS, domain fronting and a new form of censorship resistant communication domain hiding This technique is usable today with Noctilucent either as a drop in replacement for the Go TLS library or as a proxy The ESNI standard is still very much a draft and this technique will continue to evolve as it changes in adoption growth In fact, it's not even called ESNI anymore Just as with standard domain fronting, this relies on the CDN permitting it However, in this case, besides preserving the standard SNI header, we aren't really doing anything terribly out of spec I look forward to seeing this Nort and Secure Coder rules to detect and block this technique And like every good red teamer, I'll push my customers with the latest techniques hopefully before the real adversary can I want to give special thanks to Robin Wood, who first showed this technique was possible with a rough POC using a fork of open SSL Andy Wang, the creator of Cloak and Nick Sullivan from Cloudflare who helped get me data on TLS 1.3 adoption Noctilucent is available now on GitHub If you have any questions or comments, feel free to hit me up on Twitter where my personal handle is at Bad Sector Labs Thanks for watching