 Good day, afternoon, morning, morning, whatever, let's introduce ourselves first. My name is Carl Loving, I'm the technical leader of the Dutch Tax and Customs Administration Security Operations Center for, I think about 15 years. And this talk I'm going to do together with my colleague Arnold Hulsel, he's one of our security analysts in the ERT business for 15 years, and it's for him, it's more about out of control hobby, service data he loves, blogging, no, no, reading RCS, and finding the hidden treasure. Our talk is divided in three sections. First, we present a real-life fishing example. Unfortunately, it is in Dutch, but I will go through that. And the number of email standards. And finally, we present the implementation based on those technologies we have created in Spark. And finally, we also want your help. The implementation we've done, you can do that also in the Elk Sack, and add other tooling you have, which can process your logs. So don't worry, you don't have to spend millions of dollars to implement the techniques we will produce during this talk. And at the end, some future plans we develop and we'll let you know and ask you input too, please. So why do we think fighting and combating fishing is so important? First of all, the damage caused by the tax pays in our businesses. Both of them affected in their pockets and emotional well-being by confronting fishing. I assume everybody has received some fishing and the fishing in the past was fully translated in Dutch, and now Google Translate does a great thing. And they have also translation bureaus to translate the English into Dutch. And we try to build a strong relationship between the tax pays based on trust. So we think the good relationship will help us to collect taxes in an improved way. It's very crucial to detect fishing campaigns as soon as possible because the perpetrator is only interested in your and my credentials and eventually your and my money. And what we see is that in the first hour, most people are clicking on links and then it will fade out. And the main objective of our research was trying to find a way to detect those fishing campaigns as soon as possible. So and then I have the ability to break the money circle also quickly as possible. Now, this is a real life fishing example. I told you it was in touch, but in this fishing example, the recipient receives the mail stating that he has to pay a tax installment of 83 euros. But look at the domain, which is used in this email in the email address, which is belastingdines.nl, which is our domain. So they're using our domain to send fishing emails. If we had implemented these standards we produced we presenting in the stock at a specific time we had detected the fishing campaign instantly. And if both parties has implemented these standards, the email would have not be sent to recipient. And we will show you also in technique you get some insights who is sending those emails on behalf of you. But before we have to continue, we have three different starting points to consider. First of all, there should be no impact on the business. It's not allowed to lose legitimate emails. Secondly, only standard may be used. So don't invent your own standards, but use the standard description request for comments or RFCs. And finally, we will, in representation of only what prevent the fishing and the implementation is done at the sender and recipient side. The standards we're discussing today are shown on the slides. And we go to review these existing security mechanisms so we're on the same page to understand what we did. And let's start first with Start TLS. Start TLS is a method to upgrading a plain text communication channel up to a secure communication channel based on encryption. And this can be used to secure an encrypted S&P connection to a secure encrypted S&P connection. But there is a real serious issue with Start TLS because it's vulnerable to men in the middle of text. And you can mitigate that by using Dane for S&P or MTA, STS. Both standards we will present during this talk. Let's turn our attention to, and you can find the RFC, by the way, in this initiative, if you ask, are there like reading RFCs? Yep. We also have a nice overview of everything discussed at all the RFCs in one of the last slides. Yeah, so then of course the slides will be made available to you. And so you can click on, click on links. So now we know. Type it over. OK. So, Dane, you can see that as an alternative to PKI, PKI certificate authorities, it's built on Dane as Sec. So no D in the Sec means no Dane. And which can you can then verify whether a TLS collection is secure. So it's made the TLS collection less vulnerable for men in the middle of text. In the case of S&P, the alternative mail server for a domain started MX records, as you probably know, the X5 or 9 certificate of the mail server is also stored in the DNA zone file and as TLS-A records by comparing the certificate of during the start TLS connection and the one stored in the DNA zone file you can verify when you're talking to the real mail server and not some man in the middle guy who's created their own mail server. And Dane for S&P, which is an extension of Dane, can solve the problem for S&P. Because of Dane Sec, you can create your own X5 or 9 certificates, so you don't have to buy an expensive certificate or use a matching grip for that. This is an example of S&P MTA SDS, which is the mail transfer strict transport security, which is equivalent of Dane for securing your start TLS connection. This is built on a policy which you have to create on your web server. It's a fairly new standard. It's published in September 2018. The difference between Dane and MTA SDS is that you don't need a full DNA sec application. Together with that comes S&P TLS reporting, which will give you a reporting option of your TLS connections if they're trusted, if they're found a service certificate, which is rogue. And the difference between MTA SDS and Dane is that you need a valid X5 or 9 certificate, so a certificate from some other CA. In this example, you can see this is our MTA SDS policy file, which can be downloaded from this site. It has to be in a well-known directory, and the file name is either MTA SDS.txt or MTA SDS.json. And this file, we have a .txt file. We have a demo testing, and you can put your Amix records in there, or additional Amix records, and you get a Max8 file, which is the TTL of your policy file. So if you receive then a MTS report, it looks like this. This one comes from Google, and it gets the policy back, and a total successful count. If, in this case, everything is OK. But the bad, for instance, if you have a man in the middle guy, then you will see the sending and the receiving MTAs, which are not yours, and the failed count is the one, because Google failed to set up a TLS S&TP connection with your mail server, because the MTS and SDS policy is not correct. Another standard we implemented is SPS and a policy framework, probably a known all by you. I put a figure in it how it works. Somebody sends a, tries to send an email to a recipient to post email and send it to a mail server. The mail server sends it to mail server B. What it does, it checks the SPF record in the BNS server, whether that's correct. If correct, then you can add also decim signatures, decim check, and finally a DMARC, which will come later, and then if everything is OK, then the mail will be delivered to the recipient. If not, it will be handed up in your spam box. But what we did also is we added some, we looked in Arnold did, Arnold looked in the appendix of the SPF and found an option to add macros with your SPF record. And then it makes it a very interesting protocol because then you can see who is sending email on behalf of you. So you can find the macro definitions in paragraph 7.2 of the SPF RFC. And what you can see is also we, Greg from spf.all.com did some research. They found out that from the 140 million domains he has visited, only 2.6 million has implemented SPF. So a lot of work to do. And from those 2.6, of only 13,000 domains, implemented macros. And macros are very, very powerful, which we can show you later. So a lot of work to do. So what are these macros look like? If you put this in your, those macros used in your DNS file, or DNS zone file, you can get the complete sender email address, the helostring of the email server, so the identification of the email server, the local part, domain part, and also the IP address of the sending mail server. So you get insights who is sending email on behalf of you. And that will end up in your DNS log files. So DNS logs are key, are very important, don't outsource them because they are very, very valuable. And the RFC is quite old. It's from March 2016 for RFCs. But the implementation rate is very low. And the one thing is a problem. That if you use the pros and L, then it can contain valid email addresses. And those valid email addresses will end up also in log files. There is a RFC, which is 7816, which is about DNS query domain, named minimization of reasonable logs. That will avoid that, but also the implementation rate is quite low. So that's a big. So how can we detect those fishes when they're not in our infrastructure? And everything happens on the internet, but not on our servers. That's the SPF and macros come in. And here you can see the implementation we have done, which is a redirect record to underscore SPF.palasimdines.nl, the tag is SPF1. And in the resource records, you can see we're requesting the IP address, the helo string. I've already done the main part of the sending email. And you can check that if the other two lines are our mail service, which can be checked then. How it works, I'll show you later. So in this picture, you can see how the DNS queries will work. Somebody sent an email at socantpalasimdines.nl from mail server A to mail server B. There will be done a SPF lookup for balasimdines.nl. What the domain server will send back is the text record, resource record with SPF1 with a redirect to the actual SPF record. And again, there will be done a lookup for underscore SPF.palasimdines.nl. What will come back is the macro file. What will happen is the email server will put the IP address, his helo string, and the sender part of the domain into the DNS request. The receiving DNS server, which is the alternative name as a DNS server will answer that. And if you can go back in the other slide, you can see a matching resource record for this query. So what this will send back is an IP address that means SPF pass. So this is an email server which is authorized to send email on behalf of our domain. Just a minor note here, we have put in here 127001. We found out not that long ago that that not always works. So if you put in there any other IP address other than localhost or an RSG1918 address, it will work. Just put in your IP address of your mail server. And also if you have an IPv6 resource record, put an IPv4 IP address in that. In your macro response, yeah. It always needs to be an IPv4. Yes. So this is the bad situation. Somebody a perpetrator sends email on behalf of you. So the whole spin goes on, SPF lookup, redirect, et cetera, et cetera. And then what will happen is that the recipient sends back the IP address of the perpetrator. Meaning that's not an IP address which is in one of the IP addresses which is in the resource records. So what the DNS server of our DNS server will do, the author of the DNS server will send back an annex domain because that specific resource doesn't exist. Meaning now you have in your log files the IP address of the perpetrator and the helo string of the perpetrator and the sender part of the, which is actually then blushingdins.nl. So you can get more insights of who is sending email on behalf of you. Those DNS queries and responses are actually stored in your DNS logs which you can ingest in your log management or SIEM tooling like we did. So you get more insight who is sending email on behalf of you. The next time that we will present this, the main key identify mail decim. Decim is a method of detecting forged sender email addresses in email. Actually it signs the body and selected part of the S&P and it's depending on what you want to be signed. That's up to you. The digital signature is transmitted in the header of the email. And you can use namespaces which is actually handy if you outsource parts of your email environment for instance marketing email or whatever. And the sender must use the secret key of the decim pair and the public key will be in your DNS as a text resource records, which can then be used by the receiving mail server to check whether the signature is correct of the mail sent by the real mail server. DMARC is actually kind of the clue between SPF and decim. And so in your DMARC record you can put the actually verdict if both are failing what you should do. Should you put the email into the spam folder or should you quarantine it or reject it or pass it? That's up to you. That's also depending on the policy you have configured. To summarize, yeah, implemented this standard gives you more detection and prevention capabilities. We have done it. We have about 550 domains and it took about a month to implement. You have to do it in production because... Well, that's not production kick for sure. Okay, so, and if you want... And also the both sender and recipient should be implemented and that's why we're giving these talks because we think it's important that everybody implement this because then the number of phishing and mails will reduce significantly. Also in the Netherlands, the Dutch governmental organization must comply to the Dutch standardization platform to comply with explain list and there we can start the LS, SPF, et cetera, also in there. SGDN, which is the Dutch domain register give a financial incentive of 3.3 Euro 40 per year per domain if you implement it. I'm sorry, less, it's total, it's 3.40 Euro per year for a domain and you get four cents discount but if you start the LS and demark if you're 16 cents discount and if you have a lot of domains then it could be end up quite a few beers in a bar in Las Vegas where we're not allowed to go then. Yeah, okay. This concludes my part of the talk and thank you for your attention so far and Arnold, the floor is yours. Yes, let me get a sorry phone. Okay, how to implement this? Because those are all nice standards but yeah, you have to do something with it and then do something with all the information. Like Carl said, we have implemented just about everything and well, we're doing it for the greater good of things so we want to share it for free and yeah, hope that everybody implements it. So like Carl said, all these different standards will give us a lot of information. You have demark which gives you more insight in what's happening on the receiving side with your email so are those emails just processed or are they rejected or are they put in quarantine? And with the advanced SPF micro, we have more insight into where is the email coming from? Is it actually coming from our infrastructure or is it coming from somewhere else? But that's just data until you do something with it then create information out of it. So what I've done like Carl said, I created a Splunk app because well, yeah, we have Splunk and we don't want to use another tool but if you have anything else that can process log files just use that if you want to do it in Excel. I created the Splunk app and it's one of the things that it does is it processes the demark reports. Those reports are sent to an email address either as an XML or an XMLGZ file in theory. So I created a couple of Python scripts that can access the mailbox and pass out the download all the reports, process the XML files and do some error handling with them and then put it in a log file either as a key is value pair or a JSON file so that you can, well, then after that process it with whatever tool you like to ingest it in your log management environment or your scene. What you need to have is obviously a demark record in your DNS zone file. You need to have network access to the mailbox either via pop tree or IMAP and of course the user, the imposter for the mailbox. You put those in and well, it will create the JSON files and then you can ingest it. Created a couple of dashboards on that. This dashboard will in general show you how things are going. So especially in the beginning when you're implementing it, this is a dashboard that can give you a lot of insight into how is my email handled by the receiving parties. Are there any errors? Do I need to change some things? On the first row, you will see the policies that you published. So our demark record is reject now. Please start with a non-policy and non-policy means just process the email, don't reject it, don't put it in quarantine, just do whatever you think that is necessary. If you do that in the beginning, you can get already information about what's happening and then you can adjust either your mail environment or your policy if everything goes well or not. You can select what you want to group by. In this case, it is grouped by what was done with the emails, whether delivered or whether rejected. You see some counts and the adoption rate of your alignment, how are your emails aligned. In the bottom, you see more details. So where were the emails coming from? What was the IP address that the mail was coming from? How many messages were there? The first screen column is the de-kim alignment, sorry. Alignment relates to is the email coming from a subdomain or from a top level domain or from your domain, not from a top level domain, but from your domain or from a subdomain. And was it also sent by that domain? You have two policies that you can choose. You can either choose relax, which means that email may come from a subdomain or you can use strict, which means that the part after the app needs to be the same as the domain that is coming, that is sending the email. Also your SPF alignment goes for the same and the last two green columns are the action that was taken on the email and whether or not that was correct, confirmed policy that you stated in your record. Here you see that everything is green, everything is okay, that's what you want to see. What we also see sometimes is that everything is green, so everything is aligned, the results are passed as for SPF and for de-kim and still the mail is rejected. There are probably some people that don't want email from the tax office, don't know why, but yeah, it happens. Anyway, this gives you great insight into what's going on, especially in the beginning when you're working on the implementation. Yeah, it can give you a lot of insights without having to go into detail of every single report. I've also done some SPF dashboarding with it. For that you need, of course, the SPF record that, well, we showed here or something like that with the macros to get more insight into where this email coming from. You need your DNS, your DNS query logging enabled and ingest that in your work management system, so that you can get more information from that. Also very important is know what the good queries are, so fill out those macros for your mail servers. Well, obviously you need to put those in your DNS server, so know what the good queries are so you can rule them out and see what the bad queries are. There's some dashboarding. This is everything going fine. This is what we like to see. You see the IP address of the same email server, the healer, the from part, the domain part of the email address and whether that server is one of the servers that we stayed in our SPF record and what the DNS response was. Well, this is what we like to see. But what we also saw, especially in the beginning was this. These are our mail servers. Problem here is in the Halo stream. Mostly this is because there's a bad implementation of SPF. All of mail systems, what we saw were a lot of ISPs that provide free email. All you do hosting here, okay, here's an email server, do whatever you like, please. And I fill out the Halo run. They fill an unknown or local host or whatever their own DNS name. In the beginning, we got some complaints about our SPF record and that it wasn't okay, that it wasn't correct and well, we showed them what the problem was and the problem was on their side and they could fix it and everything was fine. Yeah, we have put a guideline on our website how you should configure those email servers. So what we're doing with the reference to the RSC and everything we do is completely legal and it's not sound, hoax-pocus. So yeah, but that gave us some problems in the beginning. What we also saw at the very beginning is this. What this is, the domain that is being used, lowpunlop.info is actually a domain from the Dutch text office. It is a domain that was used once for some campaign and it's now parked. We'll always keep our domains, so it's just parked. What you see here in the beginning are IP addresses that are not from us. I looked up a couple of those IP addresses. Those are mostly routers, so micro tick routers. Look them up, we show them and well, yeah, it's not that good. The Hilo that you see here is a fake one. The youngdex.net is a legitimate mail provider from Russia but the forward number O dot mail dot youngdex.net doesn't exist. We looked them up at the time of these queries and they don't exist, so it's just nothing. Also the from email domain, well, those are non-existing domains. So this is happening not on our mail servers. It's not our mail servers that send this. If you look on the other side, the querying IP is the IP of the DNS server that the mail server is using. Those are also not our servers. Those are owned by mail, the other Russian big mail provider. So this is happening entirely out of our scope. We're not a sender, we're not a receiver yet. We do see that this is happening because of the advanced SPF records that we have. If we wouldn't have those, we would never seen this. What we can do with this information is warn our public, put it on social media. If you see these numbers rising very rapidly, so you see a lot of DNS requests for those fake mail servers, we can put a warning out on Twitter, on Facebook, or whatnot to warn our customers that there is a phishing mail campaign going on and, well, that if they receive something which isn't likely, but if they receive something that they should ignore it. Also, if you want to go to the police, you have information now that you can provide them with so that they can do their investigations. There's more in the app. It has a couple of fighting strips. First of all, to process the reporting, but also to get your own SPF records and put them in a lookup so that you can use them within the dashboards. There's some more information about the RFCs. I was reading through the RFCs and you read them, you work on it. How was it again? Rereading it. Again, working on it, rereading it. So I thought, well, I make a dashboard of it so that everybody can use the summary, the highlights of the RFC. Some dashboarding about the number of queries per record type. There is a DNS help, more of a wizard, and some help with the advanced SPF records. This is the DNS record help wizard. It has some information about the dashboard, on what's used and how to use it. Well, you can just fill out the forms, fill out the fields, and then it will give you the records that you need. So here I filled it out. One remark, if you use the rough email address, be aware of properly privacy issues because there are mill headers inside which could not be yours. And there could also be privacy related information on it. So be aware of using rough. Yeah, that's one of the things. So if we take the previously abused domain, the love one love with info, and we fill out the fields here, we want a reject policy, and we want to send the rough emails to the ruby at blessings.nl. That's one important thing because that requires an additional record. I will show you later. And we want an SPF record, we want to redirect to our default SPF record and deny all other systems that will provide this. With just four DNS records, that domain is locked down. First off, you need a wildcard record for your domain. And that's because SPF doesn't inherit the policy of the main domain. So for every single existing or non-existing domain, you need a SPF policy. That was one of the problems that we saw previously with this domain. Those emails were sent from a non-existing domain. With a wildcard record and just minus all, you lock it down. So every time there is a DNS query for whatever set domain, there is a response. And yeah, if mental environments are just relying on SPF, if there is no answer for a SPF record, for the record query, for the text record, it is assumed that you don't want to use SPF. So that's why this one is important. There is the DMARC policy. That's the second line. The third line is the actual SPF policy that says, well, redirect to underscore SPF.Lustin.nl. So it's mainly saying, look over there. And the last one is for the DMARC mails. It's quite a long one, but you need that. Otherwise, you don't get any emails. DMARC related emails. That's done on purpose to prevent a denial of service attack. Otherwise, you could just register a domain, put a DMARC record on over there, and that says send out all the RUBA and RUF emails to another domain, send out a lot of phishing mails, and then the, well, the legitimate domain gets bombarded with DMARC reports, and they can be quite big. So you're creating a denial of service that way. That last one is most of the time forgotten, and people think, well, I don't receive any DMARC reports, so everything is fine, but it's not fine probably. You just don't have the correct policy set up. For our main domain, we needed a couple of more records, but that was because, yeah, we actually are using those domains. So we need to create the legitimate lookups that were done. Here we have 14 records. It's actually 15 because we also needed the underscore SPF.plusnews.nl with the actual policy. The benefit of this wizard is that you can just create a list of records that needs to be created and give those to the DNS administrators and ask them to just create those. It's more easy than that you ask them, please create a valid SPF record for me because they probably don't know what to create. Also, as you can see in the last column, it's a note as to why that record is needed and a reference to the RFC and the section that is stated there so that you can have a look over there if you want more info. Lessons learned from this. The most important thing is know your email environment. There is a good chance that if you are sending out newsletters or whatnot, that marketing or communication departments are using something like MailChimp or whatever. If you don't know that and you do not create the appropriate policies for that, those mails will fail. So you cannot reach your audience. Monitor your mail server logs if something fails. One advice is if you use those mail companies. Search party providers. Give them a subdomain of your main domain. Don't use your main domain because then you can create a very granular policy for that specific mail environment. You can give them a own DKIM key that will only be valid for that subdomain. So if something happens, they can compromise or whatever, you can just pull that DKIM key, create a new one, and you only have one party that is involved in it. Involve in it. Test it. Test your SPF policies and your DMARC policies. Test it first on domains that you have that you know that don't send out email. So you can see what's happening if it is working or not. It needs to be done in production because on production domains, you send email. But yeah, like we stated before, testing on production keeps you sharp. Don't forget a SPF record for your wildcard domains for your subdomain. So use a wildcard domain. And yeah, have a look on my GitHub. Everything is there. Download it, use it, abuse it and answer it. If you have questions, go over there. Final thoughts. Final thoughts. Well, the main one is that you can get info on things that are happening outside of your span of control. So if mail is sent from servers that are not your own to servers that are not your own, but the domain is yours, you can get info on that. Keep in close contact with your MGA administrators and your network administrators, with your DNS administrators, because they need to know if they change something, I don't know, put in an additional mail server, it has consequences. You need to create additional DNS records. If you keep them close, well, they know and they will inform you and things will go out, go working. Please use the standards, the RFCs and obey to them. Don't be that one specific, retinent-based party that doesn't send out DMARC records. That doesn't send out a DMARC report because they are important to people. They want to know what's happening with them. So yeah, then. Future plans. Shall I do that? Yeah. Okay. So future plans, we presented these standards and the implementation we have done and we had a discussion with Craig, which is from spro.com. He created that website, which he did an initial scan on the internet if SPF with markers are used, but it's not maintained anymore. So we decided to create our own site, which we tried to get updated. The basis has been done. We create the scripts and tools to retrieve data which is presented in this presentation in stock. Among tools, we used tools like Slurm, which is a high-performance computing platform for job management. Cluster refers to create a cluster file system and by Splunk, we're using Splunk. At the moment, we have indexed more than 352 million domains. I think it's actually now 3.16 million. We are starting creating dashboards with all the data we have retrieved and want to publish it. But we need your help. Which kind of information you will see, which kind of dashboard we should create. We have a lot of data. We have data about SPF. We have data about DNSSEC, DKIM, DMARC. We start using also, which is not part of the presentation yet, with scraping BIMI data. BIMI is a new standard. If you send an email to somebody who has a BIMI capable mail client, then you will see in front of your email address the logo of the organization. Actually, what's behind the main part. And then you can be more sure that the mail is coming from the original recipient where it should be sent from. So at the moment, the scripting we've built is doing about 1,000 DNS queries per second with three systems. So 3,000 DNS queries per second. Thanks to Cloudflare. Thank you, Cloudflare. You're allowed to do that. And eventually it will be published on www.rocanalyzer.net. Planning is the end of this year. Yeah. And we have a lot of domain sources already, but we need also more domains. And actually the CCTLDs are a bit problematic. So the .nl, .b, .de, et cetera, because they didn't open their zone files. If somebody has hints, the hints are welcome. At the moment, we're using zonefiles.ir, who is data, the certificate, the transparency log, and some other smaller sources. So if somebody has hints, please let us know. And if you want to work with us on making this available, please let us know, contact us, because we open, always help is always welcome. And then the resource, all of you, the Github, all the RFCs that were discussed with a reference to, well, if you also want to read them, like I did, and a DMARC adoption report, which is also quite interesting. It's from 2018, but yeah. It is very interesting. And yeah, well, questions, we'll gladly receive them. Thanks for listening and have fun at DEF CON. And hopefully we'll see you in real life next year. If we, the Europeans, are allowed to go into this. That would be awesome.