 it that way. There we are. That looks a lot better. So I was one of the founding team members at Agari. We've been around for about five years. I have helped a little bit in the fight against cybercrime. One thing that I did is I helped Microsoft and the FBI with the B54 Citadel botnet takedown. If we get some time I can maybe share a little more about that at the end. I have worked at large and small companies. I started my career at Oracle, went on to SAP, doing like application stuff. And then I branched off, went to make my internet millions, and have been CTO of a number of startups. And currently at Agari I am our Field Chief Technology Officer. It's kind of like Santa has elves. I'm sort of the CTO's elf, and I come out and speak at conferences and such. A little bit about Agari. Agari's mission in life is to eliminate email as a vector for cybercrime. And yeah, exactly. Hey, it's good to have a big goal, right? And if we even get close to that goal I think we'll actually be pretty darn happy. We've got quite a few clients that you've probably all heard of because we help people stop fishing of their brand. Well, obviously the brands that get fished tend to be companies that you have heard of. So four of the top five US banks, three of the top five banks in the UK, and four of the top five social networks use Agari to protect their email stream between them and their customers. So all right, today I'm going to cover, first of all, why do cybercriminals love email? Probably everyone in this room already knows the answer to that. I'll tell you about a few previous attempts to solve the whole email spoofing, email fraud problem. And then we'll talk about what I believe is the current best solution which is a combination of SPS, Deakin, and DMark. We'll get into the technical details of those. And then I'm going to show you some open source tools that you can use. If you guys want to go and try to implement these things yourself, you don't have to pay money to a company like Agari. There are tools out there. You can roll your own basically and not spend a dime. We'll talk a little bit about some edge cases and a few things that DMark doesn't necessarily play well with and some potential solutions there. And then we're going to do something fun. We're going to take a data breach, specifically the most salacious data breach of last year, the Ashley Madison data breach. And we're going to use that data to estimate DMark coverage by country. So that should be a little bit interesting. And then finally, if we have some time at the end, I've got some additional fraud examples that I'll share because some of them are a little bit entertaining. So the whole reason we're here goes back to 1982. So SMTP was sort of first rolled out in 1982, and the inventors, they did such a great job. We're still using the protocol today, but they forgot one very important thing, and that was authentication. Basically, I can connect to a mail server and say, hey, I am, in this case, security at WellsFargo.com. And I have an urgent message for you and you can imagine if I was actually a cyber criminal, I would lead people off to a fake Wells Fargo login page, capture the username and password, go in there and move some money. So what do criminals do with this flaw? Well, here's an example, a somewhat more benign example, unless you actually go and try to buy these products. Basically, this is what I call pharma spam. So I'm sure everyone in this room has received at least one offering to sell you usually Viagra, Cialis, various other medications without a prescription. And the problem is if you actually do go and click that link and go to this website, this is not really the Canadian healthcare mall. These are usually operated out of countries that are, how shall we say not, don't have extradition treaties with the U.S. The pills they will sell you may or may not have the active ingredient in them. I guarantee you they're not genuine Pfizer pills. They are knockoffs. So good luck to you if you go and order some of those. You might get very ill from them. Here's another example, phishing, extremely common. I had a really good PayPal example and my guy over at PayPal said, please don't use our example. And I'm like, really? Come on. Everybody knows you guys get phished off. But he's like, no, we don't want to highlight that. So okay, fine. So I used an Apple example. And this is actually a pretty poor example. I've seen much better ones. But here you can see, it's a last notice here. We're going to suspend your account if you don't click this link and give us your credentials because somehow, of course, giving my username and password is going to let them keep my account open. If you click the link, you end up over at this lovely little website which looks pretty darn genuine except to the technology people in the room who might actually check that address bar and note that not only is the little green padlock missing, it doesn't bear any resemblance shape or form to Apple. And then finally, again, a very poorly done example because all my really good, well-done examples, the folks didn't want me to show them to you today. But this is an example of malware. Again, this one's horrible. It just says invoice. And then the attachment name is called augmentingmommy.zip. I think if they'd spent maybe just five more minutes fine tuning this, they would have had a little bit better response rate. But if we take that over to VirusTotal, we find out that, yes, indeed, 43 out of 55 of the scanners they run over there that determined that it was some sort of nasty stuff, basically a keylogger. All right, so that's what the bad guys are out doing. I didn't even talk about the most common thing we're seeing lately which is the business email compromise. This is when they send your finance team and email supposedly from the CEO saying, hey, I got to wire $3 million to China and we have to do it immediately and I want you to keep it hush-hush. And you might all chuckle. Mattel lost $3.2 million. Ubiquity Networks lost $46 million. They later got about six of it back. And there was one other big one, I forget the name of it that made all the headlines that lost like another 10 or 15 million. So, yeah, it's out there. In fact, was it you? No. Yeah, everybody gets these emails and sadly once in a while somebody falls for them and it's horrible because the money's wired away and typically the banks are going to reimburse you when there's a certain degree of carelessness and stupidity involved. So, previous attempts to fix this. So, one of the first things that people tried was this idea of sender policy framework. In concept, very, very simple and practice fraught with issues. But the idea is there's my mail server. Its IP address is 1.2.3.4. That's where all my mail comes from. I'm going to publish a record in my DNS and I'm going to tell everybody, if you get a message from my domain, it better come from that IP address. There's problems with that that we'll get into in a moment. Not to be outdone, Microsoft said, hey, if SPF is cool, we're going to come up with this thing called sender ID. It's going to be just like SPF only better. It did finally die a rather inglorious death. They finally have agreed to sort of stop using sender ID. But basically sender ID offered the additional capability of not just authenticating the RFC 821 envelope, but actually could authenticate the RFC 822 from header, which seems like a great idea. The problem is nobody wanted to go and do that. And the reason is you could not render a decision during the SMTP transaction before you accepted the entire message. So, that was really the benefit of SPF was the fact that you could verify it right at the beginnings of that SMTP conversation. You didn't have to get down into the full message content. Then there was domain keys. This was something that Yahoo did. It was the idea of a digital signature applied to a message. And then Cisco sort of joined forces with them and fine tuned it and came up with domain keys identified mail. Pretty much the same thing, just a few minor differences that made it slightly more secure. And DKIM is by far the more widely accepted and adopted standard today. Then we had author domain signing practices, ADSP. The idea here was how is a receiver supposed to know if you sign your mail or not? Because DKIM is optional. You don't have to do it. And so the idea behind ADSP was if I sign all my mail I'll publish a little record in DNS that says I sign all my mail. So if you get a message that's not signed you probably ought to just drop it because it's not really from me. In practice nobody really implemented it. It was a little baby step that nobody really embraced. This next one is near and dear to me. The idea of visual trust indicators. So my last company before Agari was a company called Brand Mail Solutions. And we were in competition with Iconics and Goodmail. Anybody ever hear of any of those companies? Yeah. So the idea behind all of them was that you would have some kind of email authentication and then if the message checked out you would actually flag it in some visual way as being authentic. And here's an example from the old Brand Mail homepage showing you a typical mailbox where everything looks the same. And then you have a branded mailbox where this particular message from Quinn Direct is clearly identified as being authentic. And then if you hover over that little envelope you would get some additional detail. Brand Mail and Iconics did it based on open standards. Goodmail did it based on a proprietary technology. Basically all the companies are gone now except Iconics is amazingly still in business. In fact we sold our IP, our patents, etc. to Iconics a couple of years ago from Brand Mail. And then finally we had the famous PayPal experiment of 2007. So PayPal was a pretty early adopter of domain keys and SPF. And as they looked at their list of users the vast majority of them got their mail at Yahoo at least 50% of them. So they figured a lot of this phishing problem could go away and they just got Yahoo to check the SPF and domain keys. And if they didn't pass block the message. Don't even stick it in the spam folder. Just kill it outright. And maybe provide a little information back to PayPal. So just in case something goes wrong PayPal is aware that something has gone wrong. And amazingly it worked. When they put this in it stopped the ability of criminals to spoof PayPal.com to Yahoo inboxes overnight. But there's a problem. And that is great. It sells it for PayPal and it sells it for specifically PayPal users at Yahoo. It doesn't do anything for the Gmail users or the AOL users who also use PayPal. And it certainly doesn't do anything for JP Morgan Chaser Bank of America users who get their mail at Yahoo. And so right about that time there were some rumblings and by the time we got to about 2009 the rumblings got pretty loud. And we started actually discussing how could we take that experiment and get scale to the Internet. And what we came up with is DMARC. I'll give it one shot here. Domain based message authentication, reporting, and conformance. Should have seen me the first week that came out. I would bungle it every single time. I've gotten a little better with it. Anyway, so we're going to talk about DMARC a little bit today because in my view that is the one that's got the best legs at least at the moment has the greatest chance of solving this problem. In case any of you are interested in the standards there are some of the standards and of course the slides are available online I believe later today or whenever. So you will have access to the list of the standards if you want to really give yourself something to help fall asleep by. In terms of how this all works you've got SMTP at the bottom, right? That's moving the mail around the Internet, no authentication. Then you've got SPS. That's like basically your list of IP addresses that say this is where my mail comes from. And then you've got DKIM which is a digital signature applied by the mail server that can be validated at the receiving end. Big problems though with SPF and DKIM. The biggest problem is most people don't know where all their mail comes from. You know when I said like, oh my mail comes from that server right there. That's what all the techies think. Ask your marketing department where your mail comes from. Probably comes from a place like Exact Target or Constant Contact, Vertical Response or any one of about a thousand other companies that provide marketing services. Ask your HR team how they do recruiting. Oh, I bet you they have some company out there. Maybe Tileo which is now owned by Oracle. Sending out mail saying, hey come work at company X. We've got great jobs. Simply put if you don't know what that email landscape looks like you're going to bungle your SPF so that's not going to work. And of course if you don't go and give these guys a key to sign with and set that, you know, the public part in your DNS you're not going to be able to use DKIM either with those folks. So the big challenge here was how do you figure out where all your mail is coming from so that you get your SPF and DKIM right? As long as SPF and DKIM are working this system is brilliant but if they're not working your real mail is going to go missing. So what we came up with was DMARC and the idea was first and foremost give the domain owner a way to figure out where all their mail is coming from. Tell them where it's passing SPF, where it's failing, where the stuff is being deconsigned, where it's not and also as a side effect figure out where all the criminal email is coming from. And then once somebody uses that data and cleans up their SPF and DKIM and gets it working really well give them a way to do like PayPal did and said hey from now on I don't want to accept a mail that fails the standards. And so DMARC was born basically January of 2012 with a rather small announcement from Google, Yahoo, AOL and Microsoft who all said we support this standard and actually they had been supporting it for several weeks prior to the announcement and we had already gotten most of the kinks worked out. So let's go a step deeper here. So first of all hopefully this is all review but this is SMTP. Basically you tell them that to port 25 on the recipients mail server which by the way you can look up by looking up the MX host in DNF and essentially when you connect the server says something like here's my name and I'm ready and then as a client you say hello and you give your name and the server says something along the lines of 250, codes that start with a 2 is like a success code, 4s are sort of temporary errors, 3s are kind of just informational and then 5s are actually errors. So it says okay fine and then I say hey I've got a message from test at test.com and then they're typically going to say hey the sender is okay. And then I'm going to say I have a message for in this case my personal Yahoo account maybe I shouldn't have shown that to this room please don't start spamming me anyway and then recipients find and then we say data and then typically they say go ahead and then is the message. Now notice down here we have a from header. I'm going to get into this a little more in a second but this is the difference between the RFC 821 envelope sender and the RFC 822 from header. Finally you end with a dot on the line by itself. They say okay or in some cases not as you will see and then you say quit and now I have handed the message off to Yahoo and presumably Yahoo is going to go deliver it. Alright let's talk a little bit about identity. So here is sort of just the client side of that conversation. We have several different identities in here. We have the hello domain. We have the envelope domain. We have the signing domain if there was a decim signature. And finally we have the from header domain. Now in some cases those are all going to be the same and that's going to be fantastic. Everything is very very clear. However in some cases they are not the same and we run into a situation where we have what's known as an identifier misalignment and that's something we'll need to address with the DMARC standard. Alright here's a couple of SPF records. I love Apple's SPF records. One of the advantages of having a Class A network with all 16 million or however many, maybe even more than that, a lot of IP addresses all starting with the Octet 17. And interestingly enough they are telling the world that's the only place we send mail from. So apparently Apple does not use any at least for Apple.com does not use any third party services. PayPal has a much more complicated setup. They actually have all these little mini-includes and those includes have other includes and eventually they resolve down to lists of IP addresses. Alright let's look at some typical mail flows. Okay in the simplest world you know you've got your PostFix server and you send to a Gmail user. That green host is going to be in your SPF record. Maybe you've got Exchange on-premise and then you're sending to Office 365 and they're actually sending it out to the final recipient. In this case you would list the Office 365 server in your SPF record because it's the last pop. That's what the internet sees as your mail coming from. And then what's that? Yes exactly. And then finally you have other situations what I call relays or forwarders. Your send mail server sends to some address at UCLA but it turns out that person already graduated and they have a little pointer setup at their email address and it's going to bounce it over to say Yahoo. Well unfortunately whoever the sender of this is is going to list that send mail server in their SPF record. They don't know anything about that UCLA server so when Yahoo sees the message that message is not going to pass SPF because we've introduced an extra hop. This is why we use two standards. Why we don't just rely on SPF. So SPF breaks due to relaying or auto forwarding. Mailing lists are a huge problem for SPF and that's going to be one of my edge cases we talk about in a moment. It authenticates the envelope domain which is the mail from. You and I don't typically see that when we get an email message. We're looking at the from header. So authenticating something I never see isn't probably the wisest thing in the world. SPF has this interesting little quirk. It was designed to prevent SPF from being used to commit a DDoS attack and that is that you have to have a maximum of 10 DNS lookups required to fully parse your record. So as you have all these different includes and they kind of keep chaining or whatever, the idea is that after 10 lookups the receiver is just going to throw its hands up in the air and say give up. Yes. It doesn't include the root one. So technically you get 11 total. Basically A tags, MX tags, PTR tags, the other one includes and redirects each each one. An IPv4 or an IPv6 tag does not eat any lookup. And there are a couple of tools out there that you can use online that will actually tell you. They'll analyze the record and say oh your record has 9 lookups, your record has 11 lookups. And then finally, a lot of people forget about a special case. You guys ever go on vacation? You ever set up one of those vacation messages? I'm out of the office until January 12th or until January 25th. I'm at the Scale Linux conference. Those are out of office messages. Another similar situation. I'm sure everyone here has gotten one. You fat finger an email address and you get a bounce back, a non-delivery report. Well turns out the envelope domain in both of those situations is empty. It's null. So what the heck are you going to do the SPF check against? Well the specification says you're going to check the hello name in those instances. Well you got to make sure that you put an SPF record in for those hello names for each mail server that you have or non-delivery reports and out of office messages are not going to pass. And it looks something like this. You put one of these in for each of your MPA which means this host name and a minus all. So in other words any IP address that maps to that name will be acceptable. Alright. Switching gears a little bit. Domain keys identified mail. DKIM. This is the other standard. Now this one is kind of neat because it will survive that forwarding typically. As long as nobody tampers with the message in transit we don't care if it takes the short route or goes through a thousand servers on the way to its destination. So here's an example. This is actually added as a header in the email message and there's some interesting pieces here. If we look here this is the key lookup if you will. So Yahoo would look up this key if it received a message like the one up there with that header in it to validate the signature. And so the way they get the key is they take the selector or the S equals tag and then they take the D equals or the signing domain then they stick them together with underscore domain key in between and they do a DNS text lookup and the result they get this key which is in base 64 format. That key can then be used to validate the signature. There's some other interesting parts of the signature. If you look here I can show you. Here you can see where it says H equals date colon from, colon to, colon subject, colon etc. Those are the headers that are signed. Little helpful hint do not sign headers that are likely to change. Signing the received header may be not the best idea in the world because those kind of typically get changed and added along the way. But that's a pretty good set right there. You also don't want to not sign a header that's fairly important. If you don't sign the subject header somebody can take that message, change the subject, and still have the signature validate. So you have to exercise a little caution there when you're deciding what to sign, what not to sign. All right, there are a few issues with Decim. If Decim was perfect we would just say well don't use SPF, just use Decim, right? First of all there's such a thing as what I would call benign message modifications. Anybody ever run a mail server that you know like to take the from and the to addresses and put them in a very specific format? You know you can have the friendly part in double quotes or you can just have it without the double quotes and then you put the real address and angle brackets. There's a few different ways you can do it. Well some mail servers have a certain way they prefer to see it and they will rewrite those addresses to fit what they believe is their preferred pattern. Well that's basically a benign modification but it is a modification nonetheless that will break the Decim signature. Most notably we discovered Cornell University does this with all their alumni. So if you are a Cornell alum and you are finding you are not getting your Facebook, LinkedIn, or Twitter notifications this is probably why because they like to rewrite that. Again mailing lists typically break Decim. Now they don't have to but the main reason they typically break is because mailing lists like to mark up the subject line. They usually like to put the list name and angle brackets at the beginning or square brackets at the beginning. It's a fairly common thing. They also often like to put a little footer at the bottom. Hey this was sent to you by the such and such list. Don't subscribe, click here that kind of thing. Those are message modifications. So if you send a signed message to say the Southern California Linux distribution list when that gets re-mailed that signature is probably not going to be valid anymore. Also Decim validates that signing identity. It does not necessarily validate the from header. I could sign your message. It doesn't mean your message is any more or less authentic. All we can say is that we know I signed it. So the best situation is of course when the signer and the from header are one and the same because then we know that the signer actually did offer the message as well. Decim does have the potential for a replay attack. So now there are things to mitigate this. There's an expiration. You can put a special tag in the Decim signatures. You can say hey this signature expires in 15, 20, 30 minutes, whatever, some reasonably short period of time. Now the only thing somebody could do if they replayed this would be to just basically take a signed message that was sent by a legitimate entity and then send it to you know a billion of their closest friends. It might create some headache for the person who originally wrote the message because people would feel they were getting spammed. But it's not like you can go and change the message and make it more useful to the attacker. It just means that you would have to replay that precise identical message. And then finally with the advent of cloud computing and more importantly cheap cloud computing it turns out you can kind of basically you can factor these keys. It turns out if you have a little bit of money and a little bit of patience. And also Decim keys are typically quite long-lived. You publish them in your DNS. I've seen companies that are still using the same keys after five years. I recommend that six months to a year you might want to consider rotating them. In any case Google will not accept a key smaller than 1,024 bits. So if you sign something with a smaller key, send that message to Google they will just say oh it was signed with too small of a key. We're going to pretend it wasn't signed. So that's their solution to it. My recommendation at least use 1024 if possible use 2048 bits in your key. Alright so on to the final standard here, DMARC. So once again we're going to overload that DNS system, make it really work. It's earned its keep here. They're stored as DNS text resources and they're stored at underscore DMARC, so the special location. And they have a very special format. I've just looked the one up for Twitter and it basically says they're using version 1 of the specification. Their policy is they would like receivers to reject messages which cannot be proven to originate from Twitter. They would like aggregate and forensic data sent to those two addresses you see there which happen to be my company because like I said we process the DMARC data for many companies that want to use get the business value out of it but don't necessarily want to spend many many hours coding and putting together open source solutions and such. And then finally there's this option the FO tag which is the failure options tag. The domain owner would like forensic reports for any messages that have failed either SPF or DKIM. In other words one or more protocols. You can also add that to FO equals zero meaning you want only the things that have failed. I believe that means that only the things that have passed no standards. You can also say S or D saying I only want to know about the DKIM failures or I only want to know about the SPF failures. A little hint here. Notice the format of the RUA and the RUF tags. There's this little mail to colon in front of the email address. The idea there was that eventually there might be other mechanisms to transmit the data. Today it's sent via email. Don't forget the mail to. Your record probably won't work if you forget that and it's a very common error. Let's talk a little bit about subdomains. Not everybody sends from twitter.com. Sometimes they send from news.twitter.com or alerts.twitter.com. Subdomaining can be a really neat thing. It helps you divide your mail streams out. It helps you get around things like SPF record lookup limits. Suppose you have six different vendors that are doing various things on behalf of your company. You give them each a different subdomain and then their SPF record only has to include their hosts. You're not going to run into that lookup limit. As far as DMARC is concerned, DMARC records are inherited by subdomains automatically. This is fairly important to note. If you put your top level at reject and you didn't realize that there were a bunch of subdomains in use, you might have just stopped their mail flowing. Not a good thing. However, there is an SP tag that you can specify which will say, this is my top level policy but for every subdomain I want the default to be this other policy. You can use the SP tag. Finally, subdomains can have their own records which override the higher level record. Here I've done a couple of examples. If we had DMARC.example.com at reject with an SPF quarantine, food.example.com with a policy of none which basically means monitor only, don't take any special action, and a subdomain policy of reject, and finally bob.example.com with a policy of reject. You get the following little matrix. So top level obviously is reject. bar.example.com is quarantine because there's no specific record overriding it, and the top level record says SP equals quarantine. So that would apply. Similarly, food.example.com has P equals none. It has its own record it overrides. Billy.bob.example.com is going to be reject because it's going to take the bob.example.com. There's no SP tag so it's going to use the P tag. And then finally john.food.example.com will be reject because of the SP equals reject on the record for food.example.com. Yes. Nope. Nope, by default. So at the top level, if your top level is reject, then all of those will be reject by default. If you want the top level say to be monitor, but everything else to be reject, you can say P equals none, SP equals reject. And that means all subdomains are going to be at reject. So yeah, infinite flexibility basically. All right. One other little gotcha with DMARC. Let's suppose you want to send your data to another domain. What a great way to spam somebody, right? Publish a DMARC record and I take your email address and I say, hey, send all my DMARC data to him. Probably he's going to get annoyed pretty quickly. And the folks that thought of this particular challenge, and so if you want to send your DMARC data, you can send it to your own domain. No problem whatsoever. However, if you wanted to send it to somebody else's domain, there's one extra little step. So you need to get the domain that's going to receive your DMARC data to publish a very special DNF text record where it's basically your domain dot underscore report dot underscore DMARC dot their domain. And then all they do is they say V equals DMARC1. If that record is present, then Google Yahoo, et cetera will say, yeah, okay, they're willing to accept data for domain X. I'll give you a little hint. Google, by the way, will not, you know, how are you going to send your DMARC data to your Gmail account? Well, you're not unless you're Google because you could send it for gmail.com. However, Yahoo has a nice little wild card in there and they will happily accept DMARC data for anybody. So if you've got nowhere to send your DMARC data, set up a free account on Yahoo and send it there. I just did a dig there for anyrandomedomain.org dot underscore report, et cetera, and lo and behold, they do in fact return the all-important V equals DMARC1. All right, so let's suppose you publish a DMARC record. What's going to happen? Well, at the address you specified in your RUA tag, you're going to start getting some data. It's typically going to happen, usually happens about 5.36 p.m. because that's when UTC, you know, time zone switches over to the next day. They, you know, all of these guys, you know, run their MapReduce or their, you know, Hadoop or whatever crunch through piles of data. I can't even imagine, like, for somebody like a Google or a Yahoo, to basically go look at the last 24 hours of data that your mail has come in and prepare all of these reports. But what they're going to do, they're going to email you a zipped XML file. And that's actually a gzipped XML file. And that XML file is going to look something like this. There's going to be a header part. And the header is going to basically have certain tags. This particular thing is being reported by Yahoo. If we wanted to, you know, ask somebody about this report, well, we'd ask Postmaster at dmarked at yahoo.com. And they've given us a report ID that we could presumably refer to. Those are supposed to be unique, though in practice I've occasionally seen collisions there as well. Then we have the date range that this refers to. Those are just UNIX timestamps. And the astute among you will realize this example is a little bit old. I probably pulled it several years ago. It basically refers to a 24-hour period. It's very typical that people report on a 24-hour boundary. And it's specifically for the domain alertsp.chase.com. It's some kind of alerting that JPMorgan Chase does. They're, oh, I forgot to talk about alignment options. So remember, we've got all those different identities, right? We've got SPF checking the envelope identity. We've got decim checking the signing identity. And then dmark's all focused on the from identity. Well, the dmark standard says, if I'm going to use the decim signature, then the signing identity must be aligned with the from header. Same thing with SPF. If I'm going to do that, the envelope has to be aligned with the from header. Now there are two flavors of alignment. Strict and relaxed. Strict has to be an exact match. If it's food.bar.com here, it's got to be food.bar.com here. Relaxed, you can have parent child or even sibling domain. So you could have, you know, bar1.food.com and you could have bar2.food.com. They would both roll up to food.com. And yes, they do take into account things like .co.uk. So they know that that's actually sort of one step there. So in any case, this record is saying that Chase's dmark record for alertsp.chase.com specifies a decim alignment of relaxed and an SPF alignment of relaxed. If that was an S, it would be a strict alignment. This literally comes straight out of the original policy. So in other words, if you queried for the dmark record, basically Yahoo just playing back saying, this is the dmark record we found for you. And the policy is to reject mail. You also see the PCT percent. They would like that applied to 100% of the mail. Now you might be like, why on earth would you not want to apply it to 100% of the mail? It turns out that when you're first turning this stuff on, some people, especially at large corporations, get a little nervous and they want to ramp it up. So they may say, you know what, start off, just block 1% of the failures. Choose them at random. And then start blocking 2% of the failures and eventually get it up to 100%. I personally think if the data is telling you it's clean, it's clean, go to 100%. But you know, I don't work for a bank or any other highly regulated industry that would fire me instantly for doing something like that. So after this header, you then are going to get a whole pile of these individual records. Gotta love XML the way they can take a tiny little bit of data and make it take up this much space. But anyway, thank goodness. Now you know why they're GZipped. This particular record says that SourceIP 10610 149.115 sent one message. It passed DKIM, it failed SPF, and no special action was taken against it because of course it passed DKIM. It's an either or, by the way. You don't have to pass SPF and DKIM. You have to pass one or the other and be aligned. Then they're telling us the from header was alertsp.chase.com. The DKIM domain was alertsp.chase.com and the result was a pass. The SPF unfortunately failed. Now does anybody, what can you tell me about this? This particular record. DKIM passed SPF failed. Remember that diagram I showed you where we go through UCLA on our way to our final destination? This is one of those situations. DKIM survived, but that host 10610 149.115 does not belong to JPMorgan Chase. It probably belongs to some university, some mailing list, something, some relay. By the way, I've seen these files get as big as 600 megabytes after they're decompressed. They get big. Sorry, wait, did I say 600 megabytes? Almost a gig. All it takes is one good botnet attack and you can blow your file up in a hurry. Now there's another type of DMARC data. Not everybody participates. Specifically, Microsoft participates. Yahoo sort of participates. Yahoo will give you forensic data if you work with Agari or one other competitor where we have a private relationship with them. And the main reason they won't send forensic data just anyone they're afraid of in the event of an attack, your mail server will be overrun with them sending forensics. But forensics are basically copies of the messages that have failed one or more of the protocols. Here's an example of one they are sent in ARF, Abuse Reporting Format. If any of you are familiar with that, it's basically a way you encapsulate an ARF, an ARF C822 message inside of a MIME type that's carried along. So this is the header, basically the thing giving us a little bit of information and then there's the actual attachment that my Mac mail client here actually displayed in line but that actually is a separate, you know, ARF C822 MIME type. We're going to get into that a little more in a minute because that really lets us do some nifty things to sort of take a stab back at the criminals. So this is what happens every single day now at Yahoo, Gmail, Comcast, people who are running iron ports, Symantec and a bunch of other places. I'll walk you through this. So a message comes in out of the Internet and the real question they're asking is, you know, do SPF or D can prove the message came from the domain on the from header? Yes or no? Then is the content, okay, so assuming it passes, notice we are not skipping the spam check. Just because something can be proven to come from who it claims to come from, it could still be spam. So yes, you could still end up in the spam folder. However, if SPF and DKIM don't prove the message was sent by that entity, we go to the second site over here, we look up the DMARC policy. And if it is none, we just send it over here, run it through the spam filter, business as usual. We are going to report about it, of course. If it's quarantine, we send it to the spam folder and if it is reject, we send it to the trash. There also are the concept, there's the concept of local policy overrides. That is something where the receiver can at their option say we know better and we are going to ignore the policy. We cringe when we see that at Agari because we watch them let fraud in and we are like, why did you do that to your users? The bank tried to tell you and yet you let it through and they are doing it less and less, but there is the option to do a local policy override. So let's look at this in action. Here's a nice little session of me telemetting to Port 25 over to Yahoo. This time instead of pretending to be Wells Fargo, who is just starting to do DMARC, I pretended to be security at Chase.com who have a very mature DMARC implementation. And you will notice instead of that nice 250 OK at the bottom, we get an error code 554, message not accepted for policy reasons. In other words, it is actually rejected in line during the SMTP conversation. In other words, you get a bounce back. If you are a human user and something went wrong, you are going to get a bounce back and it is going to tell you that it failed because of DMARC. And then you are going to go talk to your email admin and you are going to say, OK, why did my message fail DMARC? All right, a couple of limitations. So DMARC does not do anything to solve the friendly from problem. What is the friendly from problem? Well, as you all know, there is the email address and then there is the display name. And these mail clients, they try to be so nice to dumb it down for all of us and instead they have made us all a lot less safe. Here is an example. ChaseCard182 at hotmail.com. I bet you it might even pass authentication. So one thing that might help here is the idea of inbox differentiation. So my now defunct company, Brandmail is no longer around to do this. However, Google was running a little experiment we called the Google Gold Key. It is in a Google lab right now where they will actually put a little key icon next to messages that they have checked the authentication on it. So this maybe might be a way forward. Not a guarantee. Only works at Google and only if you have the lab. But the main point being you still want to check that email address. Don't just look and you know if it is your CEO's name or your bank's name. Don't assume that the email address is really what it says it is. Also, you know, if you are running your own mail server, you know, like Hillary Clinton tried to do there for a little while while in the State Department. You know, you are not going to be protected by this unless you go and install some of this open store stuff we are going to talk about in a moment. However, more and more places are implementing D-Marks. So the pool is shrinking. And there is also a certain sense of herd immunity. What we have noted is that when a big consumer domain publishes a reject policy we usually see the criminals kind of go away and move on and pick a different brand. You know, if you can only hit 10% of the bank's users now and you got to guess who they are it is probably easier to just go find a bank that hasn't locked the doors yet. So there is some benefit there. Alright, a couple of years ago yahoo.com in other words the domain that all of you get if you get a free yahoo account published a P equals reject policy. And this did a couple of things. First and foremost, all the folks that basically had no idea how to go get their own email address and had simply been using their yahoo mail address to run their flower shop or their plumbing supply company or insert small mom and pop business here they all suddenly found that their thing they were running with constant contact was no longer working because of course yahoo didn't authorize that they only authorized the yahoo mail servers. The other thing that happened was a lot of folks who ran mailing lists started to scream started to scream very loudly yahoo you have broken my mailing list what did they mean by that? Well it turns out mailing lists don't work super well with DMARC for two reasons. Number one there's that extra hop so SPF forget it, it's not going to work and mailing lists like to modify stuff they like to change the subject line they like to put a photo on the message so DKIM is out the door but here's kind of the gotcha so let's suppose now somebody who's been on this mailing list for the last 10 years they have a yahoo.com address they decide to post today in the post reject world that goes to the mailing list server and the mailing list server goes and sends it to all 12,000 people on the list and a lot of those people get their mail at Gmail, Hotmail, AOL or other places that do DMARC however the message failed DMARC they're going to bounce it back but wait it gets worse most mailing lists have a little feature in there that hey if you try to send to a user three times and it bounces three times stop sending to that user forever so guess what happened as soon as about three or four people from yahoo posted on any given mailing list everybody who was on Gmail Hotmail, yahoo, etc, etc got unsubscribed automatically from the list so this is what they meant when they said yahoo broke my mailing list well yahoo didn't break the mailing list yahoo is trying to protect people from fake yahoo messages and in doing so forgot a very important edge case that was a pretty big deal for yahoo.com not such a big deal for most companies ok I bet you most of the people in this room because you're all kind of engineering types probably do participate in a lot of mailing lists we are kind of not the normal folks if you will but for the average business user it's that big of a deal what are some options though to deal with it well you can put the onus on the list participant so you can say hey list participant don't use a yahoo.com go get a gmail.com address they don't do p equals reject yet or go pick some other domain if your company has gone to reject ask them to create a subdomain that they leave it at p equals none so your mailing list traffic the big problem with that the advantage of course is you don't have to change the mailing list software at all or anything else you just change the address you're going to use the downside is you know ok so you switched to gmail I guarantee you gmail is going to do this at some point too and if you use a subdomain of your corporate domain then your company has left a gaping hole that the criminals could take advantage of we can put the onus on the list operator right instead of doing what they normally do send the messages from the list itself so yahoo user emails the list but now when it goes out to everybody instead of the from header saying the original guy from yahoo move him to the reply to and then use the list email address in the from header and that actually works pretty well you know and it keeps the mail flowing however it does change the semantics of reply versus reply all I guess mailing list users are used to when you hit reply it's going to just go back to the list is it or vice versa anyway the point being that by shifting those around some people were complaining that reply and reply all basically now re inverted their their use the other also you got to get the list operator on board the list operator may or may not be willing to adjust something that they see is not created by them fixing a problem there not all mailing list software you know has an option even to do this the good news is one of the most common ones mailman does and there is a link right there where they literally tell you exactly how to set it up which patch you need etc etc so if you guys operate a mailing list server and you want to keep your yahoo and AOL users happy you might want to have a peek at that it's a pretty straightforward thing you can also put the onus on the list operator and basically do the status quo but just don't unsubscribe people for dmark failures at least you won't you know the guy yahoo won't be able to post anymore but at least he won't unsubscribe everybody at gmail and AOL in doing so and then finally the receiver could detect that this is mailing list traffic and they could override the policy and this is actually what google does in a lot of cases however this is dangerous because now a spammer can use that mailing list as a way to send a fake paypal message for example get the list basically to deliver it and the policy will be overridden so I don't think that's really a great solution the one that seems the most promising at the moment but is way too new the jury is still out is a new thing called authenticated received chain or arc google has basically said look this is what we're going to do to fix google groups for this particular problem we're going to implement it internally and by the way we're going to put the specification out there so if anybody else who runs a mailing list wants to do this it's a great solution it basically involves a couple of extra headers that are added they're digitally signed and then every time this goes through another hop somebody can basically re-sign the thing and say that oh yes indeed I checked the authentication and I'm passing it along basically it gives a way that by the time it arrives at google assuming they trust the people that have said they checked the authentication I'm going to read more about it on arc-spec.org how we doing on time got about 10 minutes left open source tools open deakim you can go download it, what's it do for you well it's a C library so if you want to play with signatures you can code against that library and do some nifty things play around with it but it also comes as a filter it'll work with send mail of course and also with exim and with a little extra shim layer in there you can make it work with qmail basically it's a little fine for you it will also verify for you if you want it to do that yeah I believe it will so sorry post-fix yeah I believe it will I believe it will I'm 90% certain we'd have to check open source SPF tools there's yes absolutely exactly here's how you set it up I will be honest you're going to spend the better part of an afternoon reading the documentation and so on and so forth however at least based on what I've shown you today you'll have an idea in your head as to what it's supposed to do I mean that's kind of half the challenge first place I am also going to point you to a really good URL I've got a little resources section at the end there's a guy that literally walks you through the whole steps of getting this thing set up and working and he's done a huge service to the community by doing that there's a pile of tools available for SPF you know this lib SPF 2 there's various packages you can do it in Python, Java even .NET if you wanted to there's various options there but interestingly enough Open DMARC which is the thing that will actually do the DMARC verification so if you want to protect your users from fake PayPal messages fake Chase messages fake Bank of America messages the way you do that is you're going to load this Open DMARC thing or you're going to run this on your mail server once again it's a mail filter and it actually has SPF checking built into it or you can set a flag and you can use the SPF library if you are already using that so it can either generate its own SPF results or use the results from the other package it does however rely on OpenDKIM so you will at a minimum need OpenDKIM to do the DKIM check alright those are the tools I'm going to give you guys links at the end so including that awesome link that's going to literally just walk you through here's how you set it up and it's going to hopefully make everybody's life a lot easier let's talk about adoption so based on sort of what the community DMARC.org has said these are the folks that are supporting it today and a rough count of how many mailboxes they have so Yahoo supports it and they white label the mail for AT&T Rogers and a few other places Google supports it including huh this is old there's about 6 million Google Apps domains now so all those folks that use Google for their email infrastructure that is included in that coverage Microsoft is hotmaillive.com Outlook.com MSN and they're just starting to get their feet wet with Office 365 so their business users are getting it as well AOL Comcast NetEase is a big Chinese ISP QQ in China by the way even bigger than NetEase just started sending reports a couple of months ago so that's an exciting one and then some other smaller outfits you know mail.ru, Yandex, etc based on that we estimate you know US coverage of a typical consumer list at about 85% worldwide a typical consumer list at about 60% there are also things for business users yes question? yes yes Apple is definitely working on getting iCloud and mac.com, me.com getting those squared away they sent us some test files several months back and we tested them and they were close and we sent them back with a few little hey you gotta adjust this and this isn't quite working right they've gone quiet on us but I know they are working on it I wish I could give you an exact date the last word I have for them is they are absolutely committed to it they already do it on the sending side of course and they are implementing it on the receiving side so little more patience and they will get there are you an Apple iCloud user? or just oh of course yep yeah exactly okay then on the business side so for open source yeah send mail post fix and qmail you can use the open demark with qmail you need this QSMTPD which is some other little shim layer that's gonna let it work with the the milter interface on the commercial side of course there's Google apps Cisco iron port proof point Symantec cloud message systems again they use that milter message systems I think it's changing their name to spark post if any of you are familiar with them Office 365 however Office 365 will the most they'll do is the quarantine if you have a reject policy they will back it off to quarantine they're not quite confident enough that there's not a lot of mailing list users and other edge cases and all 10 M demon not of I think Irving, Texas I'll fit down there if you guys know I think his name is Arvel great guy down there runs another mail server and they are supported as well alright however you know it's one thing to say that 80% 90% whatever of US consumers get their mail at a place that you know supports DMARC however your mileage may vary and what a lot of people want to do is they want to say hey what's my coverage if I implement this how many of my users will be protected so I'm not going to do it if half my users are protected but I'll do it if 90% of them are so we start with a representative list of email addresses you know for most companies you have your list that you email on a regular basis we can look up the MX host we can see if that host is known to support DMARC by basically looking at that list of you know Google Apps you know any Google Apps domain lists the same you know set of MX hosts same thing for Yahoo same thing for Outlook you know Outlook.com and then you tally everything up at the end and you'll have your percentage coverage so I wanted to do that I went through that exercise the problem is you know I don't have a consumer list my company we sell it to businesses so I use the Ashley Madison breach data so so for those of you don't know Ashley Madison is a site where married people can make new friends there were some some hacktivists who how shall we say did not really like their business model and they said we have your list of users and all their personal details including all their communications with other users their credit card details et cetera and we will release that unless you pay us X millions of dollars and shut your site down well avidlife media the parent company of Ashley Madison said no that's not going to happen and sure enough right on target I went the list so anyway in the brief amount of time it was available on a dark web I got my hands on it and I went and ran these and so I you know obviously a lot of the users were gmail.com addresses I did the look up and yep google.com da da da check Netflix.com yeah actually if there were any Netflix users on there they use Google apps yep check UCLA.edu no they run their own mail server what's that yeah there were yes there were military addresses on there there were quite a few from most of the major corporations that all of you know there were also quite a few that were clearly fake so there were a lot of profiles that actually used either avidlifemedia.com or AshleyMadison.com is the actual email address like a very inordinate number of them in other words they were obviously chumming the waters a little bit trying to make it look a little closer to 50-50 when in fact it was about 98 to 2% in reality anyway tallying it all up drumroll please amazingly Venezuela at least amongst Venezuelan AshleyMadison users has an incredible level of DMAR coverage almost all of them 96.9% get their mail somewhere that does DMAR checking the good old USA 90.7% now you know this might be slanted slightly because I suspect a lot of the users of AshleyMadison probably set up a free email account just for that purpose and most of the common free web mail providers obviously are supported anyway you can see the numbers there most notably though and this really confirmed what we already suspected to be true Germany and Japan had really the lowest coverage and there's a good reason for this Germany and Japan both you know they have their own technologies to do these things they have their own web mail providers in Germany it's united internet so it's web.de and GMX in Japan it's Yahoo Japan which is actually softbank and is not at all part of Yahoo despite the name as well as NTT so we are at Agari we are evangelizing this and we're hoping to get the German and Japanese markets up to 80 90% coverage so that their users can be protected as well yes exactly so spewing out diesel hydrocarbons so we are at the bottom of the hour I don't know if there's an immediate intervening session I'm basically done I did have a couple of additional fraud examples if anyone wants to stick around for a moment or two this one was kind of interesting I was on my way to a meeting with western union and we saw in our data this sort of interesting spike of mail coming out of a dot mail server an army dot mail server sending out a little odd why would the army be sending out western union email and then we had a look at the subject lines that the army was sending out oh did that not there we go and final notice 7600 US dollar daily payment for you obviously some kind of a spam campaign when we see this many it's not just A B testing they were doing A B C D E F testing of their spam campaign in any case B O happened to have a buddy who did the mail for the army called them up and said oh yeah thanks another one they said yeah everyone's why don't we get a contractor on our network who's got malware installed and it starts using our server to send out a bunch of spam anyway I feel safe at night okay we had Twitter had an amazing like that you would want to get somebody's Twitter credentials because I've got about 160 followers so I don't know what you would do with my credentials but apparently folks like the AP or CNN they have a lot of followers so I guess those accounts are kind of valuable anyway Twitter found that there were as many as 100 million phishing messages going out every single day so they we worked with them we got them locked down put them to reject and literally two days after they went to reject the criminals and flatlined ever since now that's kind of a cool result but the really cool result is that Twitter was able to take several dozen people that have been pulled from their day jobs to go and do account resets because of all the people that were getting phished and they were able to send them back to revenue generating jobs so you know to me that's a pretty big result that it actually did impact the bottom line alright last example or is the next session like immediately starting or no we got a couple okay so this last one actually was a friend of mine he had sold a camera on eBay and then he gets this email and you know I'll read it to you it's a principle of small it is my pleasure to inform you that I have carried out the payment of $4,800 via PayPal I just got a receipt from PayPal so if you haven't please check your spam now I am sure you must have been notified also I want to send it to my elder brother for work I asked you to send it to my eBay address but I'm currently on a business trip to Kansas here is the delivery address and of course it is in Nigeria yes okay anyway and then he says you know hey I really want you to ship this today I gave you an extra $100 to cover the quick shipping and all and you know please give graphic carefully anyway the key point is you can see what was happening here obviously this person thought they could just send a maybe it was going to end up in spam hence the check your spam folder thinking that this person would just read that and say okay I got my money I'll send it on like never mind they're not going to actually check the PayPal account anyway the point is is that you know a non-savvy user someone who you know hasn't had 5000 of these conversations with me might have just simply sent the money on had they gotten that email but because of DMARC that's it thank you all so much and where did that go there we go yeah first of all I'd like to get that 160 followers up to maybe 200 by the end of the year if possible so by all means feel free to follow me on Twitter it's jmwsecure I mostly tweet about cyber security stuff so you're not going to if you want my political leanings all business and then here's some very key links and again it's in the presentation that's going to be uploaded and available online great thank you all