 I'm just going to turn it off, so I don't have to let you in. Hi everyone, welcome back. You guys have made it to the last day of scale. Congratulations on making it to the first talk of the day. This is a security track, so until the expo floor closes at 2 o'clock today, you can still go see the community booth. We have members from OWASP, ISA, and other community groups in the security booth. Here for our first talk, we have Carlos Mesa with DNSSEC. Test test? Okay, awesome. Sorry, guys, give me just a second to figure out where my speaker notes went, and they're behind this window. We arranged on my screen. Yeah, sorry, things moved around on me, and I got confused. Yeah, okay. Okay, okay. Why did the speaker notes are giving me such a hard time over here? 30 more seconds and I'm running with it, however I got it. Yeah, nobody's here. Oh my goodness. Okay, everybody, sorry about that. Okay, so... All right, well, definitely thank you for all making it on the last day, and I'm sure a lot of you are a little exhausted after four days of open source goodness. But let's get going. So DNSSEC, solving a decade-old vulnerability. Well, first off, I'm Carlos. Carlos, I'm a sysad man. I learned about DNSSEC when I was volunteering for InteropNet, and I didn't realize how big of an issue DNS being insecure was. It was just like, oh, DNS, who cares? But I learned that it's actually a huge issue, and DNSSEC offers a pretty cool solution. And I just want to spread awareness and hope to encourage adoption. I'm going to go over the general idea of how DNSSEC works. I will cover some DNS basics to make sure everybody's on the same page, and I'll touch on the possibilities that DNS creates, and go over some of the software that's already taking advantage of DNSSEC. So, well, what's DNS and DNSSEC? DNS is often compared to as the phone book of the Internet, and DNSSEC is trying to be like the unspeakable caller ID. What does that really mean more technically? Well, DNS, the domain name system, is translating user-friendly names to addresses that the computer can't understand. This is essential to the functionality of the Internet. And DNSSEC is trying to ensure the accuracy of that information. So, the benefits of that would help protect users from forged DNS information, strengthen trust in the Internet, now that we can make sure that we're getting the correct information from DNS. And this will, like I said, create new possibilities that we will go over, and so, strengthening DNS will make other online activities like e-commerce, banking, even software distribution, all these different types of communication will be, in turn, more secure. So, yeah, so more specifically, it's mitigating some risks of, like, specifically man-in-the-middle attacks and cash poisonings, which are often used for cybercrimes. And as a brand, it's going to help protect your business and image and even possibly help build a reputation that you actually take security seriously and as well as your customer's information. So, exactly, what's the big deal with DNS? Firstly, like I said, DNS is old, stems from the beginning of the Internet, and it wasn't designed with any security in mind, so attackers can force this data. And so, like I said, DNS is telling the computer where to go, it's trying to link the name to an Internet address. So, if that's insecure, attackers can redirect people to fraudulent sites. Hopefully I can illustrate that a little bit better. So, here we have a sign telling you where Piggy Bank is, and that's like DNS telling your computer where to go. Well, someone can just come in and disrupt the data, insert fake data, and right now there's not really much to stop that interference. So, and then that can result you in being led somewhere you didn't want to go to some fraudulent and malicious site, which has the potential risk of, you know, phishing, identity theft, this can circumvent anti-spam measures, misinformation can get disseminated, wire tapping over VoIP can happen. These are just some examples of the risks that DNS currently poses to us. So, like I said, DNS is trying to ensure the credibility of DNS information to help not lead us astray to where we don't want to go, by protecting against spoofing, even corruption of DNS data while it's in transit, and it's aimed to maintain backwards compatibility while being able to add a layer of security. So, firstly, let's go over how DNS works. Some quick terminology is IP address, just the numbers for your computer. Up there I've illustrated what an IP address in version 4 looks like and IP address in version 6. The domain name is the human-readable version of that, using an alphanumeric name. There's an example of a domain name. SQDN is a fully qualified domain name, like the proper full name for it. And DNS is doing the translation between the alphanumeric name and the internet numbers. So, the anatomy of a fully qualified domain name, at the very right, it's read from right to left, so we got root. And then the next part is known as our top-level domain, our dot, in this case a dot org. And from there, a second-level domain example, and that's usually where we've traditionally been able to start purchasing those names. Today we can now start purchasing top-level domains, but traditionally we were purchasing the dot example from the second level. And then we have the subdomain beyond that, where we can start adding like www, or we can put mail.example.org from there. In a hierarchical view, maybe this may make a little more sense. From the top we have root, and then from there we can go to our top-level domains. For example, we're using org, then our second-level example, and then down to our subdomain to create www.example.org. A little more terminology, there's more components to that structure. There's zones, the subset of this hierarchical structure. And within those zones are resource records with the actual data to map the name to the address. I will illustrate anything in a second, but just to give you some examples of possible records, specifically what I've been talking about with the address is known as an A record, but there's also records to let you know how to find a mail server and other things like that. So, yes, within the dotted box is our A record for www. And outside of that is where we have our entire zone, which can encompass all our records for us of that zone. The name server is what maintains the responsibility of the zone. This is where the whole phone book metaphor comes from because from the name server is where we can look up this information. So, using the phone book, I want to find www.example.com. So my example is changed from .org.com. But my computer doesn't know where this is to start off with, but my computer does know where root is. So my computer goes root, like, yo, where's www.example.com? I don't know, but I know where the .com name servers are. Go check with them. So they give me the information, the addresses to go find .com, so I go ask .com the exact same question. Yo, where is www.example.com? I don't know, but example.com is my neighbor. I can give you his address. And I get the name server and the address for example.com. And I ask example.com's name server, where's www.example.com? And it's like, oh, I'm responsible for that zone. And here it's a record. And from there, my computer now has the address it needs in order to communicate with the www.example.com. So that whole chain is susceptible to, every part in that chain is susceptible to being exploited. If you ever research the DNS, you'll definitely come across such terms as DNS hijacking, DNS spoofing, cash poisoning. Very famously is the Kaminsky bug, one of the original, like, proof of concept type things to illustrate what is wrong with DNS. This is a very simplified, just to illustrate the example of what can happen with the DNS poisoning. Basically we have our rogue name server, DNS server, hanging out there just like flooding the requester with answers, hoping that something will get through. And there's definitely ways and techniques to increase the odds of getting the requester to accept an answer. So just some examples of some major breaches. And I made sure to want to choose different types of breaches. DNS changer was a big one where a piece of malware actually got on your machine and changed where you were requesting information from and then they were able to provide you with the answers they wanted so that they could monetize their traffic and actually made quite a bit of money. The Brazilian ISP, the entire country fell victim to DNS cash poisoning. And they think it was a possible inside job. Turkey actually intercepted routes to Google's public DNS and pretended that their DNS servers were Google's DNS public service. And there was no real way for users to detect it unless if they were able to spoof like Twitter or any other Facebook or any other sites they're going to or their bank sites, users wouldn't be able to probably notice unless if they did a good job of cloning these sites. But I've also been told that Carlos, we assume DNS is insecure. We already work in this environment. And I'm like, well, TLS isn't addressing this problem. Just to throw in some background, SSL slash TLS, I'll probably use the terms interchangeably. TLS is just the next iteration of SSL. But that's putting the S in HTTPS. That's where we get our little lock icon when we go to our bank. And this is even what I thought was securing me when I was browsing the internet. It's like, oh, I got my lock and I got all these little logos that tell me I'm safe. That is encrypting my connection to the server that I'm connected to. And the certificate that SSL provides doesn't match the domain name. But if DNS is compromised, then there's no way for me to really be sure that the endpoint I'm connected to is who I think I'm connecting to. But that connection is secure and encrypted. So that's an issue. Also, the certificates that we use to do SSL and TLS come from a third party known as a certificate authority. There are many problems with this system. First of all, if you look up the EFF Observatory Program, they go through and map out, there's a ton of certificates at your certificate authority that your computer is automatically trusting. And any of these guys can issue certificates for anybody on the internet. There's nobody stopping Verisign and DigiCert or anybody else to issue certificates for Google. So if any of them became compromised, they can start now generating keys for people they really weren't contracted to generate keys for. Also, they are able to delegate trust to other certificate authorities. So because you trust me, I'm going to tell you my friend's cool, so you're now going to trust him to whether he really keeps all his security practices on point or not. So that's a big issue. I have a ton of examples of this, but I thought these were some more notable ones. Here at one point, Digi Komodo and Digi Notar have been compromised, and it actually resulted in finding fraudulent certificates in the wild. Recently, Symantec had some issues because Google Real found that there was some unauthorized certificates being issued for their domain. And even most recently, and that was 2015, these are also 2015, Microsoft, Dell, and D-Link all-link private keys that were responsible for generating these keys, which means that now anybody is able to create a certificate that looks like it's from Microsoft, and your computer is like, oh, I trust Microsoft, and so now security is broken and a man in the middle can see exactly what's going on. So SSL TLS is not a complete solution. We need to secure DNS for a secure Internet. This poor little girl is about to have her identity completely stolen. So DNSseq, Tempor-Proofing DNS. So like I said, DNSseq is working towards protecting us from forged DNS data, and again, which will have the benefit of strengthening the credibility of DNS information and won't break the current system we have. So we can move forward with this solution and continue without breaking the Internet, basically. So what DNS promises is to authenticate that the DNS data is originating from a legitimate sender. It's from the proper source. And the data hasn't been modified in transit. And if data has not been published in DNS, we can even verify that that data is not there. It does this all through a chain of trusts, which is basically a series of checks and validations. We'll get more detail exactly how that works in just a second. But what DNSseq is not, though, it is not encryption. It's not going to provide confidentiality. It's not going to hide your DNS data. There are other options to work in conjunction with DNSseq, but DNSseq itself is not solving that problem. It's not going to guarantee availability. It's not going to address denial of service attacks. And if your name server is compromised, it's not going to address it. It's going to just protect the data in transit. It's going to protect that when you request it, it's coming from who has control of the name server. So what a fine zone might look like. This is once we've implemented DNSseq. So on the left we have our A record and a couple other common records you may see in a DNS zone. And to the right we have what it looks like inside. So each record set will have a signature which we can validate. And then we have some other records that I'm about to go into on how it makes this all work. So when we do now a DNSseq lookup, we get a little bit of extra information back from our example.com name server. It comes with, we get our answer with our A record to tell us where the address is for www.example.com, but we also get a signature for that record that we can check. Quickly, so these new records that we'll be going over is the DNS key. This is the public key for those signatures in that zone. The signature itself is known as an rsig record. And then there's another record I haven't talked about yet, the delegation signer. And the delegation signer is basically a hash that will go into the parent zone. So if you're a second level domain, the delegation signer will be in the first level domain. If you're a second level, it will be in the first level domain. And from there you can check the domain beneath you. And that's where the Chain of Trust starts to form. I do want to let you know these slides are available. I have a link at the end. And I'll be tweeting them out and everything as well. So definitely want to make sure everybody has all the information they need. So how does all this checking in Chain of Trust start to happen? Okay, so we got our A record back. And we got the signature for that record. Well, how do we check that? Well, now we need to go back to example.com. It's like, well, so what's the key for this? And it responds with the DNS key resource record. And now we can validate that this record checks out with the key that is inside example.com. But maybe we are being tricked into talking to someone that's not really example.com. So we can ask .com, hey, do you have the hash for example.com's public key? And then it replies with the DS record. And from there, it replies with the DS record and a signature of that record as well. So we can check that the hash we're receiving is in fact the correct hash. And so we hit to ask.com. So what's the key for this record? And it replies to us with its public key. We can check that we have the correct hash. Then use that hash to make sure that example.com's public key is legit. And that moves us up into the chain of trust. And similarly, we do the same thing at root. We ask root for .com's hash. And we ask root for its public key. We check the hash. And we check the record. Then we check the hash. And we've moved all the way creating a chain of trust to root. But whoa, how do we know we're even talking to the right root? Well, that's where a trust anchor comes in. And the trust anchor, it's going to complete the chain of trust because it's basically going to be a seed to allow us to validate the root server. So that seed comes from IANA. IANA is responsible for maintaining and publishing that, which you can get the seed from their sites. So now we have, so the trust anchor is essentially a DS record on local to us that we can now check root against. And we can validate root's key and complete the chain of trust. Now we have an entire circle of trust that we can validate for. And this opens up all sorts of new possibilities. Just for completeness, I did mention doing authenticated denial of existence. Basically making sure that we know when there is, that if we get that there's no data there, we can actually verify definitively that there's no data there. This helps clients distinguish between if there's DNS sec abilities at the name server or if it's an insecure, just a regular insecure DNS record. Being able to verify that helps prevent a man in the middle of tech. So it does that with another record called an NSEC record, NEC secure record. Basically it's a record to point at the next available domain. It does have the side effect of allowing people to walk and basically copy all your DNS info for some entities that is an issue. DNS is technically a database so there's even some legal issues for some entities. So there was a solution to come up with DNS sec 3. It's kind of a work around hashing these records to make it a lot more difficult to walk. Not impossible but difficult. But DNS sec 5 looks like it may be a solution but it hasn't been ratified and standardized quite yet. But we've created a chain of trust with cryptographic signatures which means we can verify the origin of our data, we can verify its integrity, and we can verify that we're not being man in the middle then told that and make sure that we can use DNS sec when available. So like I said, DNS is basically a database but now it's a database we can trust. And it's universally a distributed database on the Internet. And this database like I mentioned is basically a directory. And so now we have an authenticated directory of information we can use. So currently the other types of information we're putting into DNS, we have measures for fighting spam known as DKIM. I think there may have been a talk that maybe happened, I don't know if it's coming up that such on these issues but if DNS wasn't secure these measures can be circumvented. Also we've stuck other information in there just even the location of our mail server base and even we've put information for VoIP and Jabber in there. So all of this information could have been faked. Did you see that? I'm sorry? Yes, I will definitely make that edit. And we have other, there's been other efforts to put keys into DNS before. We have IPsec key which is for VPNs and T6 for even doing updates to our zone and even putting SSH fingerprints into DNS. But these were actually to present a greater security risk since the source of the information wasn't safe. So they came up with a, now we have DNSX so it's actually okay to put these records in there if you are using DNSX. But they want to take it a step further and created a domain DNS based authentication of name entities. It's kind of a funny acronym but basically a way of standardizing this process of how to put keys and to look up keys in records in DNS. So basically being able to standardize this allows us to do key discovery. So applications can just go look at your DNS to see what's available. If you can do any sort of encryption and what that key is to do that encryption. So this basically allows applications to automatically establish a secure communication channel. So just a quick review of this is what, even with our chain of trust, with our certificate authorities and current TLS infrastructure, it's still susceptible to a lot of attacks from the CAs itself and other men in the middle type of attacks. So with Dane, we are no longer dependent on certificate authorities because we can move this information into DNS which we've already established a trust with. The record for that is known as a TLSA record and that can store a certificate or key information. It doesn't have to replace your certificate authority. It can actually work in conjunction with your certificate authority by actually putting in a key to lock it down to a particular certificate authority or even to a specific certificate so at least the vector of attack isn't so wide. So only one certificate vendor can issue certificates for you and you can validate against this specific certificate. So mail uses is capable of using TLSSSL as well. But currently the way it works is it has to ask the other end if it's also capable of doing it because when email started mail servers weren't doing this. So depending on the answer from the other end point is whether or not it's going to use a secure communication. If the other end point says, no I can't do TLS then there's no secure connection. So someone can just sit in the middle, lie to both sides and watch the information pass through. So basically TLS on email is not very effective. But putting that being able to put that into our DNS record our mail server can check the DNS and see if whether or not for sure it can support a TLS certificate encrypted channel or not. And we can verify that communication. And we can verify even if it cannot support that communication because we have authenticated denial of existence. So we can make sure that it's not there. So I clicked the wrong way. So SSL and TLSA with unified communications. So there's other services that can start also taking advantage of certificates in DNS the same way mail could including VoIP services, Java services. And so any service that's capable of taking advantage of certificates are now able to do this through our DNS as well. But there's other things you can start sticking in our DNS. So certificates are just the beginning. These next slides are, everything is in various degrees of development. But these are all the type of things people are working on to leverage DNSSEC. And Dain, so Dain in the Web of Trust, we can start putting PGP keys into our DNS which again allows the automatic discovery. So before PGP has the issue of I have to do that initial exchange of keys. Well here we have a secure channel to exchange those keys with our DNSSEC. And this can even happen automatic. I don't even need to know in advance if I can talk to you securely because I can just look up at your record and my application just can start running with that. Same thing with S-Mine. The mail client can just go ahead and look at the inquiry for the appropriate signatures for in DNS. And we have IPSEC which is just formalizing the IPSEC key to the Dain standard. There's also cool things like people are working on off the record protocols so we can do safe key exchanges in DNS there. This is really cool that people are even working on a payment association identifier in DNS. And there's already a couple of Bitcoin wallets working on exchanging Bitcoin via authenticating identity via DNS. Armory and net key, not sure if I'm exactly pronouncing them right, but those are the two that I'm aware of that are already attempting to leverage this technology. So why not DNSSEC? Well, it does require mass adoption before you can really take advantage of it. But I feel we are basically there. Root has already been signed. That happened in 2010. And there's been a number of huge advancements. This is the current deployment of a country code top level domain like your .us, .nl, and the majority of them are already on board and fully in production with them. Here's the second level domain growth. So the top graph is the growth of .net and .com. .net has been for some reason a little slow, but .com has a clear trajectory upward of adoption. Below that we have, I got statistics for signed second level domains in the new TLDs, the new .pizza and everything else. And they over doubled across 2015. So adoption is well underway. Also just the queries for DNSSEC, it's about 25% of queries in the United States are being validated. And company adoption. Many ISPs have jumped on board, including Comcast, Time Warner, and others. CDNs like Cloudflare, Alchemy have jumped on board. And even some large companies have made sure to support it, including like Google's public DNS service. Microsoft's products can support DNSSEC and Cisco, just to name a few. So I've also been asked, well, why should I even trust Root? Well, Root has a lot of hands on it. So there's a lot of different stakeholders. No one has any like one master key. It requires a big collusion of people to fake Root. And even if they did, changes would be noticed. It wouldn't be something that would go undetected. And even during the whole signing process, it's public, it's streamed, it's audited, and it's under a lot of control. So changes would be detected. Some have said it's expensive to deploy. Well, there is naturally an increase in overhead. We are going to be checking records. So there would be an increase in CPU load, but with modern processors that shouldn't be an issue for most people. And we still, caching is still relevant for DNSSEC. There are some variables. I mean, if you're a top level domain, that may present some more issues that went through signing like millions of zones. Or if you're constantly needing to change and sign zones, there may be a little overhead there, but I feel the benefit far outweighs the cost. In this storage, we have not bigger zones. It's about the number I've seen is about, expect like a three times bigger zone. But in general, storage is relatively inexpensive, and zones are relatively small in general. Again, unless you're doing something like top level domain. So bigger zone files, we have more loaded in RAM. So we need about four times as much RAM. But again, at least commodity RAM tends to be relatively inexpensive. There is bandwidth issues. So we are making all these more queries about five times as much DNS traffic. But remember, DNS traffic is really tiny. So it's going to be insignificant compared to the rest of the traffic going over your network, YouTube, and everything else. Compatibility issues. So there are some hardware that may have problems with it. So this is an extension. DNSSEC is an extension of the DNS standard. But this extension was introduced in 1999. So most hardware should have caught up by now. Some vendors for some reason continue to assume that DNS is over UDP and only 512 bytes. But today, putting more information to DNS, it may work over TCP and be larger. But this has already been taken into account for decades ago. Like I said, there may be some non-compliant DNS implementations. And many times I know a lot of network admins have made these assumptions when they create their rules, so ACLs and whatnot may need to be adjusted accordingly. I've also been told, well, DNS is incomplete. There's still a hole in DNS. And there is what's known as a last mile or first hop issue, depending on your perspective of how it works. But basically what they're talking about is where the chain actually occurs at is who's making the validation. And that's actually traditionally like your ISPs DNS server that your workstation is looking at to get answers from. So there is a need to secure the connection between the requester and the validating DNS server and to make sure that the validating DNS server is legit. There's several solutions being worked out. Unfortunately, there's no standard solution yet at the moment. But some of the ones in work is get DNS. They work to create a standard API to help developers make DNS-sec-aware applications so the application can validate it for itself. The door solution is to include a validating resolver with the operating system itself. So the workstation is the one doing the checking. That's changed from when I made this slide. They decided they were going to do that by Fedora 24. They postponed it for more discussion. But that's the solution Fedora is looking at implementing. Microsoft's solution is to create an IPsec tunnel from every client to the DNS server. And then there's a DNS Crip which actually looks pretty cool. But it's not gained a lot of enough adoption yet. That's actually encrypting the DNS traffic to the DNS server. And what about alternatives? Well, these are the alternatives that I have found. I've actually found there's a very popular article that is against DNS-sec. And the author actually proposes to do nothing. I don't like to see that as a good answer. What's actually really awesome is DNS curve. It applies elliptic curve encryption to DNS traffic. But it's not exactly solving the same problem. It does look like a cool technology, but unfortunately it has not gained much traction. So it's not really usable. Without today is DNS-sec. It's been signed, the infrastructure is there. And encrypt all DNS traffic outside a curve whether it's VPN or anything else. So quick notes on DNS curve. Lacking adoption, OpenDNS and CryptoStorm are the only names that I could find that have adopted it. And even OpenDNS themselves say that DNS curve is not necessarily a competing solution. It doesn't stop them from also adopting DNS-sec. So how do we actually start taking advantage of this today? Well first, I'll push out these links. But you can even Google for testing DNS-sec validation. And you can check, a lot of websites will have a little piece of JavaScript that can reach out and tell whether or not you're able to validate against their domain. So once we verify that we can actually use DNS-sec in the network that we're in, there's a really cool plug-in for most web browsers, Firefox, Chrome, I think, I may be wrong, but I think you can even get this in on IE. That might be a little crazy. But yeah, so right in your web browser you get a new button to let you know if you've gotten your domain through DNS and that you can know for sure this is the correct. The correct answer from the DNS server and is also capable of checking for Dain and PLSA certificates. And to use it on a workstation. So I'm currently using DNS-sec trigger. So on my laptop I move from different environments. I don't know always what I'm going to be capable of in different environments and if there may be filtering DNS traffic at the endpoints and I'm stuck using whatever they have that may not support DNS-sec. This tool comes packaged with a resolver, like a very lightweight DNS server, puts it on my laptop and periodically when it queries it looks to see if it can validate or not. And it can detect when it changes networks. So when it changes to a network that I'm no longer able to use DNS-sec, it will pop up with a dialog box asking me if I want to continue insecurely or do I just want to stop altogether. The reason why this is important is because if I just enable DNS-sec on my machine and I go into an environment where I can't use it, well nothing validates anymore. Basically I break the Internet for myself. So this allows me to work around if I'm willing to, whatever I'm trying to do and I'm willing to get through, I want to get out to the Internet regardless. Well this solution allows me to toggle back and forth. And I do want to note it's at like 0.12 or 0.13 but I run it on my laptop and have had no issue with it. And I'm just running a variant of Ubuntu so I'm not doing anything special here. So also if you just want to have a, being able to reach out, let's still don't want to do all the validation yourself, you can reach out to resolvers that support DNS-sec like Google and I have here listed a few others that are publicly available DNS-sec validating servers. So you don't necessarily have to do this on the workstation if you have some infrastructure, even your home router can do this for you. So most machines we have DNS-mass that is actually doing the querying for DNS for us. Up here unfortunately it doesn't come, I don't believe in most cases it comes enabled all the way. It comes, it's called DNS-sec aware but you can still break it because it's not going to, if it gets an answer that says, if I don't support DNS, it's going to be a DNS-sec, it's going to just take the answer. This forces it to actually check the params zone to see if that answer is true, like no, this guy really doesn't do DNS-sec. That doesn't come on by default because that could potentially break internet for some people because that don't understand what's going on. Like I said, if I'm moving to a different environment and my computer is forcing this check and someone is like, if scale were so crazy to stop me from doing DNS-sec then basically my internet would be broken, a normal user may have it. So they do it too for simplicity but that's a bit of an issue with adoption. And well DDWRT uses that same tool and from what I've read online putting in the options for DNS-sec should enable it. I haven't verified that myself but you go to its web GUI where it lets you put in the options and you should have DNS-sec. A router you can have in your house that does work for sure is a PF Sense and it is literally a check box and can get the keys for you and everything it needs and is aware of what it needs to do. Or Open Sense, sorry, it's the fork of PF Sense. Okay, so what about if we have some more backend infrastructure? We are the guys with the mail servers and the other communication servers. Well, we do have a couple of communication server software that are ready to take advantage of DNS-sec and I really like that Jitsie is on board and that we have a SIP solution. There's also some Jabber servers and clients that are not only able to take advantage of DNS-sec but Dain as well. So these guys can automatically check for certificates and automatically establish a secure channel for you. A couple of mail servers that are already on board, some of the most popular mail servers also can support Dain to do that automatic check and automatic creation of a secure channel. So, well, if we are a domain owner, none of this works if we haven't implemented this in our domain. So you do got to make sure that your registrar can do this. IANA has mandated that all registrars are capable of this but it does vary in actual adoption of who is doing it. I'm just throwing up a few to illustrate a handful of the more popular guys that can do DNS-sec today. Basically that means they can take the DS key and pass that to the parent for you. So how about if we actually run a DNS server? Well, Bind 9 is like the standard reference implementation for DNS. And they can even help if you have to do dynamic signing of your keys because you're constantly making changes to DNS. They have a workflow and stuff ready to go. The DNS server of my choice is definitely NSD. It's simple and lightweight. So that's why. And even Windows 2012 supports DNS-sec. They don't support Dane but at least Windows chugging along. They will support Dane in 2016 if you work in a mixed environment that may be something to be aware of. So you are managing now keys with your DNS. And there's some tools that have come out to help automate these processes. Open DNS-sec is one of the nicer streamlined turnkey solutions to automate this process. Take whatever policies you need and apply them accordingly. DNS-sec tools have offered a suite of different tools to help you troubleshoot. They've even pushed out patches to applications. They've forked Firefox and they've made Bloodhound so they have their own web browser if you want a native DNS-sec support. So there's definitely tools available to help make this task simpler. So conclusion, we're almost over. So I'm here because I wanted to spread awareness and adoption. So I hope that I've encouraged enough that you guys want to spread awareness and adoption. Take this information back to your shop and start looking at what it will take to implement this. If that's not you, even at your home, make sure that can your ISP do DNS-sec? And why not start asking your bank, like are you looking at these technologies and start pushing the demand for adoption? And even just put it on your own workstation yourself. DNS-sec trigger is a very easy app get turnkey kind of solution. Also, if you need help with adoption, please don't hesitate to drop me a line and be more than willing to send reference information or however I may be able to assist. So DNS is fundamental to the Internet but it's unsafe. DNS-sec is the next evolutionary step in securing the Internet and DNS-sec is the foundation to more possibilities of doing secure communications and secure transactions over the Internet. Thank you very much. I do want to send a shout out to Josh Koo of the Internet Systems Contortion and DeepDive Networking. He's allowed me to continue to pick his brain about some of the finer details as I've learned and make sure things are accurate. And here that QR code will take you to my site which will have the link to my Google Docs version of the slides and I'll update it with SlideShare. And as soon as I can get the video link I will put everything on my website page here. I will also be tweeting it in a little bit at Digital Roots 00 in Roots. And I don't know if I have time for questions but I've got 5 minutes. Yes, that is correct. So Dane does provide the enforcement. Oh yes, I'm sorry. So he's asking about a downgrade attack against SMTP because by default you have to ask the other side whether or not we're going to do any sort of encrypted connection. And depending on the answer, if they answer no then it's going to be out in the open and someone could just be seeing the middle lying to both sides and watching everything. And the question was in regards to making sure and enforcing that that's not happening. And that is what Dane is capable of doing. So we are able to – an endpoint has to be capable of understanding Dane and knowing to take a look at the DNS query before starting that communication, the DNS record. And it can look to see if there is a certificate for the other endpoint and then it will know for sure. And this is after we've already established that we trust DNS. And if it's not there, because we have authenticated denial of existence we can know for sure whether or not they do. So we know they really don't. So we're stuck. But we can check and we trust DNS. We can see that their record is there so we will use it. And if for whatever reason something else is breaking our connection well then it's just not going to make the connection. Yes. So the question was if you're on a network on a land that is not providing you with a DNS server that can support DNS. Can you just point to Google DNS public servers? And the answer is yes. What I was trying to articulate though was if you're in an environment where they may be filtering and not allowing you to do external DNS then you're kind of stuck. Yes. The example was given about this is common at hotels where they want you going over their Wi-Fi and want to intercept and reroute your DNS information. And I actually have experience with this. Coffee bean is where I do basically my office. And so as I was getting prepared and learning more about DNS tech, getting deeper into it I actually got to watch them learn about DNS tech because some days it will work. Some days it wouldn't. Some days I can get traffic out to my own resolvers. Some days I couldn't. And I'll send a shout out to them since I use them freely as my office. They got it working and it's been consistent for over a month now. Any other questions? Yes. I am aware that there is many. The question has to do with that. Like many of the most popular sites are not signing their zone, their domain. Including Google who has enabled DNS tech on their DNS servers but yet I go to Google.com and they are not using DNS sign zones which is a bit disappointing from Google. But regardless we are still in that adoption phase. Adoption is still growing exponentially. And even for financial institutions that do not want to deal with it, that's fine. They will go away as the way of the dinosaurs. Their new companies will come in that are willing to adopt and that are concerned about their customer data. So while it's true, we need zones to be signed for there to be something to use. Things are definitely moving in that direction. I wouldn't argue not to use it because otherwise that goes back to the argument of like well then what's the alternative? Do nothing. So I think it's clear that the trend is towards DNS tech. And eventually everything else will get caught up. Any other questions? Oh sorry, we're out of time. Though I would love to continue any discussions. If people are interested I'll be outside. You can reach me at digitalroutes00. Thank you. Alright, can people hear me? Excellent. Yeah exactly. So I'm John Wilson. I'm with Agari. I'm going to actually do a proper intro in a moment but I want to thank all of you for coming here today. I gave a talk a couple of years ago in Miami and it was the dead of winter, one of those years when they were having the polar vortex. And most of the people at the conference were from the northeast. And my session was in the afternoon on one of the last days. And not surprisingly a lot of people I guess decided that the pool seemed a lot more interesting than my session. So I was in a room about five times the size of this room and there were about eight people in the room including one woman who was right dead center in the front. I mean literally would have been right there. Which would have been a great thing except she was there because she fell asleep in the previous session. About half way through she started snoring a little bit and then all of a sudden made a sort of a and woke up panic, grabbed her stuff and left. Like literally ran out. I have to say it was very awful. So thank you all for coming today. So, oh yeah exactly. And we'll get off to a slow enough start and get to keep the interesting stuff a little bit later on. Why do I have stuff overlapped? That is bizarro. Anyway, so a little bit about me. So I'm John Wolf and I am this slide, sorry does not look anything like that on my screen. Let's try that one more time. So you told me to use an open source format. Maybe I'll just go to the PDF and do it that way. There we are. That looks a lot better. So I was one of the founding team members at Agari. We've been around for about five years. I have helped a little bit in the fight against cybercrime and one thing that I did is I helped Microsoft and the FBI with the B-54 Citadel botnet takedown. If we get some time I can maybe share a little more about that at the end. I have worked at large and small companies. I started my career at Oracle, went on to SAP doing like application stuff. And then I branched off, went to make my internet millions and have been CTO of a number of startups. And currently at Agari I am our Field Chief Technology Officer. It's kind of like Santa has elves. I'm sort of the CTO's elf and I come out and speak at conferences and such. A little bit about Agari. Agari's mission in life is to eliminate email as a vector for cybercrime. And yeah, exactly. Hey, it's good to have a big goal, right? And if we even get close to that goal I think we'll actually be pretty darn happy. We've got quite a few clients that you've probably all heard of because we help people stop fishing of their brand. Well, obviously the brands that get fished tend to be companies that you have heard of. So four of the top five U.S. banks, three of the top five banks in the U.K. and four of the top five social networks use Agari to protect their email stream between them and their customers. Sorry, today I'm going to cover, first of all, why do cybercriminals love email? Probably everyone in this room already knows the answer to that. I'll tell you about a few previous attempts to solve the whole email spoofing, email fraud problem. And then we'll talk about what I believe is the current best solution which is a combination of SPF, DKIM, and DMARC. We're going to get into the technical details of those. And then I'm going to show you some open source tools that you can use if you guys want to go and try to implement these things yourself. You don't have to pay money to a company like Agari. There are tools out there you can roll your own basically and not spend a dime. We'll talk a little bit about some edge cases and a few things that DMARC doesn't necessarily play well with and some potential solutions there. And then we're going to do something fun. We're going to take a data breach, specifically the most salacious data breach of last year, the Ashley Madison data breach. And we're going to use that data to estimate DMARC coverage by country. So that should be a little bit interesting. And then finally, if we have some time at the end, I've got some additional fraud examples that I'll share because some of them are a little bit entertaining. So the whole reason we're here goes back to 1982. So SMTP was sort of first rolled out in 1982. And the inventors, they did such a great job. We're still using the protocol today, but they forgot one very important thing and that was authentication. Basically, I can connect to a mail server and say, hey, I am in this case security at WellsFargo.com. And I have an urgent message for you. And you can imagine if I was actually a cyber criminal, I would lead people off to a fake Wells Fargo login page, capture their username and password, and then go in there and move some money. So what do criminals do with this flaw? Well, here's an example, a somewhat more benign example unless you actually go and try to buy these products. Basically, this is what I call pharma spam. So I'm sure everyone in this room has received at least one offering to sell you, usually Viagra, Cialis, various other medications without a prescription. And the problem is if you actually do go and click that link and go to this website, this is not really the Canadian healthcare mall. These are usually operated out of countries that are, how shall we say, don't have extradition treaties with the U.S. The pills they will sell you may or may not have the active ingredient in them. I guarantee you they're not genuine Pfizer pills. They are knockoffs. So good luck to you if you go and order some of those. You might get very ill from them. Here's another example, phishing, extremely common. I had a really good PayPal example, and my guy over at PayPal said, please don't use our example. And I'm like, really? Come on. Everybody knows you guys get phished off. But he's like, no, we don't want to highlight that. So okay, fine. So I used an Apple example. And this is actually a pretty poor example. I've seen much better ones. But here you can see, well, it's a last notice here. We're going to suspend your account if you don't click this link and give us your credentials, because somehow, of course, giving my username and password is going to let them keep my account open. If you click the link, you end up over at this lovely little website, which looks pretty darn genuine, except to the technology people in the room who might actually check that address bar and note that not only is the little green padlock missing, it doesn't bear any resemblance shape or form to Apple. And then finally, again, a very poorly done example, because all my really good, well-done examples, the folks didn't want me to show them to you today. But this is an example of malware. Again, this one's horrible. It just says invoice, and then the attachment name is called augmentingmommy.zip. I think if they'd spent maybe just five more minutes fine-tuning this, they would have had a little bit better response rate. But if we take that over to VirusTotal, we find out that, yes indeed, 43 out of 55 of the scanners they run over there determined that it was some sort of nasty stuff, basically a keylogger. All right, so that's what the bad guys are out doing. I didn't even talk about the most common thing we're seeing lately, which is the business email compromise. This is when they send your finance team and email supposedly from the CEO saying, hey, I got to wire $3 million to China, and we have to do it immediately, and I want you to keep it hush-hush, and you might all chuckle. Mattel lost $3.2 million. Ubiquiti Networks lost $46 million. They later got about six of it back. And there was one other big one, I forget the name of it, that made all the headlines that lost like another $10 or $15 million. So, yeah, it's out there. In fact, was it you? No, yeah. Everybody gets these emails, and sadly once in a while somebody falls for them, and it's horrible because the money's wired away, and typically the banks aren't going to reimburse you when there's a certain degree of carelessness and stupidity involved. So, previous attempts to fix this. So, one of the first things that people tried was this idea of sender policy framework. In concept, very, very simple and practice fraught with issues, the idea is, there's my mail server. Its IP address is 1.2.3.4. That's where all my mail comes from. I'm going to publish a record in my DNS, and I'm going to tell everybody, if you get a message from my domain, it better come from that IP address. There's problems with that that we'll get into in a moment. Not to be outdone, Microsoft said, hey, if SPF is cool, we're going to come up with this thing called sender ID. It's going to be just like SPF only better. It did finally die a rather inglorious death. They finally have agreed to sort of stop using sender ID, but basically sender ID offered the additional capability of not just authenticating the RFC 821 envelope, but actually could authenticate the RFC 822 from header, which seems like a great idea. The problem is nobody wanted to go and do that, and the reason is you could not render a decision during the SMTP transaction before you accepted the entire message. So that was really the benefit of SPF was the fact that you could verify it right at the beginnings of that SMTP conversation. You didn't have to get down into the full message content. Then there was domain keys. This was something that Yahoo did, and it was the idea of a digital signature applied to a message. And then Cisco sort of joined forces with them and fine tuned it and came up with domain keys identified mail. Pretty much the same thing, just a few minor differences that made it slightly more secure, and DKIM is by far the more widely accepted and adopted standard today. Then we had author domain signing practices, ADSP. The idea here was how is a receiver supposed to know if you sign your mail or not, right? Because DKIM is optional. You don't have to do it. And so the idea behind ADSP was if I sign all my mail, I'll publish a little record in DNS that says, I sign all my mail. So if you get a message that's not signed, you probably ought to just drop it because it's not really from me. In practice, nobody really implemented it. It was a little baby step that nobody really embraced. This next one is near and dear to me. The idea of visual trust indicators. So my last company before Agari was a company called Brandmail Solutions. And we were in competition with Iconics and Goodmail. Anybody ever hear of any of those companies? Yeah, so the idea behind all of them was that you would have some kind of email authentication and then if the message checked out, you would actually flag it in some visual way as being authentic. And here's an example from the old Brandmail home page showing you a typical mailbox where everything looks the same. And then you have a branded mailbox where this particular message from Quinn Direct is clearly identified as being authentic. And then if you hover over that little envelope, you would get some additional detail. Brandmail and Iconics did it based on open standards. Goodmail did it based on a proprietary technology. Basically, all the companies are gone now except Iconics is amazingly still in business. In fact, we sold our IP, our patents, et cetera, to Iconics a couple of years ago from Brandmail. And then finally, we had the famous PayPal experiment of 2007. PayPal was a pretty early adopter of domain keys and SPF. And as they looked at their list of users, the vast majority of them got their mail at Yahoo, at least 50% of them. So they figured, well, a lot of this phishing problem could go away if we just got Yahoo to check the SPF and domain keys. And if they didn't pass, block the message. Don't even stick it in a spam folder. Just kill it outright. And maybe provide a little information back to PayPal. So just in case something goes wrong, PayPal is aware that something's gone wrong. And amazingly, it worked. When they put this in, it stopped the ability of criminals to spoof PayPal.com to Yahoo inboxes overnight. But there's a problem. And that is that it's great. It sells it for PayPal. And it sells it for specifically PayPal users at Yahoo. It doesn't do anything for the Gmail users or the AOL users who also use PayPal. And it certainly doesn't do anything for JPMorgan Chaser Bank of America users who get their mail at Yahoo. And so right about that time, there were some rumblings. And by the time we got to about 2009, the rumblings got pretty loud. And we started actually discussing how could we take that experiment and make it scale to the Internet. And what we came up with is DMark. I'll give it one chat here. Domain-based message authentication, reporting, and conformance. You should have seen me the first week that came out. I would bungle it every single time. I've gotten a little better with it. Anyway, so we're going to talk about DMark a little bit today because in my view that is the one that's got the best legs at least at the moment has the greatest chance of solving this problem. In case any of you are interested in the standards, there are some of the standards and of course the slides are available online I believe later today or whenever. So you will have access to the list of the standards if you want to really give yourself something to help fall asleep by. In terms of how this all works, you've got SMTP at the bottom, right? That's moving the mail around the Internet, no authentication. Then you've got SPS, that's like basically your list of IP addresses that say this is where my mail comes from. And then you've got DKIM, which is a digital signature applied by the mail server that can be validated at the receiving end. Big problems though with SPS and DKIM. The biggest problem is most people don't know where all their mail comes from. You know when I said like, oh my mail comes from that server right there. That's what all the techies think. Ask your marketing department where your mail comes from. Probably comes from a place like Exact Target or Constant Contact, Vertical Response or any one of about a thousand other companies that provide marketing services. Ask your HR team how they do recruiting. Oh, I bet they have some company out there. Maybe Teleo, which is now owned by Oracle. Sending out mail saying, hey, come work at company X. We've got great jobs. Simply put, if you don't know what that email landscape looks like, you're going to bungle your SPF, so that's not going to work. And of course if you don't go and give these guys a key to sign with and set the public part in your DNS, you're not going to be able to use DKIM either with those folks. So the big challenge here was how do you figure out where all your mail is coming from so that you get your SPF and DKIM right? As long as SPF and DKIM are working, this system is brilliant. But if they're not working, your real mail is going to go missing. So what we came up with was DMARC. And the idea was, first and foremost, give the domain owner a way to figure out where all their mail is coming from. Tell them where it's passing SPF, where it's failing, where the stuff is being deconsigned, where it's not. And also, as a side effect, figure out where all the criminal email is coming from. And then once somebody uses that data and cleans up their SPF and DKIM and gets it working really well, give them a way to do like PayPal did and said, hey, from now on I don't want you accepting mail that fails the standards. And so DMARC was born basically January of 2012 with a rather small announcement from Google, Yahoo, AOL and Microsoft who all said we support this standard. And actually they had been supporting it for several weeks prior to the announcement. And we had already gotten most of the kinks worked out. So let's go a step deeper here. So first of all, hopefully this is all review, but this is SMTP. Basically you tell them that to port 25 on the recipient's mail server, which by the way you can look up by looking up the MX host in DNF. And essentially when you connect the server says something like, here's my name and I'm ready. And then as a client you say hello and you give your name. And the server says something along the lines of 250, codes that start with a 2 is like a success code. 4s are sort of temporary errors. 3s are kind of just informational and then 5s are actually errors. So it says, okay, fine. And then I say, hey, I've got a message from test at test.com. And then they're typically going to say, hey, the sender's okay. And then I'm going to say, and I have a message for in this case my personal Yahoo account. Maybe I shouldn't have shown that to this room. Please don't start spamming me. Anyway, and then recipient's fine. And then we say data. And then typically they say go ahead and then is the message. Now notice down here we have a from header. I'm going to get into this a little more in a second. But this is the difference between the RFC 821 envelope sender and the RFC 822 from header. Finally you end with a dot on the line by itself. They say okay, or in some cases not as you will see and then you say quit. And now I have handed the message off to Yahoo and presumably Yahoo is going to go deliver it. All right, let's talk a little bit about identity. So here is sort of just the client side of that conversation. We have several different identities in here. We have the hello domain. We have the envelope domain. We have the signing domain if there was a decim signature. And finally we have the from header domain. Now in some cases those are all going to be the same and that's going to be fantastic. Everything is very, very clear. However in some cases they are not the same. And we run into a situation where we have what's known as an identifier misalignment. And that's something we'll need to address with the DMARC standard. All right, here's a couple of SPF records. I love Apple's SPF records. The advantage is of having a Class A network with all 16 million or however many maybe even more than that, a lot of IP addresses all starting with the Octet 17. And interestingly enough they are telling the world that's the only place we send mail from. So apparently Apple does not use any, at least for Apple.com does not use any third-party services. PayPal has a much more complicated setup. They actually have all these little mini-includes and those includes have other includes and eventually they resolve down to lists of IP addresses. All right, let's look at some typical mail flows. Okay, in the simplest world, you know, you've got your PostFix server and you send to a Gmail user. That green host is going to be in your SPF record. Maybe you've got Exchange on-premise and then you're sending to Office 365 and they're actually sending it out to the final recipient. In this case you would list the Office 365 server in your SPF record because it's the last pop. That's what the Internet sees as your mail coming from. And then what's that? Yes, exactly. And then finally you have other situations, what I call relays or forwarders. Your send mail server sends to some address at UCLA, but it turns out that person already graduated and they have a little pointer set up at their email address and it's going to bounce it over to say Yahoo. Well unfortunately, you know, whoever the sender of this is is going to list that send mail server in their SPF record. They don't know anything about that UCLA server, so when Yahoo sees the message that message is not going to pass SPF because we've introduced an extra hop. This is why we use two standards, why we don't just rely on SPF. So SPF breaks due to relaying or auto-forwarding. Mailing lists are a huge problem for SPF and that's going to be one of my edge cases we talk about in a moment. It authenticates the envelope domain which is the mail from. You and I don't typically see that when we get an email message. We're looking at the from header. So authenticating something I never see isn't probably the wisest thing in the world. SPF has this interesting little quirk. It was designed to prevent SPF as being used to commit a DDoS attack and that is that you have to have a maximum of 10 DNS lookups required to fully parse your record. So as you have all these different includes and they kind of keep chaining or whatever, the idea is that after 10 lookups the receiver is just going to throw its hands up in there and say give up. Yes, it doesn't include the root one. So technically you get 11 total. Basically A tags, MX tags, PTR tags, the other one includes and redirects each eat one. An IPv4 or an IPv6 tag does not eat any lookup. And there are a couple of tools out there that you can use online that will actually tell you. They'll analyze the record and say, oh, your record has 9 lookups. Your record has 11 lookups. And then finally, a lot of people forget about a special case. You guys ever go on vacation? You ever set up one of those vacation messages? I'm out of the office until January 12th or until January 25th. I'm at the Scale Linux conference. Those are out of office messages. Another similar situation. I'm sure everyone here has gotten one. You fat finger an email address and you get a bounce back, a non-delivery report. Well, it turns out the envelope domain in both of those situations is empty. It's null. So what the heck are you going to do the SPF check against? Well, the specification says we're going to check the hello name in those instances. Well, you got to make sure that you put an SPF record in for those hello names for each mail server that you have or non-delivery reports and out of office messages are not going to pass. And it looks something like this. You put one of these in for each of your MPA, which means this host name and a minus all. So in other words, any IP address that maps to that name will be acceptable. Alright, switching gears a little bit. Domain keys identified mail. DKIM. This is the other standard. Now, this one's kind of neat because it will survive that forwarding typically. As long as nobody tampers with the message in transit, we don't care if it takes the short route or goes through a thousand servers on the way to its destination. So here's an example. This is actually added as a header in the email message. And there's some interesting pieces here. If we look here, this is the key lookup, if you will. So Yahoo would look up this key if it received a message like the one up there with that header in it to validate the signature. And so the way they get the key is they take the selector or the S equals tag and then they take the D equals or the signing domain. Then they stick them together with underscore domain key in between and they do a DNS text lookup. And the result, they get this key, which is in Bay 64 format. That key can then be used to validate the signature. There's some other interesting parts to signature. If you look here, I can show you. Here you can see where it says H equals date, colon from, colon to, colon subject, colon, etc. etc. Those are the headers that are signed. Little helpful hint, do not sign headers that are likely to change. Signing the received header may be not the best idea in the world because those kind of typically get changed and added along the way. But that's a pretty good set right there. You also don't want to not sign a header that's fairly important. If you don't sign the subject header, somebody can take that message, change the subject, and still have the signature validate. So you have to exercise a little caution there when you're deciding what to sign, what not to sign. Alright, there are a few issues with DKIM. If DKIM was perfect, we would just say, well, don't use SPF, just use DKIM, right? First of all, there's such a thing as what I would call benign message modifications. Anybody ever run a mail server that, you know, like to take the from and the to addresses and put them in a very specific format? You know, you can have the friendly part in double quotes, or you can just have it without the double quotes, and then you put the real address and angle brackets. There's a few different ways you can do it. Well, some mail servers have a certain way they prefer to see it, and they will rewrite those addresses to fit what they believe is their preferred pattern. Well, that's basically a benign modification, but it is a modification, nonetheless, that will break the DKIM signature. Most notably, we discovered Cornell University does this with all their alumni. So if you are a Cornell alum, and you are finding you are not getting your Facebook, LinkedIn, or Twitter notifications, this is probably why, because they like to rewrite that. Again, mailing lists typically break DKIM. Now, they don't have to, but the main reason they typically break is because mailing lists like to mark up the subject line. They usually like to put the list name and angle brackets at the beginning, or square brackets at the beginning is a fairly common thing. They also often like to put a little footer at the bottom. Hey, this was sent to you by the such-and-such list, unsubscribe, click here, that kind of thing. Those are message modifications. So if you send a signed message to a, you know, say the Southern California Linux distribution list, when that gets re-mailed, that signature is probably not going to be valid anymore. Also, DKIM validates that signing identity. It does not necessarily validate the from header. I could sign your message. It doesn't mean your message is any more or less authentic, right? All we can say is that we know I signed it. So the best situation is, of course, when the signer and the from header are one and the same, because then we know that, you know, the signer actually did offer the message as well. DKIM does have the potential for a replay attack. So now there are things to mitigate this. There's an expiration. You can put a special tag in the DKIM signatures. You can say, hey, the signature expires in 15, 20, 30 minutes, whatever, some reasonably short period of time. Now, the only thing somebody could do if they replayed this would be to just basically take a signed message that was sent by a legitimate entity and then send it to, you know, a billion of their closest friends. It might create some headache for the person who originally wrote the message because people would feel they were getting spammed. But it's not like you can go and change the message and make it more useful to the attacker. It just means that, you know, you would have to replay that precise identical message. And then finally, you know, with the advent of, you know, cloud computing and more importantly, cheap cloud computing, it turns out you can kind of basically, you can factor these keys. It turns out if you have a little bit of money and a little bit of patience, and also DKIM keys are typically quite long-lived. They, you know, you publish them in your DNS. I've seen companies that are still using the same keys, you know, after five years. I recommend that, you know, six months to a year, you might want to consider rotating them. In any case, Google will not accept a key smaller than 1,024 bits. So if you sign something with a smaller key, send that message to Google. They will just say, oh, it was signed with too small of a key. We're going to pretend it wasn't signed. So that's their solution to it. My recommendation, at least use 1024 if possible use 2048, you know, 2,048 bits in your key. All right, so, on to the final standard here, DMARC. So once again, we're going to overload that DNS system, make it really work, earn its keep here. They're stored as DNS text resources, and they're stored at underscore DMARC, so the special location. And they have a very special format. I've just looked the one up for Twitter, and it basically says they're using version one of the specification. Their policy is they would like receivers and reject messages which cannot be proven to originate from Twitter. They would like aggregate and forensic data sent to those two addresses you see there, which happen to be my company, because like I said, we process the DMARC data for many companies that, you know, want to use, get the business value out of it, but don't necessarily want to spend many, many hours coding and, you know, putting together open source solutions and such. And then finally, there's this option, the FO tag, which is the failure options tag. The domain owner would like forensic reports for any messages that have failed, either SPF or DECAM. In other words, one or more protocols. You can also send that to FO equals zero, meaning you want only the things that have failed. Yeah, I believe that means that the only the things that have passed no standards. You can also say S or D, saying I only want to know about the DECAM failures, I only want to know about the SPF failures. Little hint here, notice the format of the RUA and the RUF tags. There's this little mail to colon in front of the email address. The idea there was that eventually there might be other mechanisms to transmit the data. Today it's sent via email. Don't forget the mail to. Your record probably won't work if you forget that and it's a very common error. All right, let's talk a little bit about subdomains, right? Not everybody sends from twitter.com. Sometimes they send from news.twitter.com or alerts.twitter.com. Subdomaining can be a really neat thing. It helps you divide your mail streams out. It helps you get around things like SPF record lookup limits. Suppose you have six different vendors that are doing various things on behalf of your company. You give them each a different subdomain and then their SPF record only has to include their hosts. You're not going to run into that lookup limit. But as far as DMARC is concerned, DMARC records are inherited by subdomains automatically. This is fairly important to note. If you put your top level at reject and you didn't realize that there were a bunch of subdomains in use, you might have just stopped their mail flowing. Not a good thing. However, there is an SP tag that you can specify which will say, this is my top level policy but for every subdomain I want the default to be this other policy. You can use the SP tag. Subdomains can have their own records which override the higher level record. Here I've done a couple of examples. If we had DMARC.example.com at reject with an SPF quarantine, food.example.com with a policy of none which basically means monitor only don't take any special action and a subdomain policy of reject and finally bob.example.com with a policy of reject. We get the following little matrix. So top level obviously is reject. bar.example.com is quarantine because there's no specific record overriding it and the top level record says SP equals quarantine. So that would apply. Similarly food.example.com has P equals none. It has its own record it overrides. Billy.bob.example.com is going to be reject because it's going to take the bob.example.com. There's no SP tag so it's going to use the P tag. And then finally john.food.example.com will be reject because of the SP equals reject on the record for food.example.com. Yes. No, by default. So at the top level, if your top level is reject then all of those will be reject by default. If you want the top level say to be monitor but everything else to be reject you can say P equals none, SP equals reject and that means all subdomains are going to be reject. So yeah, infinite flexibility basically. All right, one other little got you with DMARC let's suppose you want to send your data to another domain. What a great way to spam somebody, right? Publish a DMARC record and I take your email address and I say hey send all my DMARC data to him. Probably he's going to get annoyed pretty quickly and the folks that thought of this thought of this particular challenge and so if you want to send your DMARC data you can send it to your own domain, no problem whatsoever. However, if you wanted to send it to somebody else's domain there's one extra little step. So you need to get the domain that's going to receive your DMARC data to publish a very special DNF text record where it's basically your domain dot underscore report dot underscore DMARC dot their domain and then all they do is they say V equals DMARC 1. If that record is present then Google Yahoo et cetera will say yeah okay they're willing to accept data for domain X. I'll give you a little hint. Google by the way will not you know how are you going to send your DMARC data to your Gmail account? Well you're not unless you're Google because you could send it for Gmail dot com. However, Yahoo has a nice little wild card in there and they will happily accept DMARC data for anybody so if you've got nowhere to send your DMARC data set up a free account on Yahoo and send it there. I just did a dig there for any random domain dot org dot underscore report dot dot dot dot et cetera and lo and behold they do in fact return the all important V equals DMARC 1. Alright so let's suppose you publish a DMARC record. What's going to happen? Well at the address you specified in your RUA tag you're going to start getting some data. It's typically going to happen, it usually happens about 5.36pm because that's when UTC time zone switches over to the next day. All of these guys run their map reduce or their Hadoop or whatever crunch through piles of data. I can't even imagine for somebody like a Google or a Yahoo to basically go look at the last 24 hours of data that your mail has come in and prepare all of these reports but what they're going to do is they're going to email you a zipped XML file and that actually a GZipped XML file and that XML file is going to look something like this. There's going to be a header part and a header is going to basically have certain tags. This particular thing is being reported by Yahoo. If we wanted to ask somebody about this report well we'd ask postmaster at DMARC.yahoo.com and they've given us a report ID that we could presumably refer to. Those are supposed to be unique though in practice I've occasionally seen collisions there as well. Then we have the date range that this refers to. Those are just UNIX timestamps and the astute among you will realize this example is a little bit old. I probably pulled it several years ago but it basically refers to a 24 hour period. It's very typical that people report on a 24 hour boundary. It's specifically for the domain alertsp.chase.com. It's some kind of alerting that JPMorgan Chase does. I forgot to talk about alignment options. Remember we've got all those different identities. We've got SPF checking the envelope identity. We've got decom checking the signing identity and then DMARC's all focused on the from identity. Well the DMARC standard says if I'm going to use the decom signature then the signing identity must be aligned with the from header. Same thing with SPF. If I'm going to do that the envelope has to be aligned with the from header. Now there are two flavors of alignment. Strict and relaxed. Strict has to be an exact match. If it's food.bar.com here it's got to be food.bar.com here. If it's relaxed you can have parent child or even sibling domain. So you could have bar1.food.com and you could have bar2.food.com. They would both roll up to food.com. And yes they do take into account things like .co.uk so they know that that's actually sort of one step there. So in any case this record is saying that Chase's DMARC record for alertsp.chase.com specifies a decom alignment of relaxed and an SPF alignment of relaxed. If that was an S it would be a strict alignment. This literally comes straight out of the original policy. So in other words if you queried for the DMARC record basically Yahoo just playing back saying this is the DMARC record we found for you and the policy is to reject mail. You also see the PCT percent they would like that applied to 100% of the mail. Now you might be like why on earth would you not want to apply it to 100% of the mail? It turns out that when you're first turning this stuff on some people especially at large corporations get a little nervous and they want to ramp it up. So they may say you know what start off just block 1% of the failures choose them at random and then start blocking 2% of the failures and eventually get it up to 100%. I personally think if the data is telling you it's clean it's clean go to 100% but you know I don't work for a bank or any other highly regulated industry that would you know fire me instantly for doing something like that. So after this header you then are going to get a whole pile of these individual records got to love XML the way they can take a tiny little bit of data and make it take up this much space. But anyway thank goodness now you know why they're GZipped. This particular record says that source IP 10610 149.115 sent one message it passed DKIM it failed SPF and no special action was taken against it because of course it passed DKIM. It's an either or by the way you don't have to pass SPF and DKIM you have to pass one or the other and be aligned. Then they're telling us the from header was alertsp.chase.com the DKIM domain was alertsp.chase.com and the result was a pass the SPF unfortunately failed. Now does anybody what can you tell me about this? This particular record. DKIM passed SPF failed. Remember that diagram I showed you where we go through UCLA on our way to our final destination. This is one of those situations. DKIM survived but that host 10610 149.115 does not belong to JPMorgan Chase. It probably belongs to some university some mailing list something some relay. I've seen these files get as big as 600 megabytes after they're decompressed so you know if there's a they get big. Sorry wait did I say yeah 600 megabytes almost a gig and yeah just all it takes is you know one good botnet attack and can blow your file up in a hurry. Now there's another type of DMARC data not everybody participates specifically Microsoft participates Yahoo sort of participates. Yahoo will give you forensic data if you work with Agari or one of one other competitor where we have a private relationship with them and the main reason they won't send forensic data just anyone they're afraid of in the event of an attack your mail server will be overrun with them sending forensics but forensics are basically copies of the messages that have failed one or more of the protocols. Here's an example of one they are sent in ARF Abuse Reporting Format if any of you are familiar with that it's basically a way you encapsulate an ARF you know an ARF C822 message inside of you know a MIME type that's carried along. So this is the header basically the thing giving us a little bit of information and then there's the actual attachment which my Mac mail client here actually displayed inline but that actually is a separate you know ARF C822 MIME type. We're going to get into that a little more in a minute because that really lets us do some nifty things to sort of take a stab back at the criminals. So this is what happens every single day now at Yahoo Gmail, Comcast, people who are running Ironport, Symantec and a bunch of other places I'll walk you through this. So a message comes in out of the internet and the real question they're asking is you know do SPF or DKIM prove the message came from the domain on the from header? Yes or no? Then is the content oh hey okay so assuming it passes notice we are not skipping the spam check just because something can be proven to come from who it claims to come from it could still be spam so yes you could still end up in the spam folder. However if SPF and DKIM don't prove the message was sent by that entity we go to the second side over here we look up the DMARC policy and if it is none we just send it over here running through the spam filter business as usual we are going to report about it of course if it's quarantine we send it to the spam folder and if it is reject we send it to the trash. There also are the concept there's the concept of local policy overrides that is something where the receiver can at their option say we know better and we're going to ignore the policy we cringe when we see that at Agari because we watch them let fraud in and we're like why did you do that to your users? The bank tried to tell you and yet you let it through and they're doing it less and less but there is the option to do a local policy override. So let's look at this in action. Here's a nice little session of me telemetting to port 25 over to Yahoo this time instead of pretending to be Wells Fargo who is just starting to do DMARC I pretended to be security at chase.com who have a very mature DMARC implementation and you'll notice instead of that nice 250 OK at the bottom we get an error code 554 message not accepted for policy reasons in other words it's actually rejected in line during the FNTP conversation. In other words you get a bounce back if you're a human user and something went wrong you're going to get a bounce back and it's going to tell you that it failed because of DMARC and then you're going to go talk to your email admin and you're going to say okay why did my message fail DMARC. Alright a couple of limitations. So DMARC does not do anything to solve the friendly from problem. What is the friendly from problem? Well as you all know there's the email address and then there's the display name and these mail clients they try to be so nice to dumb it down for all of us and instead they've made us all a lot less safe. Here's an example you know chase card 182 at hotmail.com I bet you it might even pass authentication. So one thing that might help here is the idea of inbox differentiation. So my now defunct company brand mail is no longer around to do this however Google was running a little experiment we call the Google gold key. It's in a Google lab right now where they'll actually put a little key icon that they've checked the authentication on it. So this maybe might be a way forward not a guarantee only works at Google and only if you have the lab but main point being you still want to check that email address don't just look and you know if it's your CEO's name or your bank's name don't assume that the email address is really what it says it is. Also you know if you're running your own mail server you know like Hillary Clinton tried to do there for a little while while the State Department you know you're not going to be protected by this unless you go and install some of this open-source stuff we're going to talk about in a moment. However more and more places are implementing DMARC so the pool is shrinking and there's also a certain sense of herd immunity. What we've noted is that when a big consumer domain publishes a reject policy we usually see the criminals kind of go away and move on and pick a different brand. You know if you can only hit 10% of the bank's users now and you've got to guess who they are it's probably easier to just go find a bank that hasn't locked the doors yet. So there's some benefit there. Alright a couple of years ago yahoo.com in other words the domain that all of you get if you get a free yahoo account publish the p equals reject policy and this did a couple of things first and foremost all the folks that basically had no idea would get their own email address and had simply been using their yahoo mail address to run their flower shop or their plumbing supply company or insert small mom and pop business here they all suddenly found that their thing they were running with constant contact was no longer working because of course yahoo didn't authorize that they only authorized the yahoo mail servers. The other thing that happened was a lot of folks who ran mailing lists started to scream very loudly yahoo you have broken my mailing list what do they mean by that well it turns out mailing lists don't work super well with DMARC for two reasons. Number one there's that extra hop so SPF forget it it's not going to work and mailing lists like to modify stuff they like to change the subject line they like to put a photo on the message so DKIM is out the door but here's kind of the gotcha so let's suppose now somebody who's been on this mailing list for the last 10 years they have a yahoo.com address they decide to post today in the post reject world that goes to the mailing list server and the mailing list server goes and sends it to all 12,000 people on the list and a lot of those people get their mail at Gmail, Hotmail, AOL or other places that do DMARC however the message failed DMARC they're going to bounce it back but wait it gets worse most mailing lists have a little feature in there that hey if you try to send to a user three times and it bounces three times stop sending to that user forever so guess what happened as soon as about three or four people from yahoo posted on any given mailing list everybody who was on Gmail Hotmail yahoo etc etc etc got unsubscribed automatically from the list so this is what they meant when they said yahoo broke my mailing list well yahoo didn't break the mailing list yahoo is trying to protect people from fake yahoo messages and in doing so forgot a very important edge case that was a pretty big deal for yahoo.com not such a big deal for most companies okay I bet you most of the people in this room because you're all kind of engineering types probably do participate in a lot of mailing lists we are you know kind of not the normal folks if you will but for the average business user mailing lists really aren't that big of a deal what are some options though to deal with it well you can put the onus on the list participant so you can say hey list participant don't use a yahoo.com go get a gmail.com address they don't do p equals reject yet or go pick some other domain if your company has gone to reject ask them to create a subdomain that they leave at p equals none so your mailing list traffic won't have this problem now the big problem with that the advantage of course is you don't have to change the mailing list software at all or anything else you just change the address you're going to use the downside is you know okay so you switched to gmail I guarantee you gmail is going to do this at some point too and if you use a subdomain of your corporate domain then your company has left a gaping hole that the criminals could take advantage of we can put the onus on the list operator right instead of doing what they normally do send the messages from the list itself so yahoo user emails the list but now when it goes out to everybody instead of the from header saying the original guy from yahoo move him to the reply to and then use the list email address in the from header and that actually works pretty well you know and it keeps the mail flowing however it does change the semantics of reply versus reply all I guess mailing list users are used to when you hit reply it's going to just go back to the list is it or vice versa anyway the point being that by shifting those around some people were complaining that reply and reply all basically now inverted their use the other also you got to get the list operator on board the list operator may or may not be willing to adjust something that they see is not created by them fixing a problem there not all mailing list software you know has an option even to do this the good news is that one of the most common ones does and there is a link right there where they literally tell you exactly how to set it up which patch you need etc etc so if you guys operate a mailing list server and you want to keep your yahoo and AOL users happy you might want to have a peek at that it's a pretty straightforward thing you can also go put the onus on the list operator and basically do the status quo but just don't unsubscribe people for dmark failures at least you won't you know now the guy yahoo won't be able to post more but at least he won't unsubscribe everybody at gmail and AOL in doing so and then finally the receiver could detect that this is mailing list traffic and they could override the policy and this is actually what google does in a lot of cases however this is dangerous because now a spammer can use that mailing list as a way to send a fake PayPal message for example get the list basically to deliver it knowing that the policy will be overridden that's really a great solution the one that seems the most promising at the moment but is way too new the jury is still out is a new thing called authenticated received chain or ARC google has basically said look this is what we're going to do to fix google groups for this particular problem we're going to implement it internally and by the way we're going to put the specification out there so if anybody else who runs a mailing list wants to do this you have the option of sending the files and a couple of extra headers that are added they're digitally signed and then every time this goes through another hop somebody can basically re-sign the thing and say that oh yes indeed I checked the authentication and I'm passing it along basically it gives a way that by the time it arrives at google assuming they trust the people that have said they checked the authentication they'll at least know that the thing originally passed authentication spec.org all right how we doing on time got about ten minutes left all right I got to get moving here all right open source tools open deakim you can go download it what's it do for you well it's a C library so if you want to just kind of play with signatures you can code against that library and do some nifty things kind of play around with it but it also comes as a milter which is a mail filter it'll work with send mail of course and also with exim and with a little extra shim layer in there you can make it work with qmail basically a little fine for you it will also verify for you if you want it to do that yeah I believe it will so sorry post fix yeah yeah I believe it will I believe it will I'm 90 percent certain we'd have to check so okay open source SPF tools there's you know yes oh yeah absolutely exactly here's how you set it up I will be honest you know you're going to spend the better part of an afternoon reading the documentation and so on and so forth however at least you know based on what I've shown you today you'll have an idea in your head as to what it's supposed to do I mean that's kind of half the challenge of some of these packages in the first place I am also going to point you to a really good URL I've got a little resources section at the end there's a guy that literally walks you through the whole steps of getting this thing set up and working and he's done a huge service to the community by doing that there's a pile of tools available for SPF you know this lib SPF to there's various packages you can do it in Python Java even dot net if you wanted to there's various options there and interestingly enough open DMARC which is the thing that will actually do the DMARC verification if you want to protect your users from fake PayPal messages fake chase messages, fake bank of America messages the way you do that is you're going to load this open DMARC thing you're going to run this on your mail server once again it's a filter it's a mail filter and it actually has SPF checking built into it or you can set a flag and you can use the SPF library from the other thing if you are already using that so it can either generate its own SPF results or use the results from the other package it does however rely on OpenDKIM so you will at a minimum need OpenDKIM to do the DMARC check those are the tools I'm going to give you guys links at the end so including that awesome link that's going to literally just walk you through here's how you set it up and it's going to hopefully make everybody's life a lot easier let's talk about adoption so based on sort of what community DMARC.org has said these are the folks that are supporting it today and a rough count of how many mailboxes they have so Yahoo supports it and they white label the mail for AT&T Rogers and a few other places Google supports it including huh this is old there's about 6 million Google apps domains now so all those folks that use Google for their email infrastructure that is included in that coverage Microsoft is hotmaillive.com Outlook.com MSN and they're just starting to get their feet wet with Office 365 so their business users are getting it as well AOL Comcast NetEase is a big Chinese ISP QQ in China by the way even bigger than NetEase just started sending reports a couple of months ago so that's an exciting one and then some other smaller outfits you know mail.ru Yandex etc based on that we estimate you know US coverage of a typical consumer list at about 85% worldwide a typical consumer list at about 60% there are also things for business users yes question yep yes Apple is definitely working on getting iCloud and mac.com me.com getting those squared away they sent us some test files several months back and we tested them and they were close and we sent them back with a few little hey you got to adjust this this isn't quite working right they've gone quiet on us but I know they are working on it I wish I could give you an exact date the last word I have for them is they are absolutely committed to it they already do it on the sending side of course and they are implementing it on the receiving side so a little more patience and they will get there are you an Apple iCloud user or just oh of course yep yeah exactly on the business side so for open source yeah send mail post fix and Q mail you can use the open DMARC with Q mail you need this QSMTPD which is some other little shim layer that is going to let it work with the the milter interface on the commercial side of course there is google apps cisco iron port proof point, semantec cloud message systems again they use that milter message systems I think it is changing their name to sparkpost if you are familiar with them Office 365 however Office 365 will the most they will do is they will quarantine if you have a reject policy they will back it off to quarantine they are not quite confident enough that there is not a lot of mailing list users and other edge cases and alt n mdemon out of I think Irving, Texas I will fit down there if you guys know his name is Arvel great guy down there and they are supported as well alright however you know it is one thing to say that 80% 90% whatever of US consumers get their mail at a place that supports DMARC however your mileage may vary and what a lot of people want to do is they want to say hey what is my coverage if I implement this how many of my users will be protected I am not going to do it if half my users are protected 20% of them are so we start with a representative list of email addresses for most companies you have your list that you email on a regular basis we can look up the MX host we can see if that host is known to support DMARC by basically looking at that list of Google Apps any Google Apps domain lists the same set of MX hosts same thing for Yahoo same thing for Outlook Outlook.com and then you tally everything up at the end have your percentage coverage. So I wanted to do that. I went through that exercise. The problem is I don't have a consumer list. My company, we sell it to businesses. So I use the Ashley Madison breach data. So for those of you who don't know, Ashley Madison is a site where married people can make new friends. And there were some hacktivists who, how shall we say, did not really like their business model. And they said, we have your list of users and all their personal details, including all their communications with other users, their credit card details, etc. And we will release that unless you pay us X millions of dollars and shut your site down. Well, AvidLife Media, the parent company of Ashley Madison said, no, that's not going to happen. And sure enough, right on target I went the list. So anyway, in the brief amount of time it was available on the dark web, I got my hands on it and I went and ran these. And so I, you know, obviously a lot of the addresses were gmail.com addresses. So I did the look up and yep, google.com, da, da, da, check. Netflix.com. Ah, yeah, actually, if there were any Netflix users on there, they use Google Apps. Yep, check. UCLA.edu. No, they run their own mail server. What's that? Yeah. There were, yes, there were military addresses on there. There were quite a few from most of the major corporations that all of you know and love. There were also quite a few that were clearly fake. So there were a lot of profiles that actually used either AvidLifeMedia.com or AshleyMadison.com as the actual email address, like a very inordinate number of them. In other words, they were obviously chumming the waters a little bit, trying to make it look a little closer to 50-50 when in fact it was about 98-2% in reality. Anyway, tallying it all up, drumroll please, amazingly, Venezuela, at least amongst Venezuelan AshleyMadison users, has an incredible level of DMARC coverage. Almost all of them, 96.9% get their mail somewhere that does DMARC checking. The good old USA, 90.7%. Now, you know, this might be slanted slightly because I suspect a lot of the users of AshleyMadison probably set up a free email account just for that purpose and most of the common free webmail providers obviously are supported. Anyway, you can see the numbers there. Most notably though, and this really confirmed what we already suspected to be true, Germany and Japan had really the lowest coverage. And there's a good reason for this. Germany and Japan both, you know, they have their own technologies to do these things. They have their own webmail providers. In Germany, it's United Internet, so it's web.de, eints and eints, and GMX. In Japan, it's Yahoo! Japan, which is actually soft bank and is not at all part of Yahoo! despite the name, as well as NTT. So we are, at Agari, we are evangelizing this and we're hoping to get the German and Japanese markets up to 80-90% coverage so that their users can be protected as well. Yes. Exactly. So spewing out diesel hydrocarbons anyway. All right, so we are at the bottom of the hour. I don't know if there's an immediate intervening session. I'm basically done. I did have a couple of additional fraud examples if anyone wants to stick around for a moment or two. This one was kind of interesting. I was on my way to a meeting with Western Union and we saw in our data this sort of interesting spike of mail coming out of a .mil server, an army.mil server sending out a bunch of Western Union mail. And we thought, that's a little odd. Why would the army be sending out Western Union email? And then we had a look at the subject lines that the army was sending out. Did that not? There we go. And final notice, $7,600 US dollar daily payment for you. Obviously some kind of a spam campaign. When we see this many, it's not just A-B testing. They were doing A-B-C-D-E-F testing of their spam campaign. In any case, our CEO happened to have a buddy who did the mail for the army call them up and said, oh yeah, thanks, another one. They said, yeah, every once in a while we get a contractor on our network who's got malware installed and it starts using our server to send out a bunch of spam. Anyway, I feel safe at night. Okay. We had Twitter had an amazing amount of fishing. I mean, I never would have thought of Twitter as a place that you would want to get some of these Twitter credentials because I've got about 160 followers. So I don't know what you would do with my credentials. But apparently folks like the AP or CNN, they have a lot of followers. So I guess those accounts are kind of valuable. Anyway, Twitter found that there were as many as 100 million fishing messages going out every single day. So we worked with them, we got them locked down, put them to reject. And literally two days after they went to reject, the criminals figured out this was not working anymore and they just gave up. It's been flatlined ever since. Now that's kind of a cool result but the really cool result is that Twitter was able to take several dozen people that have been pulled from their day jobs to go and do account resets because of all the people that were getting fished and they were able to send them back to revenue generating jobs. So to me that's a pretty big result that it actually did impact the bottom line. Alright, last example. Is the next session like immediately starting? No, we got a couple. So this last one actually was a friend of mine. He had sold a camera on eBay. And then he gets this email. And I'll read it to you. It's the principle of small. It is my pleasure to inform you that I have carried out the payment of $4,800 via PayPal. I just got a receipt from PayPal. So if you haven't, please check your spam now. I am sure you must have been notified also. I want to send it to my elder brother for work. I would have asked you to send it to my eBay address but I'm currently on a business trip to Kansas. Here is the delivery address. And of course it is in Nigeria. Yes, okay. Anyway, and then he says, hey I really want you to ship this today. I gave you an extra $100 to cover the quick shipping and all. And please give graphic carefully. Anyway, the key point is you can see what was happening here. Obviously this person thought they could just send a spoofed PayPal message. Maybe it was going to end up in spam, hence the check your spam folder. Thinking that this person would just read that and say, oh, okay, I got my money. I'll send it on. Like never mind, they're not going to actually check the PayPal account. Anyway, the point is that a non-savvy user, someone who hasn't had 5,000 of these conversations with me, might have just simply sent the money on had they gotten that email. But because of DMARC, they didn't get that message. So save somebody some money. Anyway, that's it. Thank you all so much. And where did that go? There we go. Yeah, first of all, I'd like to get that 160 followers up to maybe 200 by the end of the year if possible. So by all means, feel free to follow me on Twitter. It's JMW secure. I mostly tweet about cybersecurity stuff. So you're not going to, if you want my political leanings and all that other stuff, that's my private account and whatever, but this is all business. And then here's some very key links. And again, it's in the presentation that's going to be uploaded and available online. Great. Thank you all. So is this mic working? Can you hear me in the back? Yeah. Okay. So anybody that didn't get one of the coloring books, they're all available online for printing, I think if you Google SE Linux coloring book, you have a good chance of finding it. This year we did one called the container coloring book, which was available in the Red Hat booth. And I think they gave out several hundred of them. They also gave out a lot of SE Linux coloring books, which were supposed to be saved for this. So I told them to stop. I'll mention it to you. We can talk about it afterwards. Yeah. Yeah. That's a question towards the end. That's a question. I'll cover that. I'm the ex-lead on SE Linux. I'll explain that in a second. I'm now the lead on Docker. So container technology at Red Hat on the head of it, which I use SE Linux for. I do. 15 years. Nope. Listen to my voice. You'll figure out where I'm from. I'm from Boston. So I have this on. Engineering at Red Hat is actually based out of Westford, Massachusetts, yeah. Just the business people. Just the business people in Raleigh. We have an office of about 600 people, 500, 600 people in Westford. And we have an office of about over 1,000 in the Czech Republic of Anginais. Yes. I'm here to sign. Hi everyone. Back to the security track. This is a three o'clock talk on Sunday. Up next is a fresh look at SE Linux with Daniel Walsh. Okay. Thanks for having me. I just wanted to, a lot of times I'm going to go through SE Linux. I'm going to try to teach you. If you've had SE Linux taught you before, I often think it's taught it way too difficult to level. And so I'm going to try to go through it at a much lower level. I'm going to use the coloring books. Sadly, I ran out of coloring books for everybody. About a year and a half ago, I heard about this bash exploit. And so I quickly wrote a blog. I quickly did an investigation and see what SE Linux did about the bash exploit. And later that day on the ride home, I was listening to MPR, and they came up and they talked about shell shock. And I said, what the hell is shell shock? I had no idea what it was. And after writing this article, so if you see this article, it's actually talking about shell shock, but I called it the bash exploit because whoever adds cool names to exploits had not reached me yet to tell me that they renamed it. So an interesting thing with shell shock. So shell shock instantaneously made every single web server in the universe that did bash, ran bash, or CGI scripts, was vulnerable. It could be totally taken over every single web server in the world. The only thing to stop that exploit was SE Linux. So the day it happened, any machine that was running with SE Linux in enforcing mode blocked it. How many people in this room were running the way? I'm not going to ask this. I hate to hear this. But that's the power of SE Linux. SE Linux is not catching known problems. Not necessarily catching known problems. It's catching the unknowns. So SE Linux puts everything into a box in control. So I'm going to talk about that a little bit later. I just wanted to point this out. We're going to talk a little bit about virtualization, multi-tenancy, and some of the stuff. So for those of you who don't know who I am, I used to call myself off in Mr. SE Linux. So I started at Red Hat 15 years ago. At the time I have a security background. I work way back, way on in Project Christina Kerberos. I worked on VPNs back when VPNs were just being created, firewalls. I worked for Digital Equipment Corporation. I worked on the first firewall. The first firewall at the White House. WhiteHouse.com. WhiteHouse.gov. WhiteHouse.com was a porno site. I remember seeing Hillary and Bill on it one time. So 15 years ago NSI came to Red Hat. 15 years ago I started working at Red Hat 249.11 to give you an idea of how long ago it was. And a couple months into my tour, about six or seven months into it, NSI came to Red Hat and asked, this seats up front here if everybody wants to squeeze in. Came to Red Hat and said, we have this really cool new security mechanism that we really need to get mainstream. So we want to work with you guys to get it into the upstream kernel and everything else. People went around Red Hat and looked for anybody that had the word security in their resume. The name popped up and I became the SE Linux guy. So for the next 10, 11, 12 years, however it was, I worked on SE Linux. I also worked on a thing called namespaces. So in REL 5 we introduced a concept of namespace. It was the PAM Mount Namespace. The Mount Namespace was introduced for a thing called MLS. So SE Linux allowed us to get to MLS multi-level security. We're going to cover that a little later, but we actually introduced this concept of the Mount Namespace. Eventually the namespaces grew into a bunch of other namespaces. So I became, I am now the head of container technology at Red Hat. So I have the Docker team underneath me. So we submit patches all the time to Docker and I'm constantly fighting with them to get my patches accepted. But that's what I do now. I also did the Docker part of SE Linux. So let's get into it. But you're not here to hear my history. Let's go in and look at that. So I call this topic. I actually combine two different talks here together. And I'm not sure we run out of time. Are we the last talk of the day? Oh, shoot. Okay. I was hoping to go over. I can go 30 minutes over. All right. So they throw me out. Okay. So SE Linux, when you get in SE Linux era, it's trying to tell you one of four things. And that's really a lot of what this talk is going to be covering. And so you have something wrong with your labels. You change your file system's fault and tell SE Linux about it. Applications or SE Linux has a bug or you've been compromised. That's really the only thing it's going to tell you. The problem is trying to figure out which one of the four it is and how to react to it when you get one of those. So let's... So the first, the most common error for SE Linux is you have something wrong with your labels. So everyone please stand up. This is the interactive section. Everybody's ever seen me. They've had to do this before. I make everybody do this every single time. Okay. You're going to read the screen. Okay. In unison. This is the church of Dan Walsh. Ready? Yeah, you don't have to do that one. Okay. That's it. Everybody knows what SE Linux is. Okay. Sit down. So SE Linux is a labeling system. Okay. Simple as that. I don't want to talk about... We're going to cover some other deeper topics in it. But it's a labeling system. If you label some wrong, things are going to blow up. Anybody see the canonical talk yesterday keynote? Okay. He put up a slide that I would love to steal and probably won't let me. And he put up the way the Linux operating systems have evolved over the years. Or actually Unix has evolved since I was, you know, a young stuff. And then we have applications. They look like amoebas, right? So you install them. And these applications can be configured 100 different ways. Okay. So you have the Apache website. How many different ways can you configure Apache? It's basically 1,000 million. Millions of ways, right? And then you have databases. They can listen on network sockets. They can listen in local sockets. You can change the socket. So you can change all this configuration. Then we have to throw in other applications. MariaDB, slightly different than MySQL. So with this constant change in the operating system, and SE Linux wants to control all of that. So if you install and reconfigure anything in SE Linux, you have to fix the labels. Okay. Let me tell you about another labeling system that no one in this room probably thinks about. Discretionary access control. Does anybody even know what that term means? Discretionary access control? Okay. About half of you do. Now let me tell you what it is. Ownership, permission flags. Read, write, execute, okay? That's what's called discretionary access control. So I have, every process on the system has what? An owner and a group, right? It actually has an owner. So there's an owner or a process. Every file system object has a label on it. The label is the ownership, the owner, the group, and the read, write permission flags. So if you get permission denied anywhere on the system, the first thing you do is you go and fix the labeling system. It just happens to be the labeling system you are familiar with. So you're not going to go in and say, a patch you got permission denied, let me make sure the file is owned by the Apache owner. Let me make sure that the search path has proper rights the Apache owner can search through it. Okay. At which point, you run it, everything looks good, you run it again, it blows up, at which point you say SE Linux sucks. Okay. And that's what you guys all do, because you don't think about fixing the labeling from an SE Linux point of view. Tom Hayden. Everybody's gone to everybody's seen this before? This is actually a website, stopdisablingselinux.com. It's done by Major Hayden, he's one of the top security guys that, oh, I'm at the blogging, he's the name of his company, Bragg Space. So he put this up, he's a good guy, and he actually gave me the Set-and-Force one shirt. So here's the real world example. This actually happened. The guy in the next cube for me had an egg dispenser. What is this egg dispenser dispenser? M&M's. Someone put skittles in the M&M's dispenser. Okay. Talk about permission denied. That is, I hate skittles, okay? And he wanted this slide going up. Okay. So everybody get out there, SE Linux textbook. You can start looking through it. This textbook, the SE Linux coloring book, is probably the best document I believe ever written. So you can go out to Amazon on SE Linux. You can go out to Amazon by those big thick textbooks, or you can read this, okay? And you will have an understanding of SE Linux. You can also, for those that are not in the room, it's available on the internet. One of the reasons people really like it is Maureen Duffy does it. She works for Red Hat, I'd say. All sorts of UI stuff. So you can get this and print it out. We've given out hundreds of thousands of these. Well, I'm probably not hundreds of thousands of thousands. And actually, interesting story. Any Japanese in the room? Okay. When we gave this out, Japanese Red Hat came to us and said, can you make it a comic book because Japanese people don't know what coloring books are. Is that true? I said that sounds like the stupidest thing I've ever heard of. Anyways, I told them to color it in themselves and call it a comic book. Okay. So SE Linux is a labeling system. If the labels are wrong, SE Linux will generate errors. Solution, picture, labels. I stopped myself. Okay. Okay. So we're going to talk a little bit about the labels now. So we're going to go a little, this is where we're dropping down a level and this is going to cover the coloring book. There's two types of enforcement that SE Linux takes advantage of. The first one we're going to talk about is called type enforcement. And this is where we really define the type. You're in a patchy process, right? So if you look at the SE Linux label, it's kind of ugly looking, but there's user, role, type, and level. And we're really talking about the third field after the colon. And this is the type field. Every process on the system has a type. When I log in to a system, most people log in, it's administrative, they run confined T. That's the unconfined type. You're able to do anything you want in the system. But if a process starts as a system process, like a patchy, that's going to run as the Apache type, which is HTTPD underscore T. When the shell shock happened, if the shell shock person logged into the system, he was running as HTTPD, he actually is the CGI script, which runs as HTTPD, this CGI script T. So when the CGI script was running bash, the way bash exploit worked is you were able to say something, you were able to pass something to the end of the bash command, or the end of the CGI script, that the bash command would then execute. So when SE Linux are, the CGI script came up and said, hey, I want to look at user's home directories. And SE Linux stepped in and said, no, you're a bash, you're a CGI script. You're only able to do CGI things. And said, hey, I want to try to run sudo. No, you're a CGI script. You're only able to do things with CGI script. Had nothing to do with permissions? Nothing. You got a world readable files on the system? Anything? SE Linux will block it. So there's the standard type for Apache web server. And here's the content type. An interesting thing about SE Linux, that people that start to struggle with it often think, is that if I have the type of the process, I want to assign it to the type of the file, right? So you don't assign process types to file types. So there's a separation. You have a process type, and you have objects, data objects. So this is the type that Apache is able to write to, most types are read only. So most types that Apache is allowed to read to, are read only. But let's take it back a step. So when we look at type enforcement, let's look at a simple example of a cat and a dog. So I have a type, and I have there, this is a cat type. And then I say this is a dog type. These are processes on your system. And then we have objects on the system. We have cat chow, and we have dog chow. So those are objects on the system. Then I write a rule to the policy engine, and this is exactly what the policy rules would look like. This policy rule says allow the cat process to eat cat chow food. So food is like file, directory, whatever, but basically it's a category of the object. So basically we write thousands of these rules. It says allow cat to eat cat chow, allow dog to eat dog chow. That's the Linux system like all firewalls rules is denied by default. If I don't have an allow rule on the system, you are not going to be allowed to do it. The process is not going to be allowed to do it. So the cat's able to eat the cat chow. The dog's able to eat the dog chow. The cat's able to ask the colonel for more cat chow. But when the dog tries to eat the cat chow, the colonel steps it. Any questions? That's what type enforcement is. It's that simple. It's type enforcement. We're going to get a little bit further into this. So that's the Linux labels. Now we're going to cover... So that's the standard way we protect systems. Now I'm going to put on my other hat. So type enforcement works real well for use cases where you have like an Apache server and you have a Apache data or you have a database serving a database data and you have home directory data. Where it doesn't work well is what I call multi-tenant systems. Multi-tenant systems are things like Docker, containers, VMs. I'm going to show you OpenShift in a second. So there's lots of use cases where you're going to have hundreds of VMs running on your system and you want them to have the VM and this is VM data. But what I want to do is I want to make sure that VM A is not able to read VM B's data. I could create all these types but all of a sudden I have thousands and thousands of types. So what we needed is the different types of security for the system. So this is back in the SBIRT, probably around 2008 timeframe. We invented the concept of what we called MCS separation or SBIRT, secure virtualization. Secure virtualization, by the way, I'll give you an example where SC Linux is ruled. There's been two breakouts at KVM on RHEL systems over the last, I don't know, six, seven, eight years. There's been two breakouts through the KVM. So if you have a process running in a VM, the process breaks through security in the VM, then it breaks through the hypervisor and actually gets control of the host system. So that's the object of any hacker that's running in EC2 right now is to get to that point because then you can take over the universe. There's been two known breakouts, two known vulnerabilities in that. Both of those were blocked by SBIRT. SC Linux controlling what's going on. All right, but let me go a little further into what this type of protection is. So we can identify everything as a Docker type or as an OpenShift type or as a VM type, but we have to have some other piece of data that gives us separation. So an SC Linux system has this fourth field called the level field. Anybody in here ever do anything with trusted systems? All right, just one or two, three people. So things like trusted Solaris or SC Linux all the way back since REL 5. The concept was this multi-level system where you would have a top secret process and a secret process. And there's rules that were written many, many years ago back in the 1970s that said a top secret process can read secret data and a secret process can write top secret data, but a top secret process cannot write secret data. Pretty strange rules, right? But that's what MLS, standard MLS, and that's not really totally the way they work, but the idea was that if I have a secret process running a secret process reading secret data, I don't want that secret process to be able to write non-secret data. Okay? That is the WikiLeaks stuff, right? I don't want to take the top secret data and let it get down to secret, so that's why top secret processes can't write secret data. Similarly, the other one that's weird is that a secret process can write top secret data, and I always equate to the soldier sees the tanks over the hill and he's going to tell the commander that this tank's over the hill, so he's actually writing data that might be... he might be only secret, but when he tells the generals, all of a sudden it becomes top secret data. That's the basic idea. So anyways, the problem with MLS, which we introduced in REL 5 back in 2006, is that almost nobody in the universe, other than the military, sets up their systems to run in that way. So we had this fourth fail that we weren't taking advantage of, so most targeted systems we've played with at SELINX So we decided to take advantage of that. We made a new thing called MCS, which is multi-category security, and what we did is we basically allow us to assign sort of random labels here, and the rules basically say that that random label has to match. So if I have a process that has the label of S0, C1, C2, I have to have an object in the operating system that has the same label, and that's what SBIRT is. So we'll go look at it. So SBIRT, if I'm running a virtual machine, and then LibVirt, the tool that launches virtualization in KVM, will pick out a random MCS label and assign it to it. Then it will pick out a file on the file system, assign the same label. And then, basically the kernel will say that these two have to match, otherwise you're not going to be allowed to read or write that file. MCS, so it's multi-category security based on the multi-level security, which I went over pretty quickly. Next chapter of the SC Linux coloring book to cover MCS security. So in this case, we have two dogs. So they're both the dog type, but what we want to do is we're going to assign a random label. In this case, it's not so random. We're picking a name, but it's FIDO and SPOT. We're going to now use our example of dog chow for both of them, but now we've added an MCS label part of the dog chow. So we've said that that's going to be FIDO and this dog chow is going to be SPOT. And if SPOT's eating, in this case, the colonel must have been said permissive because FIDO is eating SPOT's dog food. So in enforcing mode, FIDO would be allowed to eat his dog food and then if the colonel would stop FIDO from eating SPOT's dog food. So that's basic idea of MCS. And now let's go MCS enforcement protects liked processes from each other, VMs, OpenShift gears, SE Linux sandboxes, Docker containers. I figured this thing out. I've got a big hammer and all I've been doing is looking for nails since then. So everything that is a multi-tenant environment needs this. Rocket now supports this. So if you're using Rocket Container System, the N-Spawn supports it. So all these environments support this type of security. The tools that launch the containers, that launch the object, all content, launch the process, same label. And then the tools, the tool is in charge of guaranteeing the uniqueness. So Docker has to guarantee uniqueness. Libbert has to guarantee uniqueness. Rocket, any other tool that's going to do it has to guarantee the uniqueness. So let's take, hopefully this is not too, too loud. Today I'm going to demonstrate how SE Linux controls what an OpenShift gear is able to do on a system even if the OpenShift gear becomes root. I start out by showing that the process on the system is running as unconfined T and I'm about to use RunCon to become OpenShift, the OpenShift label. I also show that I'm running as root on the system. So if I run the RunCon command here, I'm going to switch over to a label of statue on system R, OpenShift T S0 colon C0 comma C1. It is launched on an OpenShift system. They are always launched with the OpenShift T type, but each OpenShift gear also gets attached to it a unique MCS label. Here it says 0 colon C0 comma C1 is associated with the process. Also the gear gets its content also labeled with the same MCS label. So now that I'm running as OpenShift T, I'm going to show you that I'm running as root on the system, but now if I try to cat at the shadow, I'm going to get permission denied. If I try to create content on the system, I'm going to get permission denied. And even if I go into the byline OpenShift directory where the different gears store that content, in this case I set up two directories for my demonstration. One called Gear 1 and one called Gear 2. If you notice Gear 1 has the MCS label 0 colon C0 comma C1 and Gear 2 has S0 colon C3 comma C4. So if I show I'm in OpenShift gear trying to attack another gear, in this case representing myself as Gear 1, I'm going to try to go into the Gear 2 directory and look around. In this case I get permission denied, but if I attempt to go into my own directory I can see content in there and if I want to create new content I can create it. If I look at the labels on the content that I create it's all labeled with the MCS label of S0 colon C0 comma C1. So now if I get frustrated with SC Linux I want to turn it off. I can attempt to turn it off with setting for 0 and of course we're also blocked from being able to disable SC Linux on the system. So this little video demonstrates what SC Linux can control OpenShift instance is able to do even if the OpenShift instance somehow became root on the system. Okay, so to give you an idea OpenShift, does everybody know what OpenShift is? Okay, a few of you do. So OpenShift is a PADS server that Red Hat introduced I think four or five years ago now. Okay, matter of fact that the guy that developed it was Mike McGrath who's now works with me and the Docker team but this is OpenShift V2 that we're demonstrating. OpenShift V3 is based on top of Docker containers with Kubernetes and stuff but basically it's going to use the same technology. So OpenShift is sort of taken off. I think they have, they now calculate they have about two million users of it. OpenShift is based in EC2 cloud instances. Anybody in the world can get an account there all you have to do is provide a username and password provide an email address and you're able to get an account. So OpenShift V2 machine. So we give you Shell because we figure if we're giving you the ability to build web services on top of a PADS server you're going to figure out how to get to Shell on top of that system. So right out of the box we give you Shell. We take away no security, no tooling from you. So we allow you to use any tools you want. The only thing we do is we wrap you with SC Linux to make sure you break out. Now we have had over four years with two million users known exploits. Now maybe the NSA has done something. I don't know. But we've had nobody broken through the security with a Shell account on the system. We also monitor the heck out of this, make sure the kernel is totally up to date things like that. So the attack back to here is it would be through the kernel. Yes. Because of the NSA my answer to that is if I worked for the NSA and I wanted to exploit all machines in the world I would not put an exploit into the thing that comes from the NSA. I would have someone get a random email account, give a patch to NVIDIA driver or something like that and that would be my attack vector. This is open source right. So it's open source. Everybody in the world has looked at the SC Linux code. It's gone through multiple review processes. It's far more likely that they haven't, if the NSA had an exploit, it's far more likely that it's going to do that. So turning off a security feature that has no effect on the kernel. That's just based on the kernel. You don't know what's going on in the kernel whether SC Linux is enabled or disabled. The code is in the kernel. That to me always seems ridiculous. That's coming up in the set. That's coming up in the, well the labels are written in policy. So this policy right is right, so we, the policy right is right to rule. So I'm going to show you how the labels get onto processes and labels get on. So when Libvert launches the virtual machine, it picks out an MCS label to assign to the process. Then it assigns the image to the same label. When Libvert takes down the container, it basically kills the process and then re-labels the label on the VM to something that no VMs are allowed to read. So it can change the labels all the time. It just doesn't leave permanent content on it. Yep. It's just a sequence of letters. There's more to it than that. Those labels in an MLS system mean something. In an MCS system, we're just taking advantage that we have this construct of MCS labeling and we're just using it. The reason we use two labels, so we use S0 is the sensitivity level. So in an MLS system you'd have S0 to S15. We only use one sensitivity on the system and we have 1024 capabilities. So we can use 1024 two character combinations. That gives us a half a million potential unique containers The uniqueness is basically 1024. The half a million unique VMs you could run in a system. So when we designed it in 2008 we thought that was pretty good that we wouldn't have a problem. That was before we got the containers. Now theoretically containers are going to, as we see with OpenShift they have two million users all of a sudden we have potential problems with that. But guess what? I can add a third label to this. So I can use S0, comma, S1, S3 and all of a sudden I get another multiple of 1024 times that. Minus some numbers, but anyways it can get huge if we really find that to be a problem. So most SE Linux issues are caused by labeling issues. SE Linux should work by default if everything is labeled correctly. Sadly not everyone wants to store their content the way I tell you to store your content. Alright. There's a problem. Every process and object the machine has a label if you file the label correctly access might be denied. Alternative path to confined domains SE Linux needs to know. So if you want to install your Apache server in some random directory on the system you got to tell SE Linux about it. If you want to move your content your Apache content to random directories like slash you. I don't know why people say slash you but it must be an old Solaris thing. Or you have slash server slash web. There's certain directories that people put their content in. You have to tell SE Linux about it. The way you do it is there's an SE manage command which is the SE Linux management command and that's SE manage file context. So this is basically saying I want everything under this directory to be labeled as Apache content. When you use an SE that just tells SE Linux what the default should be. It does not change the I nodes on the system. I didn't mention earlier that SE Linux is based on I nodes. So the same way discretionary access works the label is associated with the I node on the system. It's not path based but it gets assigned from a path based system. It gets assigned to the labels get assigned via the file context file. After I set this default I have to run a command called RestoreCon. If you play with SE Linux for a while you're going to get real familiar with RestoreCon because RestoreCon basically says change the labels of this file or this directory back to the default. Go read the file context file and change the default. Lots and lots of tools on the system have file context built into them things like RPM. So when I install a package on the system RPM will automatically set the correct label. So when I install the default Apache server it gets installed. So that's how the labeling initially gets created on the system but if you want to put contents in random directories you're going to tell SE Linux about. So file labeling. So the file context file is stored into this directory at the SE Linux targeted context file context. So all the labels are stored there. If you want to go and look at that file you're going to see thousands and thousands of reg X's that way we've learned content gets stored. File labels are usually stored as X adders. Almost always stored as X adders. Certain cases file systems don't support X adders. Yes. Yes it performs perfectly zero performance hit. Is there a cost to discretionary access control? Okay. So it's the same basic cost. Pretty much with the modern they just did some studies on it. We published them. SE Linux has negligible effect. You can set up horrible situations. It's not the labeling on the system that's going to cause it. It's basically the kernel checking access control on there. But the kernel is already going through lots of access checks every time you access an object. So for the most part anybody that disables SE Linux because of performance unless you're Wall Street trying to get an extra micro second you know whatever you're fooling yourself. There's so much CPU that it's not really a problem. Yes. I'm never going to get through this. It's going to go about three hours. Right. So even in that situation the one place where to me you would have to really worry about is network throughput. So that's where you really need to check because again for the situation where you want packets flying as fast as possible the SE Linux system is going to be checking every single packet that comes in. It potentially could check every single packet come in to make sure that it's a secret packet versus a top secret packet. So there are those checks built in the system. So MATCH Kafka is another tool that you can use to figure out what the path is going to be. On the SE Linux system if you played with it any core utilities, any commands you want to run always look for the dash capital Z. So that's sort of the SE Linux flag that tells you what the labels are associated with objects on the system. SE managed context files used to change the default labeling and restore can't command takes the default labeling and applies it to the X adders on the file system. I already showed that. So file labeling so government entity uses Tivoli. I always change these to protect the innocent. Tivoli stores log files and via IBM Tivoli Common Cod Logs. Anybody have any idea why vialogs wasn't good enough for IBM? It's this type of stuff that drives me crazy over the years. It's like we'll just take a minute. If you ever looked at an Oracle system okay so we don't know where Tivoli stores its logs so someone installs a log package in here they put the logs in there. They tell the log rotate tools and the log analyze tools that this is there. They forget to tell SE Linux about it. SE Linux has a problem with logging tools because they run his root. They analyze random code that can be generated by anybody and then they take actions based on that code. Well that seems like a good exploit target vector right? So we needed to have that. I see ABC on confined domain saying it's trying to access VAT. VAT is the default label of VAR. We don't have labels for IBM storing in random locations. I see labels. So how do you figure out what the label of this be? It has the word logs in there. I instantaneously say I wonder what the label on Vialogs is. And I go look at that and it says LS is Vialog and I see it's VialogT and so I run an SE manage command that says the manage fcontext add VialogT to IBM's weird location for its logs and run RestoreCon to set it. I usually go at a fairly global level to run these but you could have done it down to the path. Problem solved. Default T files. So SE Linux knows about all the stuff that's in the cache that comes from the distribution. Any other crap you put into slash I have no ideas and SE Linux has no idea what it is. It could be top secret data. It could be credit card data. So we don't allow any confined domain on the system to look at anything that's in the slash directory that we don't know what it is. Similarly we don't allow it to look at VART. So VART we don't know what that content is. So by default all non-distribution directories in slash will be labeled default T. SE Linux has no idea what content is. I already said that. So big retailer moves its home directories to slash U slash home. This is another Solaris thing I think. I don't know why slash home wasn't good enough for them but they wanted it in slash U slash home. So slash homes, I guess what you're going to label slash home as. Anybody know what directory we should look at? Slash home. Very good. Okay. So this means all home directory content is labeled as default T. So we have confined domains like SSHD and Apache that want to look at content in your home directories and I don't know if it's credit card data. So we block it. We probably want you home to be the same label as slash home directory. So what do we do? Well one of the problems with just doing what we did with SE managed there is we could probably have multiple things labels underneath slash home differences. My home dot SSH directory is probably labeled differently than my home slash public HTML directory. So I don't want to go through and execute the SE managed command 400 times because you know so what I do is I can do equivalency. So equivalency label says f context add I'm actually doing a regular label here so I'm doing it saying slash U I'm just going to name it the same as slash home but then I do an equivalency and all it says is slash U slash home as if it was under slash home and when restore con U slash home goes in and asks the system it actually asks but slash home de-wash set the labels instead of slash U slash home. So it's pretty simple. Labels under all files under U home as if they were labeled under home slash U home de-wash gets labeled to that and public HTML gets labeled to that. So it'll use the same labels as if under home and of course you have to run the store con on it. So a common problem creating contents in my home directory and then I move it somewhere with the MV command. Everybody knows the rule about the MV command when you move content, right? It maintains the security attributes of the object that you're moving to the new location. When I move a file to a root system it does not change from de-wash to own a root. It stays as de-wash. If I copy a file slash directory and that file does not pre-exist. I'm running as root when I copy it because I wouldn't be able to do it as de-wash. What happens to the security attributes of the file I create? It's owned by root. So guess what? When I move an object on a system from say I'm creating index.html in my home directory which is use a home t, it's use a home content probably where I store my credit card data and I move that index.html to www.html. It maintains the security attributes that it's de-wash and it maintains the security attributes that it's use a home t. So now I go and run Apache. Apache gets permission denied. I go and say, oh, I forgot to change the ownership to Apache. I change the ownership. I run Apache again. It gets permission denied. And what do I say? SELinux SELinux sucks. Follow me. Follow me on this. Okay. So this is the use case and this is probably one of the biggest use cases for Apache content getting mislabeled. It's either created in the slash temp directory and slash root slash home directory. I'm happy with that. I run a Firefox. Now I want to move it into production and I forget to fix the SELinux labels. So I get that. So I check the permissions. I already got that. Okay. So what I can do after I move the object into the place is I can run RestoreCon because RestoreCon is going to look at the path and force everything to be the way SELinux likes underneath that directory. Yes. If a confined root user, first of all, SELinux knows nothing about the root user. It's a label on a process. As I showed you in the OpenShift example, that was the root user trying to cast Etsy Shadow. It's labeled as OpenShiftT. It's not allowed. So root means nothing. Root is a discretionary access control non-SELinux issue. So move preserves labels. There's a move in REL7 now. There's a move capital Z. And what that does is everybody that uses move with SELinux expects it to do the RestoreCon for you. So now we have a move minus Z which will relabel the content of the system. So if you do this, which I have on my system, suddenly things start to work much better. The NSA hates the idea of this. Not the discretionary, but if you think about the top secret, if I move a file around the operating system and all of a sudden it goes from top secret to unclassified, that's a problem. But for targeted systems, it makes some sense. That's why we can never change the default to be minus Z, but you can change it yourself. Not a common problem. My machine's labeling is so wrong, it will not boot. So there's a kernel parameter you can use called enforcing equals 0. It's different than SELinux equals 0. If you boot a machine with SELinux equals 0 because you think SELinux is screwing up your system, you can disable SELinux. By SELinux, the operating system is smart enough to know that you booted without SELinux. So what it's going to do, it's going to put a little file slash .auto relabel and the next time you boot with SELinux, it's going to relabel everything because it has no idea what you created while you were logged in without SELinux. So it forces the relabel. If you boot with enforcing equals 0, that means that the system is running without SELinux. So the labels will continue to get created correctly, but SELinux will not block anything. Touch auto reboot will reboot the label of the entire system and if you have the example you were using where you have 8 million files there, a mail server type thing, go get lunch. Because it's going to walk through the entire file system looking at every single iNode and checking to fix the iNodes. Usually on a modern system it probably takes 5 to 10 minutes on a normal laptop, but on massive systems it can take massive amount of time. So brace yourselves. How do labels processes get on labels? SELinux implements process transitions. I can write a rule and policy that says when process labeled A executes a file labeled B exec T the kernel will transition the process to B underscore T. So I'll give you an example. When the iNode system which is running as iNode T runs Apache which is labeled as HDPD exec T it will run it as HDPD T underscore T. So that's how transitions work. That's how processes get their labels. So it's a rule that looks like that. Look what happens at boot. Okay, so this is a fairly old slide that doesn't even cover system D, but when a system boots up in SELinux the first thing the system does when it comes up this is somewhat of a lie because there's no SELinux when a machine boots up. So the machine's booting up it's going to read in the SELinux policy off the file system and what the policy says any process on the system by default at the point that it reads in the policy is going to be kernel underscore T. The policy has a rule that says if kernel T executes an init program labeled T it will run that process as init T. So kernel T runs system D. System D is labeled this. It runs as init T. Now we go down and it runs an init script which is labeled init R C exec T transitions init R C T that goes down and runs K log exec T which transitions to K log exec T or init name D exec T so obviously this is a REL 6 system because system D works a little bit differently but not too much differently. System D running gets down to login at which point the login stuff works a little weird. So login program has SELinux knowledge in it. So that login program is going to figure out what the login the user in so it has SELinux. It picks the labels that log in it. But this basically is how all the processes get on the system. Most processes you exec stay in your context. So if you're running as unconfined T and you execute a program it's probably running as the user T. You execute Firefox and it will run in Firefox T. So we have that type of transitions for user space. How files get created on creation. So two things to happen. We've told you how they get initially labeled via RPM in the file context files in RestoreCon and we told you how processes get labeled. Now we're talking about an administrator or some object on the system is creating new content. How does that get labeled? Multiple ways. The first way is how it gets created based on its parent directory. So if you create a file inside of a file directory labeled SET by default it will get created as SET. If a tool like SELinux can override that so RPM for example tells the kernel I'm about to create an object please create it with this label. RestoreCon changes it. For example password creates two contents when it creates objects on the system. It creates SET password and SET shadow. So it's going to create these with different labels. So password has to have SELinux awareness. Policy writers can write rules to do what we call file creation transitions. And this basically means when a process is labeled network manager T and it creates a file in a directory labeled SET we want it to create it as netconf T. That means when it creates Seresolved.conf it creates it with a label of netconf T instead of being SET. So we can write all these rules into policy and we can do this is what the policy would look like network manager T creating SET we'll label it as netconf T and create it as a file. So there's a fourth one that just came this is only in rel7 not in rel6 and this is actually a huge advantage to usability in rel7 rel6. We can write rules that says when you create a file you can do a transition based on the name of the file. Not the path of the file. Path means nothing to Linux kernel. Names of file objects mean something. Paths mean nothing. So I can write a rule that says when an unconfined T process creates a directory in a user home D dirt T which is the home directory label I want to create it as SSH home T only if it's named .ssh. So one of the biggest bugs that we get for rel6 and rel5 is admins set up a brand new SC Linux system they go into root.ssh they create the root.ssh directory and all of a sudden they try to set up public key automatic logins and it blows up because this is HD and it's not able to read content in the root home directory because they have to run a stock con. This actually fixes that problem. So we have hundreds of different types of content so if this happens to go on creating random content that we know about we're going to label it so if he creates that to resolve.conf we label it correctly so we can set up all these rules. We actually have patents on this to improve security of the system because we can actually tighten up some of these file transition rules. I've just beaten the living hell out of labeling. But there's a hell of a lot more. If we really wanted to go through it we could go over three days but the first thing I've talked about up to this point is if you pick up one thing it's for stock con and the sdmanage command. Those are the two most important things we just covered. I gave you a little more insight of what's going on in the system. Section two is you change the system to false but do not tell sdlinux about it. sdlinux needs to know. These are when we write policy if I write policy to just allow Apache to work I end up writing unconfined T because Apache can be configured in 4,000 different ways. Things like FTP. I set up an FTP server. I want FTP server to share via FTP to the internet. Someone else wants to use FTP to be able to share home directory content to the internet. Someone else uses FTP and they want to share everything on the system to the internet. How should I set up the default policy for the FTP? We introduced many many years ago the idea of Booleans. That allows the policy writer to write these Booleans into policy and allows an administrator to toggle it a little bit. Booleans are if then else rules written into policy. If you want Apache to send mail you can do that command. How many people here set up their Apache service to send mail? Good. Very few. The default is that it's not allowed to. Why is that bash exploit happens all of a sudden you become a spam bot. Out of the box you would expect your Apache service not to be able to send mail. Except for the two guys in the room who wanted to send mail and they're going to say SELinux sucks. Okay. They're going to do 7-4-0 and disable SELinux to get the hell out of my way. Okay. We have the Booleans to do that. If you want to use it to use FTP to access this home directory you can set a Boolean that says FTP home dirty. How do I know about the Booleans that are available in the file context? The Boolean list will show you all the Booleans. We actually have in rel 7 there's over a thousand man pages. Every single confined domain in the system has an underscore SELinux after it. So it's not bad to do man-k SELinux and then grip for some strength. So if you wanted to look up a man htpd underscore SELinux. By the way I didn't write a thousand man pages. Okay. So these are all auto generated but it does a good job of that. And this thing will cover all the stuff like what are the Booleans available for the Apache Deam and what are the types available for the Apache Deam. So there are lots and lots of man pages to help you out. There's a tool called SE Troubleshoot which usually runs on desktops and anybody that's played with the Linux desktop or Fedora desktop has seen SE Troubleshoot. SE Troubleshoot I used to say is Dan's brain in a box. And then gives you a response that says hey it looks like Apache's trying to send mail. You might want to turn this Boolean on. Okay. If it wants to send mail. If you don't want Apache to send mail on you're being hacked. Okay. I don't put that pot in so I hope people just don't read this and say oh I didn't know my Apache wanted to send mail. Let me still send mail. There's a tool Auto to Allow that we're going to cover in a little bit. Auto to Allow basically can analyze your log error. So SE Linux error is going to the audit log system by default. They're also now in Ralph's sub and they're going into I think in Ralph's sub or at least in Fedora now they go into the journal as well as the auto log. So if you run Auto to Allow to look at what happened on your system Auto to Allow will give you a message that says here's the rule you'd want to add to your policy but hey there's two Booleans available to allow you to allow this access. Auto to Allow to Diagnose Issues when Apache is running passenger you end up with something that looks like this. So if you wanted to run passenger you can get these Booleans and turn them on. Problem solved. You didn't change the system to false but you changed the system to false but you didn't tell SE Linux about it SE Linux needs to know. SE Linux also controls port types. So we basically say Apache is only able to listen to port 80. SSHD is only allowed to listen to port 23 22. I never remember which. So out of the box if I see an Apache web server trying to listen to port 492 or SSHD and trying to listen to some random port SE Linux is going to block it, report it and say hey something strange going on in the system. So I want SSHD to listen to port 55 SE manage port is the tool you're allowed to change that and this basically says treat port 55 with the SSH port. How do I tell what ports are available? Well, man SSHD and underscore SE Linux will tell you all the port types are available. There's also an SE manage port list which will tell you a huge amount of ports. Port prefix usually matches the type associated with so anybody guess what HTTPD underscore T's port type is? HTTPD underscore port underscore T okay. In REL7, SE policy network is very handy. That shows you stuff like this. SE policy network is basically tools to analyze the policy. It tells me which port is assigned to SSHD port T. And if I want to know all the stuff about the SSHD and T as far as networks are concerned these are all the ports that are allowed. It shows you some Boolean information so there's lots of information available. So those are the two things. So two major causes of SE Linux labeling issues. The second one is you change some configs. You change some normal config and SE Linux needs to be told about it. The next one is SE Linux errors. SE Linux and application errors. Over the years as we've evolved REL4 we had a ton of errors. REL5 we had less errors. REL6 we got better. REL7 we got better. We're always getting better at figuring out these because we've been running SE Linux since Fedora 3 and now we're on Fedora 23. So we've been doing this for 20 versions of Fedora. So SE Linux policy will code pass configurations that we never tried, redirection of standard out. So SE Linux, all we can do with SE Linux is run an application under a confined domain and then wait to see what blows up. So we learn over time. It's like doing static code analysis. We can do a little bit of that but you really need to run these things in the wild to figure out all the things that an application is able to do in the system. And then we write rules about that. I've been not fixed yet. Applications have bugs. Everybody knows what a leaked file descriptor is. So 30 years ago when they were developing the C language they decided that when I did a fork exec I want standard out, standard error and standard end to be leaked to the new process. So when I do a fork all the open file descriptors in my original process get inherited by the child process. Do you know that it's not just 0, 1 and 2 that get inherited. When the fork work socket open it'll get inherited by the child process. Every single process that now becomes a child grandchild on down has full access to that process. When we first started using SC Linux many years ago we found the init system was leaking dev init control. That's the socket that you're able to send a packet to to change the run level of the system. So every process on the system had full access to that unless an intervening process went through the file descriptors. So we were finding all these domains that basically could change the run level in the system. No other system in the world would figure that out except for SC Linux. But anybody that writes code every single developer in this room that writes code probably never thinks to close the file descriptor before an exec. So there are calls that you can basically say this file descriptor I want to close when anybody execs out of here but you have to code that way. So we found hundreds of those. Executable memory, this is basically buffer overflow attacks. SC Linux will block those but a lot of code is badly ridden. Badly built libraries cause these problems. Report bugs to bugzilla if you find them we will fix them. Applications or SC Linux have bugs. You can tell SC Linux to just allow those using to auto allow. So SC Linux is blocking Postgres SQL. Did you make the labeling correct? Did you make sure there were no appropriate booleans? Yes? Most important thing you're going to learn today are the most abused thing you're going to learn today. How do you use auto to allow to build policy? So use auto to allow to build your own custom policy module. If SC Linux blocks something on Postgres SQL you can go through auto to allow. You can go through your audit logs to allow. Give it a dash M and tell it to create mypostgressql.pp policy package. Then you can install the policy package. This basically says SC Linux shut the hell up and let me do what I want to do. Okay? The problem with this is if you do this all the time you're allowing the hacker to do anything you want. So you really should look at what you're doing here. Understand it but if you just want to get if you don't want to send this is better than it's much better than disable SC Linux but it allows you to get by. But when this happens you should really look at what you're adding. Try to figure out what you're adding for rules. Make sure you're not allowing too much and you can always ask for help. There's lots and lots of people on the internet that will help you with your SC Linux issues. You can even send me email. Report the bug to your TIM, your support, your bugzilla mailing list somewhere. Report it if it becomes a bug. So the big one. So all of a sudden we have shell shock we have your Apache web service trying to send 4-0 on the system. Okay? It might be something we want to investigate. You probably don't want to do a send 4-0 audit to allow. I'll let the CGI script turn SC Linux off. So if you have a confined domain that tries to load a kernel module, turn off SC Linux in the forcing mode. Right to SC or right to shadow T. SC tables rules. You might be compromised. The problem is the tooling really can't tell the difference. So SC Troubleshoot tries to do a good job of looking for these known signatures of attack vector like disable SC Linux and figure out no confined domains ever going to be allowed to disable SC Linux. So those are good things to watch for. So SC Troubleshoot has a few diagnosis tools but SC Troubleshoot and SC Linux is not an intrusion detection tool. All it is is protecting you in case you had a vulnerability. Questions? There's more slides after this if we have time. Yes? Okay. So the question is, he's asking about confined users. So I didn't really cover confined users here. On most systems by default the user that logs into the system is unconfined T. That basically means he's able to, he basically has allow star star, colon star star. That's the rules for the unconfined user. So if you have a user that logs into the system is root, he has full access to the system. He logs in your home directory, you have full access to the system. You go through sudo, become root, you can do anything you want. That is the default unconfined user on the system. I think we have a couple slides in the next section that might cover confined users. We can control the user process. The problem is if I controlled the user process out of the box we would have everybody disabling SCLinux because nobody as an administrator wants to be confined. I actually run confined. So I run a staff T and the staff T, the only way staff T user can become root is actually through sudo. And I have a transition rule through sudo that can allow me to come administrator at which point I become unconfined T. There's actually even tighter things called sysadminT that are used in like MLS systems. I have policy, remember I showed a picture where I can find my wife with SCLinux, she runs the thing called ex-guest. She has no reason to ever connect to random ports. All she does is use the web browser to get onto the internet so I can actually control that she can only use connect to the Firefox type port. She can't run su, sudo, there's no way. So if she downloads an application that tries to run su, sudo, SCLinux will block it. Same thing with my parents. My parents run in the ex-guest basically it's a user that can log into a box and do very little. There's even a tighter policy called guestT, which is basically for SSH accounts. If you have an SSH account in your system that you allow hundreds of users to log into you really want to put those guys in a box so you don't want them leaving the network. When I do the longer talks I call it the Hotel California. You can come in anytime you want but you can't leave. So it basically controls who's able to log in the box but you can't SSH off the box. And these are for web services. So my content, fdp.redhat.com, fedora.people.redhat.com, they give me an SSH account so I can update my public HTML but they don't want me running su, sudo and all these tools to try to become root on the system so if I can lock those down with SCLinux. So I have the capability to do that but SCLinux is more about, by default, is more about controlling network-facing services rather than users that log into the system. You can enable the log is used in the system more of an advanced SCLinux thing. You can do a really nice job too of controlling sudo. So you can actually have a confined staff T user and you can set up a user that can execute a shell script as root and then write policy on that process to control it. Yes. I mean you could build some kind of system if you cut off certain categories and treat those differently but pretty much no one to my knowledge is combined. Yes. How do I handle this diversely? Well, SCLinux can run in the container switch, not containers it can run in the VMs at separate policies. So you can run a REL6 box with a REL6 policy on top of a REL7 as long as you're using KVM for virtualization. If you're running inside of containers we basically we don't disable SCLinux inside. We lie inside of the container but SCLinux is enabled because we don't want any process inside of the container accidentally attempting to do SCLinux actions. So if you're inside of a container if you did it again in force it'll say disabled. If you get outside of the container same process in the system it will be in enforcing mode. So in containers we only have one policy and my rule about containers is what happens in Vegas stays in Vegas. So we allow you to do anything you want inside of your container or at MCS. That's basically how OpenShift works too. But yeah, as long as you can use full KVM virtualization separate kernels then you can have separate policies for each object. Anybody else? Yep. There has been some effort to get Fuse to work over the years and there is effort like NFS. So Fuse file systems, certain file systems that don't support X adders can be able to handle it. So NFS for years didn't support it and basically with an SCLinux system you basically label the entire mount point with a single label. So NFS by default NFST. NFS now if you use REL7 clients and servers and hopefully soon NetApp and other services will support. NFS protocol now supports it's just a matter of getting the services all the clients and service to start supporting X adders. There are some weird situations that have never quite been fixed. Anybody using a Samsung phone? Anybody using the Nexus? This is the Nexus phone. You guys disable SCLinux on this? Did you know SCLinux is on it? No. So SC Android. Why is SC Android easier than SCLinux on a REL box? Why is it able to confine it easier? Anybody have any idea? Remember the canonical talk yesterday the amoebas. The amoeba applications crawling all over each other. FTP being able to do 15 different configurations and all this weird configuration stuff that's happening. When I'm using an app on top of one of these phones what is it? It's a box as Space Man was saying yesterday. In our terms we call them containers. So when we move to containers SCLinux becomes a lot better because I can treat every single container as the same object. I no longer have to worry about these random directories and random way you set up things. SCLinux on top of virtual machines on top of containers almost becomes second age. Anybody that disables SCLinux on Docker you are out of your friggin mind. Okay? Especially if you're running a fellow coworker of mine called Docker is all about downloading random shit off the internet and running his root on your system. Okay? And that's really what it is, right? Because you're just going and grabbing random content and you put it in a container and you're giving it root. And there's very little separation. There's some separation I do a much longer talk on that was rejected to here on Docker security but SCLinux provides the best file system separation again because it can basically say you are a container process running on a system to read database data. You're not going to be allowed. You try to read. That's your shadow. You're not going to be allowed. Basically what happens in Vegas stays in Vegas. You try to break out of Vegas. SCLinux is going to shut you down. Okay? And because it's all built into the system it's built into Docker. The only time you have to worry about labeling is if you volume mount into it. There's a colon Z, lowercase Z and a colon uppercase Z which actually will fix the labels it will fix the SCLinux label of whatever you volume mounted. Someone back in the room. No. Because SCLinux is not a mechanism for getting around discretionary access control or other types. Security is always in depth. You always want when I run Docker I have read-only file mounts. I have discretionary access control drops. I have capability drops. I have SCLinux drops. I have Seccomp. I want all these things. What I don't want is one of them installed and bypasses the other ones. So you have to configure everything. So we're not giving you back doors with SCLinux. So in order to do in order to become root on the system if you're a confined application you have to not only change your label SCLinux label to do stuff but you also have to change through sudo to become root UID zero. So that's different. There's no secret SCLinux bypass mode to bypass things like discretionary access control. Are we running out of time? Yeah. No. To use live fuse? Lib fuse? I don't know. I don't know what that does. Yeah, I don't know fuse. No. I'm just worrying about it. Okay. I'm going to upload all these slides. I think we're totally out of time now. The second part of this talk covers how to manage SCLinux in a distributed environment which is really critical and we don't, again, this talk would take three days if we wanted to go through it all but the main idea is you configure a lot of people go on and you configure one machine the way you want it. So you go on and you do all your SCManage commands you do all that. SCManage commands also have the ability to extract your configuration all your SCLinux configuration into a flat file you can then move that flat file to another machine and load it. Okay. And what that will do is basically take all your booleans, all your file context changes, any poor changes, things like that. So if you configured one Apache server the way you want it and you want to duplicate that Apache server on 100 machines you can do that with these tools. Lastly about SCLinux, again, this covers it. The question we've often gotten asked about SCLinux, don't we have some magical tool that makes managing SCLinux easy? And I always say yes we do. Ansible, CF Engine, okay. Whatever... SCLinux is configuration data. It happens to be security configuration data but it's configuration data on your system. If you manage your systems using the satellite with RPMs you can package SCLinux and satellites on RPMs. If you manage using Ansible, great idea. We now own them. You can do it with Puppet. You can do it with other tools, okay. So managing SCLinux in the enterprise the way that Fedora does it the way that OpenShift does it they use Ansible to manage hundreds to thousands of machines using standard tools, okay. Thank you for coming and did the Patriots lose? The Patriots lost? I don't know. Wow. That's correct. Yeah, Dutch. Van Harding. Yeah, Van is German and Van is Dutch. And my name means Garden. Garden? And Dutch. That's the extent of my Dutch speaking skills. You're probably better off that it's your middle name because my life has spent telling people it's two words and finding my name under V or is it under T. A lot of technology software doesn't handle two words, airlines they're not sure the intent in Norwegian. Oops, I think we have to get started. We'll catch up after it, yeah. It's the last session. People want to go home. Hi everyone, thanks for sticking it out. This is the last talk with the security track here of Scale 14x. Thanks for sticking around. Our speaker now is going to be Chris Vantyne. It's a security, state of mind container security, a.k.a. compliance and vulnerability audits for containers. Hello everyone. Thank you for having me here at Scale and my name is Chris Vantyne, Chief Technologist for the Western Region at Red Hat. I've been with Red Hat for about 10 years working with strategic partners and customers who are adopting Red Hat's emerging technologies. You know, Red Hat will be on Linux in the middleware, software defined storage, virtualization, container and cloud management. And, you know, certainly one of the hot topics with our customers is around containers. And everyone's talking about Docker or Linux containers. You know, customers are asking us questions. Are containers ready for the enterprise? You know, what about security implications of deploying containers in the enterprise? And what are the tools that I could actually use for security compliance? Checking for security vulnerabilities. And so today we'll be talking a little bit about OpenScap. But before we dive into the technology, we do want to dive into, you know, why Linux containers in the enterprise? What's really accelerating the adoption? What are Linux containers from a technical perspective as well as container security? And then go through some example use cases and the different tool sets that are available with OpenScap. Which is an upstream open source project. You know, does anyone recognize what this is a picture of? That's correct. It's the Tesla assembly line. And the reason I show this is that you know, when was the last time cars actually improved after you left the dealer a lot? That's exactly what happens with Tesla. And software is a huge enabler of making that happen. You read in the papers how they release a software update to improve the 0 to 60 acceleration time, update navigation systems. They also are able to do vehicle recalls, if it's a software issue, over the wire without having the owner bring in the vehicle into a dealership. And so that's saving tremendous amount of money and totally transforming the automobile industry. And so software is a huge part of that. So this need for accelerating software application delivery from development into production is being driven by business need. And so IT has really got to respond to this. Gartner calls this the era of the Mode 1 and Mode 2. Mode 1 being your traditional applications, maybe it's an Oracle database, really relied on the underlying infrastructure, wasn't updated very frequently, maybe 12, 18 months. You see that on the left side. And there's a shift in really the how, what, and where of application development process, application architecture, and the underlying infrastructure. And containers are a big part of that because they're enabling the business to accelerate application delivery and improve overall quality. And so there's a key problem facing IT today. We talk a lot of enterprises and they have on the left, you have the business really driving the need to move faster like Tesla and deliver updates to their customers. And they're putting a lot of pressure on the developers to write that code. And the problem with the developers, they want to have the access to the latest technology, the latest tool sets. But on the right side in the enterprise, you have operations in security and compliance. They're trying to maintain uptime. They're trying to maintain security. And it's causing friction between the two sides of the house. And so how do we eliminate that friction? And containers are a big part of that. So containers, the promise is that they're going to improve application delivery and enable the developers to have access to the tools they want and be able to bring in the dependencies for those applications and allow them to control that. Yet at the host level, allow operations to have control and lock down the systems in terms of a security perspective. And so this is the end state of where you have the developers doing what they need to do and delivering to the business in a timely manner. And operations having the ability to have their environment secure and meet the security and compliance standards without having this friction between the two groups. And so that's the promise of containers. And containers really the key thing there is that it allows you to package once and deploy anywhere. It can take a container and deploy it on physical, virtual, private, public cloud. And it provides a lightweight isolation of the process, the network, as well as the storage. And containers have been around for a while. Solar zones, Linux has been in the kernel. Google did a lot of early work. But what is around until recently, the past couple of years when Docker came along and kind of standardized provided a standard API, an image format, and also a runtime but also a delivery and sharing model. And so those have a lot of security implications as well. And one of the things about adopting and accelerating technology quickly is coming together as an industry with different standards. And so that's where the Open Container Initiative, which is a part of the Linux Foundation, it's really a consortium of all these different companies who came together and said, hey, we need to standardize in terms of the Linux container technology in terms of the packaging format as well as the runtime. And so they took Docker's standards and made those the standard of this consortium. And so as a result, we can move forward and continue to innovate at a much faster pace. And so when you take a look at containers from a security perspective, really three things. One is you're building the containers. As a developer, you have a Docker file. From a best practice, you want to keep those Docker files in a source code repository so you have versioning and control. Secondly, you ship those containers, images into a shared repository whether it's a public or private. A private repository provides a lot more control. Also, you know exactly what's going in there and who put it in there versus a public registry. And then lastly, you're going to be running these images on your physical virtual cloud systems. And then there are security implications for having many containers running on the same host, which we'll get into. And so from a technical perspective, what's the difference in terms of a container architecture versus a traditional system? On the traditional system, you have an application maybe application A and application B, and they're sharing all the different libraries on the host operating system as well as in a virtual machine. Each virtual machine has its own dedicated kernel. Now take a look at the container side. When you build the container, the application pulls in its dependencies into that container package. So you can have different versions of libraries all dedicated to a particular container and have those run side by side. But a big difference here is that these containers share the exact same kernel. The container doesn't have its own dedicated kernel. So very different than a virtual machine. And so think about from a security perspective, now a security vulnerability in a container environment can not only impact container A, but also can impact container B and the host as well. And so patching becomes that much more important, and the kernel is that much more important as well to the overall environment. All the applications. It's not isolated. And so what is the underlying technologies that really bring containers to the real world? And there are several key technologies. First off is control groups. This provides resource management. Google did a lot of early work around this. It provides the ability to limit the consumption of CPU, memory, disk, and network. So providing quality of service which is really important, especially in a public cloud where you have different customers running on containers on the same system. And you want to be able to provide guarantees in terms of quality of service in terms of CPU availability, memory, etc. Secondly, namespaces they provide process isolation. So this is a namespace such as a user namespace user ID and a container of, let's say, root is different than the user at the host level. And so this is an area that's being developed in the upstream community right now in this technology preview in terms of user namespace. And we'll talk a little bit about other namespaces in a minute. SC Linux provides additional security. So if you're here for Dan's talk prior to this Mr. SC Linux over at Red Hat he talks about how SC Linux provides mandatory access control and allows Red Hat to deliver containers in a much more secure manner because it basically enables the container to be locked down so that if there's a security breach in the container, it doesn't negatively impact other containers or the host running on that system by limiting the files that can be accessed by that container user. Also another key part of security around containers are the images themselves. So Docker container images have different layers. So you typically would start out with a base layer. You want to make sure that base layer such as rel7, sent off, etc. you want to be sure where are you pulling those images from? Is it a reliable and a trustworthy source? And then in the container image as you write or add applications or make changes those layers are layered on top separately from that base image. So let's talk about container security. Does anybody recognize some of these brands? Well here are some of the most recognized brands in America and what are they having in common is that in 2014 they experienced some of the largest data breaches. They represent over a billion data records that were hacked in 2014. And so what were the top reasons for these data loss? Malicious outside attacks, accidental loss but also malicious insider attacks. And so we did a poll Forster pulled whether the top challenges for enterprises adopting containers and certainly security security is the number one concern. So as you deploy into enterprise you need to definitely have a sound security strategy policy as well as tooling to implement that. And so over the years I've worked with many customers when I first got to Red Hat I'd ask them about their security practices and the common response was we have a firewall that is our security process and tool. But as we saw malicious attacks on the inside are also a key concern and Dan speaks about FELinux and containers do not contain and that's a key thing to understand. And they don't contain because not all resources are namespaced in Linux. So you have the current key ring which is shared amongst the containers the kernel itself and all the modules any device the system time and in Red Hat Enterprise Linux user namespaces as well. There's one tech preview of turning those on by default it's disabled but you can enable it by specifying the kernel boot option that's listed here. And so what does this mean is that you know today the root user in a container is the same as the root at the host level. So if you get out of the container you then have root access to the entire system of the containers running on that system. In Red Hat Enterprise Linux 7.2 it's a tech preview so it was so in Red Hat Enterprise Linux 7.2 and sent off it's in technology preview at the kernel level it hasn't been enabled for Docker yet to take advantage of that so from a Docker perspective there are other distributions where it has been enabled but Red Hat is focused on enterprise so we want to make sure that this new technology is hardened and ready for prime time in terms of enterprise deployments. So it's an active area of work upstream and certainly there's parts of it that have been back ported into REL and if you specify this boot option you can try it out in the REL 7.2 which is the current release. So what are some of the security risks with containers? You know kernel exploits so if you exploit the kernel you didn't have access to all the different kernels potentially denial of service attacks container breakouts poison images we'll talk about that in a minute also compromise secrets because being able to specify those in the Docker file and you know why does the container image matter? So we did an analysis of the container images on the public repository Docker Hub where you can go and pull down images many of the developers probably in your enterprise are doing that and you know 36% of those images had a high priority security issue such as heart bleed or shell shock 28% had a medium priority so 64% the images probably a lot of the developers are pulling down into the enterprise don't have security issues and that's just at the time of pulling them down just think about what happens to that image over time and let's take a look at that so here is an example of different container images and the dependencies that they pull in into that container image so for example we have a C application on the far left that pulled in Glib C and Bash Java bringing in JRE Node.js application and then a Perl PHP application you can see the different components it pulls in to the container image well these little triangles show you the number in there of how many times a security update has been released for that component since RHEL 7.0 GA PHP over 20 notifications and security updates so the key thing here again is at the point of time you pull it down it might be completely secure from a vulnerability perspective but six months down the road you may have 5, 10 different vulnerabilities that need to be patched so what's your process to update these containers over time and here shows you in terms of how many vulnerabilities per package lasts in 2014 for the different components and the kernel number one in terms of having security issues also Java another victim of security issues as well so the kernel is so important in the container environment really need to have a security process and tooling to make sure that constantly update it as well as the programming applications and so this brings us to OpenSCAP OpenSCAP upstream project to address compliance and vulnerabilities how many have heard of NIST so NIST they provide a government database of CVEs basically releasing to the public without security vulnerabilities to different software packages they also have checklists to help you define policy for your systems to conform with security standards by the government agencies and so CVE what it looks like it has the impact important critical etc the release date and in Red Hat associated bugzilla and also the details around the security vulnerability so as you're maintaining your environment you have a good idea of why I would actually want to address this issue and how critical is it to my environment and then CVEs there's a checklist database of different policies that are recommended to keep your environment secure or your container secure for example there's a check to make sure how long is your minimum password setting on your Linux host and so OpenScap consists of really three things content, scan and reports content Red Hat is providing security guides for our distribution other distributions are also providing guides as well the CVEs are driving the checklist for compliance and then the CVE database is really driving the vulnerability notifications as well and then there's the tooling so OpenScap we'll go into the different tools in a minute but it's a set of different tools and technologies Foreman is one of the tools that it integrates with which is an upstream provisioning project for host, virtual machines, etc and then the scan outputs different reports about your system or set of systems in terms of is my container what vulnerabilities does it have what packages need updating and then also from a policy compliance perspective what checklist failed in the scan in terms of maybe a minimum password what services are enabled that should be disabled, etc and so the OpenScap toolset really consists of currently five different key things one is the OpenScap base project this is the CLI in library and we'll go through some of the use cases OpenScap Damon a service that runs in the background can automate the running of scans SCAP workbench is a way to actually edit and define and customize policies for OpenScap SCAP, the money is a middleware that can be integrated and today it integrates with Foreman provides a database so that you can have a history of your scan information and then OSScap, the Anaconda add-on is a plug-in essentially for the installer the Anaconda Linux installer so that when you install a system it can run the scan, the policy check at system creation so you start on a good foot and we'll get an example of that as well so let's take a look at OpenScap and some of the use cases so first use cases, you want to be able to scan for compliance, you know things you might be wanting to do are password quality requirements that are obsolete services enabled like Telnet, you want to make sure that's off is OpenSSH properly configured is flash temp on a separate partition and so to do this you would install the OSScap utilities and there are a variety of command line options to customize the output of where it's going and then an example in real-time of running the OSScap tool shows you each individual check and whether it passed or failed so this is running on a host system and then as an output of that scan you can then go into a web browser and view the evaluation report have some information about the particular host but then you can see an overall compliance and scoring for this system so here you can see 34 out of 68 checks passed that's a lot of failures and also shows you the criticality 3 high, 16 medium so then I can drill down and see what the issues are and then line by line I can see the individual checks that are in the policy again this policy is customizable and the nice thing is you can actually drill in into each check you get the details about the check but there's also a remediation script that can be applied to bring that container, that virtual machine that host into compliance with that policy so you can press the button and it will run this script on the host or you can just copy that and customize it a little bit and run it as well so very flexible another use case for OpenSCAP is the scan for known vulnerabilities these are out of date you know RPM packages for instance what ones need updating was the criticality of these vulnerabilities what's the vulnerability what CVs have not been addressed on this particular host and so I can go ahead and pull down the security content and then run the vulnerability scan here with the OSCAP using the Oval option now and then you can actually view the report and in real time you'll see the output on the command line of every CV check and whether it's false or positive and then once it's done I can go ahead and view the report in the web browser and so particular host and then line by line I can see where the CVs failed and I need to actually patch these systems to resolve that issue so those were examples of running on a host or a virtual machine but you also can run these checks and these scans on containers so you want to know is the Docker image compliant is the image patched is the Docker container compliant so the image being at rest the container being running so you can do it for either use case the first thing you want to do is install what's called the OSCAP Docker utility so these are all examples on Sintos or Red Hat Enterprise Linux I'm going to install I do install Docker here and what I'm going to do is fire up a Docker image as well REL6.2 so I'm running 6.2 on a REL7 host and I can do offline and online so the first example I'm going to check a Docker image just on my file system it's not running as a container yet and I can do a compliance scan or a vulnerability scan so a lot of ways you can automate this or run it at scale maybe against your repository and then also maybe you have a production environment running containers you can do a scan on those containers that are running as well without bringing them down so you can do a vulnerability scan or a compliance scan so the slides will be available if you want to look at the details in terms of the command line options and so then there's also the OSCAP Daemon so this is a service that runs in the background you can evaluate machines, containers according to the schedule provided so like a CRON schedule you may want to continually evaluate a container or machine for a specific policy today it's available on Fedora it'll be available on other distributions as it matures mention the OpenSCAP Workbench so this is a GUI tool easy way for average user to really define policies customize them for a single machine or a remote machine and you can actually do remediation through this tool as well it's available on a lot of different operating systems Fedora, REL, CentOS Scientific Linux, Debian Ubuntu and even a Windows and OSX version as well and then here's the OSCAP Anaconda add-on so just showing you an example this is the CentOS or REL installer so as a part of that process you'll be presented with the option of selecting a profile from a security perspective that you want to apply to this box as a part of the install process it will remediate any issues in the base OS through the installation process and so let's take an example of the value this is providing on the left side this is doing a base install on REL and then running this particular policy check and there was only 64% compliance score this is with selecting the policy and having it applied during install process and then running the scan after and it was roughly 99% compliant so from the get go your systems are secure and not exposed to any vulnerabilities right away so good practice for enterprise deployments and I mentioned scaptomony this is the middleware and it today has been integrated with formin which is the provisioning solution for bare metal virtual machines etc and so this provides a dashboard to manage not just one system but hundreds or thousands of systems and so you get a summation and a database so you have audit trail and records of these scans over time and so this is just an example showing you the different hosts you can see the pass failed and then you can actually drill in and view the report so upstream project open source formin and so what are some of the container best practices so first thing is only run container images from trusted parties you're deploying in the enterprise you want to know where are your developers pulling these images from what's inside these images who built these images also running containers as root not a good idea so drop privileges as soon as possible in your containers especially until user namespaces is fully implemented and supported host operating system matters right the kernel is critical to a container environment even more so than before in a virtual environment so you want to make sure you have a solid kernel you want to make sure that you're updating that kernel and you have a reliable source and provider of security updates for that kernel again apply kernel security fixes because of that on a regular basis how many of you disable se linux today right in a container world not a great idea because if you hack the kernel you get access to the host and all the containers so by having se linux policies it will restrict where they can get to and that example we had earlier over time container images of course will have more security issues than when they are initially pulled down from the repository so you want to have an ongoing process to scan and remediate any issues in the containers in terms of security vulnerabilities or policy failures additional resources in terms of best practices where it has a security guide you can download on our website in terms of hardening se linux Dan had an earlier talk has a lot of great tips and overview of se linux he also has a coloring book to really simplify and make it easy to understand se linux audit logs syslog, systemd, journald identity management so having user management security security blog we publish a blog with different security tips on linux but also containers docker, etc. great place and resource to check out the three pigs coloring book that Dan Walsh at Red Hat who leads our docker engineering team and se linux has one on containers as well as se linux and the container one talks about the three little pig story and how that relates to containers versus virtual machines versus hosts really showing you that a container kernel there's a single kernel that impacts the containers whereas a physical machine has its own dedicated kernel as well as a virtual machine so really important to understand the implications of that and that's all I had today so thank you very much for attending today's session happy to answer your questions about OpenScap project encourage you to check out the OpenScap website as well you can get a lot more detailed information I think there's some demonstrations and then also community as well to get help or even contribute to the upstream projects so thank you very much the policies are defined by the government and then we are a distributor for our distribution of these there's like PCI compliance there's standard REL security guide etc so we've also customized these a little bit but the original source of this content is this and the workbench allows you to customize them and remove certain checks that may not be applicable to your environment without actually modifying the files it's a layering technology so yeah not edit the file so it's tech preview and they need to enable it with Docker typically tech previews I don't think we've committed on a date or a release yet to kind of get customer feedback and see how it goes but you know typically certainly in the REL 7 timeframe I would anticipate so Foreman can do that for instance so it has the ability to use systems like an agent base yeah satellite server for Red Hat allows you to do that oh it's the latest version has SCAP integration any other questions great well thank you again